Learning in cognitive systems with autonomous dynamics
Abstract
The activity patterns of highly developed cognitive systems like the human brain are dominated by autonomous dynamical processes, that is by a selfsustained activity which would be present even in the absence of external sensory stimuli.
During normal operation the continuous influx of external stimuli could therefore be completely unrelated to the patterns generated internally by the autonomous dynamical process. Learning of spurious correlations between external stimuli and autonomously generated internal activity states needs therefore to be avoided.
We study this problem within the paradigm of transient state dynamics for the internal activity, that is for an autonomous activity characterized by a infinite timeseries of transiently stable attractor states. We propose that external stimuli will be relevant during the sensitive periods, the transition period between one transient state and the subsequent semistable attractor. A diffusive learning signal is generated unsupervised whenever the stimulus influences the internal dynamics qualitatively.
For testing we have presented to the model system stimuli corresponding to the barstripes problem and found it capable to perform the required independentcomponent analysis on its own, all the time being continuously and autonomously active.
1 Introduction
It is well known that the brain has a highly developed and complex selfgenerated dynamical neural activity. We are therefore confronted with a dichotomy when attempting to understand the overall functioning of the brain or when designing an artificial cognitive system: A highly developed cognitive system, such as the brain, is influenced by sensory input but is not driven directly by the input signals. The cognitive system needs however this sensory information vitally for adapting to a changing environment and survival.
In this context we then want to address a twofold goal:

Can we formulate a meaningful paradigm for the selfsustained internal dynamics of an autonomous cognitive system?

How is the internal activity process influenced by sensory signals, viz which are the principles for the respective learning processes?
With respect to the first problem we note that an increasing flux of experimental results from neurobiology supports the notion of quasistationary spontaneous neural activity in the cortex [1, 2, 3, 4, 5]. It is therefore reasonable to investigate possible setups for artificial cognitive systems centrally based on the notion of internally generated spontaneous transient states, as we do in the present investigation.
2 General Outline
We will present the model and the issues being studied in two steps, starting with a general overview, discussing technical details in a second step.
2.1 Transientstate and competitive dynamics
There are many notions of complex system theory characterizing the longterm behavior of a dynamical system [6], such as regular vs. chaotic behavior. The term ‘transientstate dynamics’ refers, on the other hand, to the type of activity occurring on intermediate time scales, as illustrated in Fig. 1. A time series of semistable activity patterns, also denoted transient attractors, is characterized by two time scales. The typical duration of the activity plateaus and the typical time needed to perform the transition from one semistable state to the subsequent one. The transient attractors turns into stable attractors in the limit .
Transient state dynamics is intrinsically competitive in nature. When the current transient attractor turns unstable the subsequent transient state is selected by a competitive process. Transientstate dynamics is a form of ‘multiwinnerstakeall’ process, with the winning coalition of dynamical variables suppressing all other competing activities.
Similar processes have been proposed to be relevant for various neural functionalities. Edelman and Tononi [7, 8] argue that ‘critical reentrant events’ constitute transient conscious states in the human brain. These ‘statesofmind’ are in their view semistable global activity states of a continuously changing ensemble of neurons, the ‘dynamic core’. This activity takes place in what Dehaene and Naccache [9] denote the ‘global workspace’. The global workspace serves, in the view of Baars and Franklin [10], as an exchange platform for conscious experience and working memory. Crick and Koch [11] and Koch [12] have suggested that the global workspace is madeup of ‘essential nodes’, i.e. ensembles of neurons responsible for the explicit representation of particular aspects of visual scenes or other sensory information.
2.2 Competitive dynamics and sensitive periods
Within the setting of a generalized neural network we utilize a continuoustime formulation with rateencoding neural activity centers, characterized by normalized activity levels . One can then define, quite generally, via
(1) 
the respective growth rates . Representative time series of growth rates are illustrated in Fig. 2. When the the respective neural activity increases, approaching rapidly the upper bound, as illustrated in Fig. 1; when it decays to zero. The model is specified [13, 14], by providing the functional dependence of the growth rates with respect to the set of activitystates .
During the transition periods many, if not all, neurons will enter the competition to become a member of the new winning coalition. The competition is especially pronounced whenever most of the growth rates are small in magnitude, with no small subset of growth rates dominating the all the others. Whether this does or does not happen depends on the specifics of the model setup, in Fig. 2 two cases are illustrated (upper/lower graph in Fig. 2). In the first case the competition for the next winning coalition is restricted to a subset of neurons, in the second case the competition is networkwide.
When most neurons participate in the competition process for a new winning coalition the model will have ‘sensitive periods’ during the transition times and it will able to react to eventual external signals.
2.3 Sensitive periods and learning
So far we have discussed in general terms the properties of isolated models exhibiting a selfsustained dynamical behavior in terms of a neverending time series of semistable transient states, as illustrated in Fig. 2, using rateencoding equations specified in previous work [13, 14].
The importance of sensitive periods comes when this model is coupled to a stream of sensory input signals. It is reasonable to assume, that external input signals will contribute to the growth rates via
(2) 
where the encode the influence of the input signals. Let us furthermore assume that the input signals are suitably normalized, such that
(3) 
For the transient states the for all sites not forming part of the winning coalition and the input signal will therefore not destroy the transient state. With the normalization given by Eq. (3) the total growth rate will remain negative for all inactive sites. The input signal will however enter into the competition for the next winning coalition during a sensitive period, providing an additional boost for the respective neurons.
This situation is exemplified in Fig. 3, where we present simulationresults for a system containing neurons subject to two sensory inputs . The selfgenerated time series of winning coalitions is both times redirected by the strongest component of the sensory input, the details depending on whether the input signal partially overlaps with the current winning coalition or not.
We then have a model system in which there are well defined timewindows suitable for the learning of correlations between the input signal and the intrinsic dynamical activity, namely during and shortly after a transition period, that is the sensitive period. A possible concrete implementation for this type of learning algorithm will be given further below.
This setup would allow the system also to react to an occasional very strong input signal having . Such a strong signal would suppress the current transient state altogether and impose itself. This possibility of rare strong input signals is evidently important for animals and would be, presumably, also helpful for an artificial cognitive system.
2.4 Diffusive learning signals
Let us return to the central problem inherent to all system reacting to input signals having at the same time a nontrivial intrinsic dynamical activity. Namely when should learning occur, i.e. when should a distinct activity center become more sensitive to a specific input pattern and when should it suppress its sensibility to a sensory signal.
The above developed framework of competitive dynamics allows for a straightforward solution of this central issue: Learning should occur then and only then when the input signal makes a qualitative difference, viz when the input signal deviates the transientstate process. For illustration let us assume that the series of winning coalitions is
where the index indicates that the transition is driven by the internal autonomous dynamics and that the series of winning coalitions take the form
in the presence of a certain sensory signal . Note, that a background of weak or noisy sensory input could be present in the first case, but learning should nevertheless occur only in the second case.
Technically this is achieved by defining a
suitable diffusive learning signal
(4) 
viz when it makes a qualitative difference. Here is the internal contribution to the growth rate, i.e. the value it would have in the absence of the input signal and the are the growth and decay rates for the diffusive learning signal respectively. The diffusive learning signal is a global signal and a sum over all dynamical variables is therefore implicit on the righthand side of Eq. (4).
2.5 The role of attention
The general procedure for the learning of correlation between external signals and intrinsic dynamical state for a cognitive system presented here does not rule out other mechanisms. Here we concentrate on the learning algorithm which occurs automatically, one could say subconsciously. Active attention focusing, which is well known in the brain to potentially shut off a sensory input pathway or to enhance sensibility to it, may very well work in parallel, to the continuously ongoing mechanism investigated here.
3 Associative Thought Processes
So far we have described, in general terms, the system we are investigating, having sensitive periods during the transition periods of the continuously ongoing transientstate process, with learning of input signals regulated by a diffusive learning signal.
Our system consists of two components, as illustrated in Fig. 4. For the component generating an infinite timeseries of transient state we employ a dense homogeneous associative network (dHAN). The dHANmodel has been studied previously and shown to generated a time series of transient states characterized by high associative overlaps between subsequent winning coalitions [13, 14]. The time series might therefore be interpreted as an associative thought process, it carries with it a dynamical attention field [13].
3.1 Input datastream analysis
The input signal acts via Eq. (2) on the dHAN layer, with the contribution to the growth rate of the dHAN neuron given by
(5) 
where we have denoted now with the superscripts the dHAN and inputlayer respectively. For subsequent use we also define in Eq. (5) an auxiliary variable which quantifies the influence of inactive inputneurons. The task is now to find a suitable learning algorithm which extracts the relevant information from the inputdata stream by mapping distinct inputpatterns onto selected winning coalitions of the dHAN layer. This is the setup typical for an independent component analysis [15].
The multiwinnerstakeall dynamics in the dHAN module implies individual neural activities to be close to 0/1 during the transient states and we can therefore define three types of interlayer links (see Fig. 4):

active (‘act’)
Links connecting active input neurons with the winning coalition of the dHAN module. 
orthogonal (‘orth’)
Links connecting inactive input neurons with the winning coalition of the dHAN module. 
inactive (‘ina’)
Links connecting active input neurons with inactive neurons of the dHAN module.
The orthogonal links take their name from the circumstance that the receptive fields of the winning coalition of the target layer need to orthogonalize to all inputpatters differing from the present one. Note that it is not the receptive field of individual dHANneurons which is relevant, but rather the cumulative receptive field of a given winning coalition.
We can then formulate three simple rules for the respective linkplasticity. Whenever the new winning coalition in the dHANlayer is activated by the input layer, viz whenever there is a substantial diffusive learning signal, i.e. when exceeds a certain threshold , the following optimization procedures should take place:

active links
The sum over active links should take a large but finite value , 
orthogonal links
The sum over orthogonal links should take a small value , 
inactive links
The sum over inactive links should take a small but nonvanishing value ,
The , and are the target values for the respective optimization processes. In order to implement these three rules we define three corresponding contributions to the link plasticities:
(6) 
where the inputs and to the target layer are defined by Eq. (5). For the signfunction is valid for and respectively, denotes the Heavisidestep function in Eq. (6). The interlayer links cease to be modified whenever the total input is optimal, viz when no more ‘mistakes‘ are made [16].
0.008 0.001 0.001  0.9 0.3 0.2  0.4 0.2  0.25 
Using these definitions, the link plasticity may be written as
(7)  
where is an appropriate threshold for the diffusive learning signal and are the desired contributions to the growth rate for active/inactive postsynaptic neurons, and the respective rates and and the critical activity reservoirs defining an active/inactive postsynaptic center respectively. A suitable set of parameters, which has been used throughout this work, is given in Table 1.
We note, that a given interlayerlink is in general subject to competitive optimization from the three processes (act/orth/ina). Averaging would occur if the respective learning rates // would be of the same order of magnitude. It is therefore necessary, that
3.2 Homeostatic normalization
It is desirable that the interlayer connections neither grow unbounded with time (runawayeffect) nor disappear into irrelevance. Suitable normalization procedures are therefore normally included explicitly into the respective neural learning rules and are present implicitly in Eqs. (6) and Eq. (7).
The strength of the inputsignal is optimized by Eq. (7) both for active as well as for inactive layer neurons, a property referred to as fanin normalization. Eqs. (6) and (7) also regulate the overall strength of interlayer links emanating from a given layer neuron, a property called fanout normalization.
Next we note, that the timescales for the intrinsic autonomous dynamics in the dHANlayer and for the input signal could in principle differ substantially. Potential interference problems can be avoided when learning is switched/one very fast, viz when activation and when the decay rate are larger, i.e. when is smaller than both the typical time scales of the input and of the selfsustained dynamics.
4 The Bars Problem
A cognitive system needs to autonomously extract meaningful information about its environment from its sensory input data stream via signal separation and features extraction. The identification of recurrently appearing patterns, i.e. of objects, in the background of fluctuation combinations of distinct and noisy patterns constitutes a core demand in this context. This is the domain of the independent component analysis [15] and blind source separation [17], which seeks to find distinct representations of statistically independent input patterns.
In order to test our system madeup by an inputlayer coupled to a dHAN layer, as illustrated in Fig. 4, we have selected the bars problem [18]. The bars problem constitutes a standard nonlinear reference task for the feature extraction via an independent component analysis for a input layer. Basic patterns are the vertical and horizontal bars. The individual input patterns are madeup of a nonlinear superposition of the basic bars, containing with probability any one of them, as shown in Fig. 5.
4.1 Simulation results
For the simulations we presented to the system about randomly generated input patterns of the type shown in Fig. 5, including a small noiselevel of about . The individual patterns lasted with about for the time in between two successive input signals. These timescales are to be compared with the timescale of the autonomous dHANdynamics illustrated in the Figs. 2 and 3, for which the typical stabilityperiod for a transient state is about . This implies that we presented the system sparse (in time) sensory input. We also note that the number of training patterns in our simulation is exceedingly small, standard neural algorithms use routinely and more training patterns [19].
The results for the simulations are presented in Figs. 6 and 7. For the geometry of the dHAN network we used a simple 15site chain containing 14 potential winning coalitions, as illustrated in Fig. 7, namely , , …, .
In Fig. 6 we present the response
(8) 
of the 14 potential winning coalitions to the 10 basic input patterns , the isolated bars. Here denotes the set of sites of the winningcoalition and its size, here .
The individual potential winning coalitions have acquired in the course of the simulation, via the learning rule Eq. (7), distinct susceptibilities to the 10 bars compare Fig. 6. We note that the learning is unsupervised and quite fast. This exampleproblem is overcomplete, there are more potential winning coalitions than statistically independent basic pattern. Perfect signal separation can therefore not be expected. E.g. the secondlast vertical bar in Fig. 7 is not resolved.
4.2 Receptive fields
The receptive fields
(9) 
of the 14 potential winning coalitions are given in Fig. 7. The interlayer synaptic weights can be both positive and negative and the orthogonalization procedure, Eq. (6), results in complex receptive fields.
Physically separated target neurons, viz a ‘singlewinnertakesall’ setup, are normally used for standard analysis neural algorithms performing an independent component analysis [15]. The winning coalitions of the dHANlayer are however overlapping and every link targets in general more than one potential winning coalition in the dHAN layer, in our case two for (and one for ).
The unsupervised learning procedure, Eq. (7), involves therefore a competition between the active contribution and the and , as given by Eq. (6). The interlayer synaptic weights are therefore only changed to the point necessary for recognition, but not up to the saturation point, as evident from the data presented in Fig. 7. This behavior is consistent with the ‘learning by mistakes’ paradigm [16], which states that a cognitive system needs to learn in general only when committing a mistake.
5 Conclusions and Future Works
A cognitive system has its own internal dynamics and we studied here the interplay of these selfgenerated activity states, the timeseries of winning coalitions, with the sensory input for the purpose of unsupervised feature extraction. We proposed learning to be autonomously activated during the transition from one winning coalition to the subsequent one.
This general principle may be implemented algorithmically in various fashion. Here we used a generalized neural net (dHAN  dense homogeneous associative net) for the autonomous generation of a time series of associatively connected winning coalitions and controlled the unsupervised extraction of inputfeatures by an autonomously generated diffusive learning signal.
We tested the algorithm for the bars problem and found good and fast learning for the case of sparse temporal input. Preliminary results indicate that the learning algorithm retains functionality under a wide range of conditions. We plan to extend the simulations to various forms of temporal inputs, especially to quasicontinuous input and to natural scene analysis and to study the embedding of the here proposed concept within the framework of a fullfledged and autonomously active cognitive system.
Footnotes
 The name ‘diffusive learning signal’ [6] stems from the fact, that many neuromodulators are released in the brain in the intercellular medium and then diffuse physically to the surrounding neurons, influencing the behavior of large neural assemblies.
References
 M. Abeles et al., Cortical activity flips among quasistationary states, PNAS, vol. 92, 1995, pp 86168620.
 D.L. Ringach, States of mind, Nature, vol. 425, 2003, pp 912913.
 T. Kenet, D. Bibitchkov, M. Tsodyks, A. Grinvald and A. Arieli, Spontaneously emerging cortical representations of visual attributes, Nature, vol. 425, 2003, pp 954956.
 J.S. Damoiseaux, S.A.R.B. Rombouts, F. Barkhof, P. Scheltens, C.J. Stam, S.M. Smith and C.F. Beckmann, Consistent restingstate networks across healthy subjects, PNAS, vol. 103, 2006, pp 1384813853.
 C.J. Honey, R. Kötter, M. Breakspear and Olaf Sporns, Network structure of cerebral cortex shapes functional connectivity on multiple time scales, PNAS, vol. 104, 2007, pp 1024010245.
 C. Gros, Complex and Adaptive Dynamical Systems, A Primer, Springer; 2007 (in press).
 G.M. Edelman and G.A. Tononi, A Universe of Consciousness, New York: Basic Books; 2000.
 G.M. Edelman, Naturalizing consciousness: A theoretical framework, PNAS, vol. 100, 2003, pp 55205524.
 S. Dehaene and L. Naccache, Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework, Cognition, vol. 79, 2003, pp 137.
 B.J. Baars and S. Franklin, How conscious experience and working memory interact, Trends Cog. Science, vol. 7, 2003, pp 166172.
 F.C. Crick and C. Koch, A framework for consciousness, Nature Neurosci., vol. 6, 2003, pp 119126.
 C. Koch, The Quest for Consciousness  A Neurobiological Approach, Robert and Company; 2004.
 C. Gros, SelfSustained Thought Processes in a Dense Associative Network, in KI 2005, U. Furbach (Ed.), Springer Lecture Notes in Artificial Intelligence, vol. 3698, 2005, 366379; also available as http://arxiv.org/abs/qbio.NC/0508032.
 C. Gros, Neural networks with transient state dynamics, New J. Phys., vol. 9, 2007, 109.
 A. Hyvärinen, and E. Oja, Independent component analysis: Algorithms and applications, Neural Networks, vol. 13, 2000, 411430.
 D.R. Chialvo and P. Bak, Learning from mistakes. Neuroscience, vol. 90, 1999, 11371148.
 S. Choi, A. Cichocki, H.M. Park, and S.Y. Lee, Blind Source Separation and Independent Component Analysis: A Review, Neural Information Processing, vol. 6, 2005, 157.
 P. Földiák, Forming sparse representations by local antiHebbian learning, Biological Cybernetics, vol. 64, 1990, 165170.
 N. Butko, and J. Triesch, Learning Sensory Representations with Intrinsic Plasticity, Neurocomputing, vol. 70, 2007, 11301138.