Three levels of neural reuse through the dynamical lens: structural network, autonomous dynamics and transient dynamics.
The brain in conjunction with the body is able to adapt to new environments and perform multiple behaviors through reuse of neural resources and transfer of existing behavioral traits. Although mechanisms that underlie this ability are not well known, they are largely attributed to neuromodulation. In this work, we demonstrate that an agent can be multifunctional using the same sensory and motor systems across behaviors, in the absence of modulatory mechanisms. Further, we lay out the different levels at which neural reuse can occur through a dynamical filtering of the brain-body-environment system’s operation - structural network, autonomous dynamics, and transient dynamics. Notably, transient dynamics reuse can only be explained by studying the brain-body-environment system as a whole and not just the brain. The multifunctional agent we present here demonstrates neural reuse at all three levels.
Keywords: multifunctionality; neural reuse; neural networks; dynamical systems theory; brain-body-environment systems
Submittted to the Cognitive Science Conference, 2018
Three levels of neural reuse through the dynamical lens: structural network, autonomous dynamics and transient dynamics.
Madhavun Candadai (email@example.com) Cognitive Science Program, Indiana University Bloomington, IN 47406 USA Eduardo Izquierdo (firstname.lastname@example.org) Cognitive Science Program and School of Informatics, Computing, and Engineering, Indiana University Bloomington, IN 47406 USA
A crucial aspect to adaptation in cognitive beings is their ability to exploit regularities in the environment and reuse existing resources/skills across multiple behaviors. Extensive empirical evidence shows that neural resources developed during the course of learning one behavior is reused for others (?, ?). This multi-functional ability of neural circuits has been demonstrated in the seemingly small and simple nervous systems of the nematode worm Caenorhabditis elegans(302 neurons) (?, ?) as well as in the macro scale of the human brain (100 billion neurons) (?, ?). The mechanisms that facilitate this phenomenon have largely been attributed to neuromodulation and synaptic plasticity (?, ?, ?, ?).
The goal of this work is to show a concrete example of how interaction between brain, body and environment enables neural networks to perform multiple behaviors and elucidate the dynamical aspects that lead to it. Reuse in embodied recurrent neural networks unfold over three levels - structural network, autonomous dynamics of the neural network, and transient dynamics of the neural network (Fig. 1). Structure is defined by the neural circuit itself, the intrinsic parameters of the neurons, and the synaptic strength of connectivity between them. While it is possible that an agent possesses specialized circuits for performing different behaviors (Fig. 1A), reuse at this level involves utilizing overlapping circuits to produce multiple behaviors (Fig. 1B,C,D). The next level, when structure is reused, is that of the neural network’s autonomous dynamics isolated from the body. Each behavior is associated with a set of phase-portraits corresponding to the inputs the agent experiences while performing them. The sets of phase-portraits (and the attractors therein) could be overlapping (Fig. 1C,D) or could be unique to each behavior (Fig. 1B). The set of all attractors from all phase-portraits corresponding to a behavior are also referred to as attractor set of the behavior in this paper. The third level of reuse is that of ongoing transient dynamics as the agent is in continuous closed-loop interaction with the environment. When there is attractor reuse from the previous level, it is possible that multiple behaviors navigate different transients around those attractors (Fig. 1C) or, they might be reused too (Fig. 1D). We show for the first time to our knowledge that reuse of transients, namely indistinguishable neural dynamics, can produce starkly distinguishable behaviors in embodied dynamical neural networks. This can only be observed when the brain is studied in conjunction with the closed-loop interaction between the body and the environment.
We believe that this three level approach, where the earlier levels subsume the later ones, provides a systematic framework to study neural reuse in biological systems. Each level could further be expanded into sub-levels. For instance, structural reuse can further check if specific types of neurons are reused, or if network properties such as communities or rich clubs are reused and so on. In this paper, we have explored autonomous and transient dynamics reuse in greater detail.
This section outlines the agent design and its neural controller, the categorization and pole-balancing task set up and the evolutionary optimization algorithm. The agent and environment are continuous time systems, that were simulated by Euler stepping them with a step size of 0.1.
The agent design is identical to (?, ?) being circular with a diameter of 30 units and is equipped with 7 sensory rays radiating from its center, equally distributed over an angle of . The rays constitute the “eye” of the agent and they have a range of 265 units. Each sensory ray feeds into a sensory neuron, and the magnitude of input is inversely proportional to the distance at which that ray is intersected by an object (Fig. 2). The sensory neurons are stateful units that are governed by the following dynamics where . refers to the time-constant which is fixed to be the same across all 7 neurons, refers to the state of the neuron, and to the sensory input received from the corresponding ray. Neuron output is where is the sigmoid function of the state with a gain, , and bias , that are also fixed to be the same across all sensory neurons. The sensory neurons are fully connected to an interneuron layer, made up of a fully recurrently connected continuous-time recurrent neural network (CTRNN). The interneurons are governed by
where the last term refers to the input to interneuron defined as the weighted sum of the outputs of the sensory neurons, , with the weight from the sensory neuron being . Being a fully connected recurrent network, the interneuron dynamics also incorporate the activity of other interneurons weighted by (from to ). The state of the interneurons are denoted by , their gains by (=1) and their bias by . The interneuron network size is denoted by . They project to two motor neurons that take on dynamics similar to the interneurons but they are not connected to each other. Unlike in (?, ?), the motor neurons control the effective acceleration of the agent, a, as follows where refers to the gain, and are the internal state variables for the right and left motor neurons respectively, is the standard sigmoidal activation function and is the common bias term for both motor neurons.
Categorization task design
The categorization task involves differentiating between the falling objects (circles and lines) by moving towards one (circles) and away from the other (lines) (?, ?). The circles’ diameter and the line’s length were both set at 30 units. To encourage generalization, each evaluation of the agent’s performance was conducted over 8 different trials for each type of object, with the objects’ starting horizontal offset from the agent uniformly distributed in the range . The objects fall with a constant velocity of 0.3 units per second. Performance in this task was quantified by averaging over for the circle trials and for the line trials; where is the normalized distance between the center of the agent and the center of the object when the vertical offset between the agent and the object reaches 0 (offset of over 45 units was clipped at 45).
Pole-balancing task design
The second task, pole-balancing, was inspired by the first implementation in (?, ?). It was re-designed such that the agent has the pole attached to its center and can sense it through the same rays used for sensing the falling objects (?, ?). The sensory input, as the pole sw.pdf across a ray at angle , increases linearly from 0 at reaching the maximum value at and falling back to 0 at and vice verse. Note that the agent senses the pole only when it intersects with a ray but it “disappears” from view while passing between rays. The pole Was considered dropped if it fell beyond the rays, or if the agent moved over 45 units on either side from its starting position. Performance was calculated by averaging at each time step, where is the pole angle with the vertical, over the 500s of evaluation duration. Just like the categorization task, to promote generalization of the behavior, performance was averaged over 16 trials with the pole starting from 4 different angles on either side of the agent in with angular velocity -0.1 or 0.1.
An evolutionary search algorithm was used to optimize the parameters of the agent - time-constant, gain and bias for sensory neurons (), weights from sensory layer to N-interneurons (), recurrent weights between interneurons (), bias and time-constant for each interneuron (), weights from interneurons to motor neurons () and gain, bias and time-constant for motor neurons (): totaling parameters. The search starts with a random population of 100 solutions encoded as -dimensional genotype vectors with each element in [-1,1]. These elements were scaled and mapped on to the different parameters to build the agents. Gains are scaled to be in [1,20], time constants in [1,2], biases in [-4,4] and all weights are scaled to be in [-5,5]. These agents were then evaluated for their performance (fitness). Based on fitness, an elitist fraction of the top 4% solutions were retained while their copies were subject to a Gaussian mutation noise with mean 0 and variance 0.3 to produce a new population of 100 solutions, and this process was repeated for a fixed number of generations.
Since optimization is stochastic, 100 independent runs were carried out for the single and multi-task scenarios. For the individual tasks, optimization was carried out over 1000 generations in each run. In the multi-task setting, fitness was computed as the product of fitness in each task. These experiments were conducted in three different task presentation paradigms: (1) evolve for both categorization and pole-balancing over 2000 generations, (2) evolve only for pole-balancing for the first 500 generations, and then evolve for both tasks for 1500 generations and (3) evolve for categorization for the first 1000 generations, and then for both tasks for another 1000. The 500 generation limit for paradigm 2 and 1000 for 3 was based on number of generations taken to acquire good performance in each task when optimized individually. The product of the individual task fitness was used as opposed to sum or average because it guarantees good performance in both tasks, while still keeping the fitness in .
Minimal neural resource requirement for each task
In order to evaluate the level of reuse in the multifunctional network, we first systematically explored the minimal resources required to solve each task individually. We used an evolutionary optimization technique to train networks of different sizes for both tasks: pole-balancing and categorization. 100 independent evolutionary runs were performed for networks of different sizes for each task. The smallest network that could perform pole-balancing had two interneurons. The best of these agents achieved a 98.44% fitness and was able to move the pole to its upright position from a broad range of initial positions and keep it balanced for an extended duration of time (Fig. 3A). The smallest network that could perform the categorization task also had two interneurons. The best of these agents had a fitness of 98.5% and was able to successfully catch all circles and avoid all lines falling from the full range of starting positions (Fig. 3D).
Effectively addressing multifunctionality using these two asks is only possible if the two tasks involve sufficiently different behaviors: solving one task, at most, only helps partially solve the second, and vice verse. In order to test this, all agents that have been optimized to perform one task were evaluated on the other. Agents that were trained to balance the pole were as good as random agents at the categorization task (Fig. 3B,E), while agents that were trained to categorize could balance the pole only slightly better than random controllers (Fig. 3C,E). This suggests each task requires its own unique set of sensory-motor skills and that ultimately solving one task does not guarantee good performance in the other task.
Structural network reuse: Fully overlapping circuits were used to perform both behaviors.
The highest level of reuse is that of structure - an agent performing more than one behavior could acquire specialized circuitry to perform each behavior or could share neural circuits between them. In order to test this, using the same evolutionary optimization approach described previously we evolved networks of different sizes to perform both behaviors.
Interestingly, agents with networks no larger than the ones that could solve the individual tasks could also solve both tasks. The best 2-neuron multifunctional agent could perform categorization with a fitness of 95.8% and pole-balancing with a fitness of 95.4%. This agent used the same circuit to successfully catch circles while avoiding lines and also balance a pole (Fig. 4). It is to be noted that the agent had no external signal indicating which task to solve. Furthermore, the agent also had no synaptic plasticity or neuromodulatory signal that could be responsible for re-configuring the circuit for the different tasks. Since performing each behavior individually required at least 2 neurons, it follows that the 2-neuron multifunctional agent shared the same structural resources.
Autonomous dynamics reuse: Overlapping sets of attractors were used to perform both behaviors.
Given that the circuit is the same across the two behaviors of categorization and pole-balancing, going one level deeper into its autonomous dynamics, we evaluated reuse at three levels of increasing specificity - attractor composition, basins of attraction and location of attractors in the sets of phase-portraits associated with the different behaviors. A phase-portrait is the dynamical landscape of the neural network upon subjecting it to a fixed input under different initial conditions. A phase-portrait for a particular input, could have one or more attractors. The set of phase-portraits associated with a behavior (say circle catching) can be obtained by fixing the inputs to what the agent experiences during that behavior (circle at fixed positions relative to the agent), and allowing the network to settle into its attractors from different initial states. The three sets of phase-portraits corresponding to circle catching, line avoiding and pole-balancing were compared on the above mentioned levels to evaluate reuse.
All phase-portraits associated with all three behaviors in the best multifunctional agent were composed of one fixed-point attractor, thus showing reuse in the highest level of autonomous dynamics, namely attractor composition. This denotes that the set of phase-portraits of one behavior do not have to undergo bifurcation to lead to the set of phase-portraits of the other. Even though the phase-portraits are qualitatively matching across behaviors, they could have different basins of attractions, which could lead to different behaviors operating in its own region of the phase-space (dynamical modularity). In this case, since only one fixed-point attractor exists in each phase-portrait, there exists only one basin of attraction in each of them and hence is the same across all behaviors.
Even with shared phase-portraits and basins of attractions, the locations of the attractors from phase-portraits of different behaviors do not have to be the same. Differentiated by their location, each behavior could have a unique set of attractors or they could overlap to different extents. Upon analyzing their locations we discovered that, even quantitatively, the multifunctional agent reused its attractors between these behaviors (Fig. 5). This reuse was only partial since each behavior also had its own set of unique attractors that were not shared. This reuse means that different inputs from different behaviors were mapped to the same phase-portrait suggesting an inherent degeneracy between the sensory inputs and the requisite behavioral pattern for the three behaviors. In fact, agents may evolve to take advantage of such a degeneracy, where two distinct perceptual scenarios might require similar interpretations by the interneurons or indeed a similar action, and hence are mapped to the same phase-portrait.
Transient dynamics reuse: Phases of the behaviors reused the same transient dynamics.
Our analysis started at the level of structural reuse and went on to discover reuse at the level of autonomous dynamics in the best multifunctional agent. The next level is that of ongoing dynamics as the nervous system coupled with its body and environment performs the behaviors. Note that in the previous level, attractors were identified by fixing the relative position of the agent with the object and then allowing the network to settle. However, during behavior, both the agent and the object are in constant movement. Therefore, at any given time, for a particular relative positioning of the agent and the object, the sensory input might change before the network settles into the attractor associated with that fixed input. As a result, dynamics of the network are in constant transient movement across the phase-portraits (and the attractors therein) associated with that behavior. Therefore, at this level, the relevant question that explores reuse is whether multiple behaviors have their own unique transient trajectory or if they could possibly share them.
Transient dynamics were shared partially between the circle-catching and pole-balancing tasks in the best multifunctional agent. In order to explore this, we looked at the entire sensorimotor loop as this agent performed both behaviors. While in each case there were times when their transients were different, for a particular phase during these behaviors, the dynamics almost exactly matched (Fig. 6A). Interestingly, inputs to the interneurons, their outputs, and motor neuron outputs were all identical suggesting that the agent’s nervous system does not differentiate between these phases of the two behaviors. This leads to two interesting questions (1) Are the two behaviors indistinguishable during this phase? (2) If not, where does the difference come from?
Although the dynamics in the inter-neurons are identical, the behaviors were different in the circle-catching and pole-balancing tasks. In the former, the agent was receiving oscillatory inputs only along ray 1, meaning that the agent oriented itself so as to track the circle along that ray before catching it (Fig. 6B). In contrast, the sensory inputs during the same transient dynamics in the pole-balancer shows that during this phase of the behavior, the agent mostly maintains the pole oscillating around ray 3 while bring it around across rays 4,5 and 6. This is an interesting outcome since it demonstrates that neural activity that is completely indistinguishable to the level of transient dynamics could still produce behaviors starkly different from each other. The difference arises from the parts of the behavior leading up to this shared transient phase, where the agent has its own unique dynamics for each behavior through its interaction with the environment. Note that the weights to the interneurons from sensory rays 1 and 3 are not the same. This transient reuse emerges purely from the recurrent and dynamical properties of the inter-neurons.
The work presented here builds on (?, ?), (?, ?) and (?, ?) by developing a computational model of a brain-body-environment system that performs multiple behaviors using the same sensory and motor capacities. In the work by Izquierdo et. al., the same neural network without any changes in parameters or mechanisms of modulation was shown to perform two qualitatively different behaviors while placed in two different bodies. Williams et. al. showed that when different motor systems are used for different tasks, the qualitative difference in environmental feedback drives the same network differently to produce different behaviors. Agmon et. al. presented a model where different sensory apparatus in the agent, sensitive to different stimuli in the environment, performed different associated behaviors using the same motor control systems. In these models, although the neural network remained the same, the body was changing. The model presented in this paper, used the same sensory and motor control mechanisms for the two tasks - object categorization and pole-balancing. We also show through dynamical analysis that reuse to the level of transient dynamics can be observed when the brain, body, environment and their interaction are taken into account.
To summarize, we first evolved embodied recurrent neural networks to perform object categorization, pole-balancing and then both. We then systematically explored the different levels of reuse discovering reuse of neural circuits at the structural level, followed by reuse of autonomous dynamics with qualitative sharing of phase-portraits, overlapping basins of attraction and reuse of attractors identical in the location of their fixed-points. Furthermore, we discovered reuse of transient dynamics in the best multifunctional agent. The two main contributions of our work are (1) the same neural circuit can perform multiple behaviors using the same sensory and motor systems in the absence of explicit task identifying signals or processes such as neuromodulation (2) indistinguishable neural activity, displaying reuse to the level of transients, can still produce completely different behaviors.
The rationale behind transient reuse in this multifunctional agent and can be explained by observing the environment-body relationship and transient dynamics. The similarity in dynamics arises out of the agent generalizing between the two behaviors by learning to align an object along a single ray - pole along center ray and circle along corner ray (Fig. 4 and 6). Generalization requires learning to use only one ray because the pole only intersects one ray at a time. The circle had to be balanced along the corner ray because otherwise it would intersect multiple rays and the pole needs to be balanced along the center ray to maximize fitness.
The disparity in behavior, however, arises out of the unique transient dynamics prior to shared transient phase of the behaviors. The unique dynamics in circle-catching orients the falling object along the corner ray even if the circle starts from the center, whereas in pole-balancing it brings the pole to the center, thereby setting up the system to perform the generalized oscillatory “object tracking along a single ray” for both behaviors (Fig. 4B). This is possible because of the structure provided by the environment and the body. Objects intersect only one ray or multiple rays, yet the agent is required to align with the object in both cases. Thus, multifunctionality in this agent is enabled by the closed-loop interaction between the body, environment and a nervous system that affords reuse at all levels of dynamics.
The three-level framework presented here is rooted in dynamical systems theory and has the potential to explain reuse in any model or biological system. While an information theoretic approach would also be able to explain reuse in analogous levels, the key point to be noted is that there are levels of reuse that can be explained only by looking at the brain-body-environment system as a whole, and not just the brain (the neural network). The model presented in this paper, while using only a small neural network for two seemingly simple tasks of categorization and adaptive control, raises several cognitively interesting questions. The analogous nature of these two tasks in catching and balancing led to reuse here. Are specific environmental structures associated with reuse at different levels? Since the interneurons was limited to two, reuse was forced in this case. Does reuse always happen whenever possible? Can the same set of behaviors be performed with different levels of reuse? Transient reuse shown in our model hints at embodied neural network modeling approaches to higher level cognitive phenomena such as analogy making and empathy.
- Agmon Beer.Agmon Beer. Agmon, E., Beer., R. D. (2014). The evolution and analysis of action switching in embodied agents. Adaptive Behavior, 22(1), 3-20.
- AndersonAnderson Anderson, M. L. (2010). Neural reuse: A fundamental organizational principle of the brain. Behavioral and brain sciences, 33(4), 245-266.
- Barto, Sutton, Anderson.Barto et al. Barto, A., Sutton, R., Anderson., C. (1984). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics., 5, 834-846.
- BeerBeer Beer, R. D. (1996). Toward the evolution of dynamical neural networks for minimally cognitive behavior. From animals to animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, 421-429.
- Briggman Jr.Briggman Jr. Briggman, K. L., Jr., W. B. K. (2008). Multifunctional pattern-generating circuits. Annual Review of Neuroscience, 271-294.
- Candadai Izquierdo.Candadai Izquierdo. Candadai, M., Izquierdo., E. (2017). Information bottleneck in control tasks with recurrent spiking neural networks. arXiv preprint arXiv:1706.01831..
- GettingGetting Getting, P. A. (1989). Emerging principles governing the operation of neural networks. Annual Review of Neuroscience, 185-204.
- HobertHobert Hobert, O. (2003). Behavioral plasticity in c. elegans: paradigms, circuits, genes. Developmental Neurobiology, 203-223.
- Izquierdo BÃ¼hrmannIzquierdo BÃ¼hrmann Izquierdo, E., BÃ¼hrmann, T. (2008). Analysis of a dynamical recurrent neural network evolved for two qualitatively different tasks: Walking and chemotaxis. ALIFE., 257-264.
- Lizier, Heinzle, Horstmann, Haynes, ProkopenkoLizier et al. Lizier, J. T., Heinzle, J., Horstmann, A., Haynes, J. D., Prokopenko, M. (2011). Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fmri connectivity. Journal of computational neuroscience, 30(1), 85-107.
- Morton H.J.ChielMorton H.J.Chiel Morton, D., H.J.Chiel. (1994). Neural architectures for adaptive behavior. Trends in Neurosciences, 413-420.
- Williams Beer.Williams Beer. Williams, P., Beer., R. (2013). Environmental feedback drives multiple behaviors from the same neural circuit. Advances in Artificial Life, ECAL, 268-275.