By Philip Shamash

Hardwired through millions of years of evolution, instinctive behaviours are powerful tools for survival. In fact, many of the most striking animal behaviours do not involve learning. Young birds can migrate thousands of miles to a wintering site for the first time without any guidance. Spiders know how to spin intricate webs as soon as they’ve hatched. Vervet monkeys can alert their troop to different types of predators using a set of innate alarm calls. These elaborate actions, however, share one crucial limitation: they are inflexible. Genetically encoded behaviours cannot deal with situations that are unique to any particular individual’s experience.

Fortunately, animals can also learn about the world. Many species have the capacity to build up knowledge about their environment and use this knowledge to come up with new solutions to their problems. This is perhaps the most celebrated mental process in neuroscience, as it underlies cherished human abilities such as inventing tools and using language.

In systems neuroscience, this genre of learning is often investigated in a simpler context: rodents exhibiting a map-like understanding of familiar surroundings. These animals are able to plan out paths that they have never taken before by using mental representations of the external environment. Studies of rats and mice allow us to extend tight experimental control and home in on the behavioural strategies and neural computations guiding flexible spatial reasoning.

This distinction between rigid instincts and higher cognition is central in the field of systems neuroscience. However, it overlooks a key link: instinct is the only tool available for inexperienced individuals trying to build up knowledge in the first place. Even the most advanced feats of human intelligence initially depend on innate behaviours to catalyse learning. For example, babies pick up the nuances of facial communication thanks to an inborn drive to examine face-like structures; infants learn to speak during an innate series of stages of experimental sound production (e.g. babbling); and children become masterful in tool use after years spent following an instinctive urge to play with small objects. In each case, the path to intelligent behaviour is initiated by a set of instinctive propensities and some basic learning rules.

In the mouse — a vulnerable prey species with a three-week juvenile period — the learning process must occur much more rapidly than in humans, making it more amenable to scientific study. Released into a new environment, young mice immediately search for crevices to hide in, test out routes to safety, and inspect any objects that could obstruct their path. If any sign of threat should appear, they use what they have learned to quickly find their way to safety. But what information do they extract from the environment while exploring it? What innate propensities and learning rules allow them to acquire this knowledge? And how long does it take them to develop flexible spatial reasoning? My colleagues in the Branco Lab at the Sainsbury Wellcome Centre and I have addressed these questions in a recent preprint using a series of carefully controlled behavioural experiments, high-resolution quantification of behaviour, and theoretical modelling.

We found that within 20 minutes of entering a new environment for the first time, mice could expertly navigate within it as if they had a mental map of its spatial layout. However, this ability developed in an unexpected way. Animals are commonly thought to learn by observing unfamiliar features in the environment and incorporating them into an abstract internal representation. We found, however, that mere observation did not improve their ability to navigate. Instead, mice exhibited an innate propensity to run toward the edges of objects that divide up an environment, i.e. obstacles. Only after targeting an obstacle edge with one of these running movements did mice gain the insight to chop up subsequent routes into two stages: first directly targeting the edge location — the subgoal — and from there to final goal, such as a source of food or safety.

This case study on mouse navigation represents a step toward understanding how instinct and learning unite to produce intelligent behaviour. Subgoal memorisation, a new explanation for how animals implement multi-step behaviours, has piqued the interest of artificial intelligence researchers looking to animal behaviour for inspiration. In addition, these findings have applications for our understanding of the brain. The mammalian forebrain, involved in higher cognition, and the midbrain, a centre for instinctive responses, are tightly interconnected, but it is not known why. Investigating the neural basis of this behaviour is well poised to help us to understand how the forebrain controls flexible reasoning through interactions with the more ancient brain structures that originally evolved to control instincts.