By April Cashin-Garbutt

Everything we do in life is compositional. From assembling plastic building blocks as a child, to putting chairs around the dinner table. Our understanding of the world, and the problems it presents, comes from the way we can reuse pieces of information and compose them together in new ways.

“In neuroscience, researchers have mostly focused on how the brain builds models of the world using a single example, such as how you build a model of the room you are in. This ignores the fact that your model of the room is built from building blocks that can be reused to make any other room or scene,” explained Tim Behrens, SWC Group Leader.

In a new paper published in Cell, Behrens and colleagues at University of Oxford, UCL, Beijing Normal University and DeepMind, explored how the brain reuses such building blocks. They found that generative replay in the hippocampus and medial prefrontal cortex underpins such compositional computations.

Using fMRI and MEG to study hippocampus and medial prefrontal cortex

Damage to the medial temporal lobe in humans is known to cause deficits in the ability to solve visual compositional problems and so it was thought that hippocampus was likely involved in our ability to put together pieces of information. However, the exact representations and neural mechanisms were not clear.

Behrens and colleagues designed a rich task where human participants were asked to solve problems involving visual building blocks while inside a functional magnetic resonance imaging (fMRI) or magnetoencephalography (MEG) scanner. The participants were shown a silhouette of a compositional structure and asked how this could be built out of a set of building blocks. They were given a few seconds to solve the compositional problem and then presented with a question to confirm if they had come to the right solution.

The team used fMRI to determine whether they could figure out from the participants’ brain representations of the silhouettes how the compositional structure was made. They tested if the representation of the silhouette stores a plan to reconstruct the structure from the building blocks.

The fMRI recordings confirmed that the frontal cortex does store a representation of how to build the compositional structure. However, the researchers couldn’t look at very rapid patterns of neural activity with fMRI as the technique is constrained by the blood-oxygen-level-dependent (BOLD) response. This is because fMRI measures very local blood flow to parts of the brain, but that blood takes several seconds to arrive there.

To explore this rapid activity, the team used MEG, which is a technique that looks at magnetic fields. This allowed them to observe very rapid transitions that happen in the millisecond range of neural activity. This meant that they could use MEG to understand the mechanism for how the brain completes inferences to solve the compositional problem.

Uncovering the role of generative replay

The researchers found that in the frontal cortex the representation of a whole compositional visual scene is stored as a plan of how to build it. That plan is figured out by rapid sequences of replay that slowly change from the incorrect to the correct plan.

“Replays are sent with candidate solutions for the problem. As you are solving the problem, the candidate solutions get increasingly better as if they are sampling from where you currently are in a solution space,” explained Behrens.

The team found that the computations seemed to be solved by replay as the content of the replay changed over a couple of seconds. They observed the replay start by playing out the wrong answer and then proceed to play out different solutions until it reached the correct one.

From visual plans to general plans

“My hope is that our findings are not just true of visual plans, but also true of general plans. These patterns of activity in the frontal cortex could contain sequences of behaviours that will be useful to achieve a general goal,” commented Behrens.

To explore this hypothesis further, the Behrens lab is working with mouse data in a different task. They are hoping to be able to show that replay is involved in figuring out how to configure those frontal representations to be the correct plan.

The team are also collaborating with neurosurgeons working on intracranial human data. They are looking to measure hippocampal sharp waves directly so they can observe these processes at a finer scale in humans.

Find out more

• Read the full paper in Cell, ‘Generative replay underlies compositional inference in the hippocampal-prefrontal circuit’
• Learn more about research in the Behrens Lab