How to build a General Intelligence: Circuits and Pathways

Figure 1: Our headline image is from the Cognitive Consilience: An atlas of key pathways cross-referenced to supporting literature articles. The complexity and variety of routing within the brain can be appreciated with this beautiful illustration. Note in particular the specialisation of cortical cells and the way this affects their interactions with other cells in the cortex and elsewhere in the brain. Explore this fantastic resource yourself.

By David Rawlinson and Gideon Kowadlo

This is part 3 of our series “how to build an artificial general intelligence” (AGI). Part 1 was a theoretical look at General Intelligence (follow the link if you don’t know what General Intelligence is).

We believe that the Thalamo-Cortical system is the origin of General Intelligence in people. In Part 2 we presented very broadly how the Thalamo-Cortical system is structured and organised. We applied some core concepts, such as hierarchy, to help us describe the system.

We also looked at the cellular structure of the Cortex and in particular introduced Pyramidal cells.

This article is again about what we can learn from reverse-engineering the Thalamo-Cortical system, but this time from its connectivity, which we present in terms of circuits and pathways.

Pathways and Circuits

A pathway is a gross pattern of sequential connectivity between brain regions - for example, if part A is highly connected to part B, and activity in A is followed by activity in B, we say there exists a pathway between A and B. Cells in the Thalamo-Cortical system are connected to each other in quite restricted and specific ways, so these pathways are quite informative.

Circuits are more specific and precise details of both connectivity and functional interaction between neurons. In computational neuroscience there exists a concept called the Canonical Cortical Micro-Circuit. The specifics of this circuit are not widely agreed, because (a) the Cortex is complex and (b) many of the evidence-gathering exercises are statistical observations (e.g. “X% of outputs from A and Y% of output from B projects to region C”) which may obscure fundamental functional or topological features. For example, outputs from A and B may project to cells with exclusive roles, but physically co-located in C. Statistical, regional approaches will not capture such distinctions.

In the neuroscience literature there’s a frustrating habit of selectively reporting supporting details while ignoring others. Perhaps this is simply because it's impossible to describe any part exhaustively. In particular, there is a lot of contradictory information about Cortical circuits. But the research can still shed some light on what is happening. Just don’t expect all sources to be consistent or complete!

Key Cortical Pathways

There are several widely-cited and well established cortical pathways (i.e. routes with at least one end in the Cortex). To understand these, it is important to remember both the physical and logical structure of the Cortex as described in the previous article. Physically, the cortex is made of layers, and logically, Columns within the Cortex form a hierarchy.

The hierarchy defines a structure made of Columns, and determines which Columns interact. Pathways describe the patterns of interaction between cells within a Column, and between Columns. We assume that all Columns are functionally identical prior to training.

Cells within Columns are usually identified by both the physical location of cell bodies within particular Layers in the Column, and by the morphology (shape) of the cell. Data flow to and from Cortical cells is largely restricted to a handful of core pathways that begin and terminate in particular cell types in specific cortical layers.

There are many descriptions of cortical pathways and circuits in the literature. We will first introduce just 4 well-established cortical-cortical pathways, and then some thalamo-cortical pathways. Note that although the existence of these pathways is unambiguous, their purpose and function is poorly understood. They appear to be consistent across various somatosensory regions of the cortex, especially in comparison to variations in other brain tissues.

Hawkins’ Hierarchical Temporal Memory (HTM) introduces 3 of the 4 pathways in a single, coherent scheme and relates them to a general intelligence algorithm. We will borrow this terminology and describe them in detail below. Their names describe the direction of data flow and the routing used:

Feed-Forward Direct Pathway: C2/3 → C4 → C2/3
Feed-Forward Indirect Pathway: C5 → Thalamus → C4 → C2/3
Feed-Back Direct Pathway #1: from C6 → C1

We are also interested in a second Feed-Back Direct “pathway” implemented by cortically projecting C6 pyramidal cells with axons that terminate in both C6 and C1 in hierarchically lower regions (see here for a diagram).

Feed-Back Direct Pathway #2: from C6 → C6

Note that cells in all cortical layers (except, perhaps, C4) receive input via their dendrites in C1. In other words, feedback from C6 to C1 is then used as input to many layers. Feedback from C6 to C6 is generally not input for other layers.

In neuroscience, Feed-Forward usually means the flow of data away from external sources such as sensors (towards greater abstraction, if you believe in a cortical hierarchy). Feed-Back means the opposite - data flow towards regions that have direct interaction with external sensors and motors.

Direct pathways are so-called because data is routed directly from one cortical column or region to another, without a stop along the way. Indirect pathways are routed via other structures. The  “Feed-Forward-Indirect” pathway described by Hawkins is routed via the Thalamus.

Figure 2, derived from a Hawkins/Numenta publication, shows graphically how information flows between columns and between layers within columns, as part of these 3 pathways according to the HTM theory. As mentioned before, the community is welcome to contribute by updating and adding to the figure.

Hawkins assigns specific roles to these pathways, but we will be re-interpreting them in the next article.

Figure 2: Routing of 3 core pathways, based on a diagram from the HTM/CLA White Paper. Note the involvement of specific cortical layers with each pathway, and the central role of the Thalamus. The names of the pathways indicate direct (cortex-to-cortex) and indirect (cortex-thalamus-cortex) variants, with direction being either forward (away from external sensors and motors, towards increasing abstraction) or backward (towards more concrete regions dealing with specific sensor/motor input). 

The role of the Thalamus

Let’s recap: The Cortex is composed of Columns, organised into a hierarchy. Cells pass messages directly to other Columns that are higher or lower in the hierarchy. Messages may also be transmitted indirectly between Columns, via the Thalamus.

The Thalamus is often viewed as having a gating or relaying function. The Thalamus is particularly associated with control of attention.

This section will describe indirect pathways involving the Thalamus. Figure 3 is a reproduction of a figure from Sherman and Guillery (2006) that has two new features of interest. These authors use the terminology “first order” to denote cortical regions receiving direct sensor input and “higher order” to denote cortical regions receiving input from “first order” cortical regions. This corresponds with the notion of hierarchy levels 1 and 2.

The Thalamus is a significant part of the “Feed-Forward Indirect” pathway. This pathway originates at Cortex layer 5 and propagates to a nucleus in the Thalamus. There, the nucleus may react by transmitting a (presumably corresponding) signal to one or more other Cortical Columns, in a different region. In some theories of cortical function, the target Column is conceptually “higher” in the hierarchy. The Thalamic input enters the Cortex via Thalamic axons terminating in Cortex layer 4 and is then propagated to Cortex Layer 5 where the pathway begins again.

Figure 3 also shows Cells in Columns in Cortex layer 6 fairly accurately form reciprocal modulatory connections to Thalamic nuclei that provide input to the Column via C4 and C5! Therefore, a Column within the Cortex has influence on data that it receives from the Thalamus. In effect, the Cortex is not a passive recipient but works with the Thalamus to control its input. The figure also depicts C6 cells projecting to C6 in lower regions (our second feedback pathway).

Figure 3: Pathways between cortical columns in different regions, showing layer involvement in each pathway and the role of the Thalamus. Sherman and Guillery use the terminology “first order” to denote cortical regions receiving direct sensor input and “higher order” to denote cortical regions receiving input from lower (e.g. “first order”) cortical regions. This corresponds with the notion of hierarchy levels 1 and 2. Note that in addition to the 3 pathways shown in the previous figure, we see additional direct feedback pathways and reciprocal feedback from Cortex layer 6 to the Thalamic nuclei that stimulate the cortical region. Image source.

Motor output

At this point it is interesting to look at how the Cortex can influence or control behaviour, particularly the generation of motor output. There are two pathways that allow the cortex to influence or control behaviour:

Cortical Control: Basal Ganglia → Thalamus → Cortex → Motors
Cortical Influence: Cortex → Basal Ganglia → Motors

Note that in both cases, the origin of action selection is the Basal Ganglia. In the first case, the Basal Ganglia control signals emitted by the Thalamus, with these signals in turn affecting activity within Cortex layer 5 (C5). C5, particularly in motor areas, has been studied in detail. 10-15% of the cells in these areas are very large pyramidal neurons known as Betz cells, that can be observed to drive muscles very directly with few synapses in between. These cells are more prevalent in primates and are especially important for control of the hands. This makes sense given that manual tasks are typically more complex and require greater dexterity than movements by other parts of the body. The human Cortex is believed to be crucial for innovative and sophisticated manual tasks such as tool-making.

Within the Cortical layers, C5 seems to be uniquely involved in motor output. Figure 4 shows some of the ways output from Pyramidal cells in C5 project output to areas of the brain associated with motor output and control. In contrast, pyramidal cells in C2/3 predominantly project to other areas of the cortex and are not directly involved in control.

Figure 4: Pyramidal cells in C5 project output to areas of the brain associated with motor output and control. In contrast, pyramidal cells in C2/3 predominantly project to other areas of the cortex and are not directly involved in control. Image source.

The second way that the Cortex can influence motor output is via the Basal Ganglia. In this case, we propose that the Cortex might provide contextual information to assist the Basal Ganglia in its direct control outputs, but we found no evidence that the Cortex is able to exert control over the Basal Ganglia.

We suggest Cortical influence over the Basal Ganglia is less interesting from a General Intelligence perspective, because the hierarchical representations formed within the Cortex are not exploited, and execution is performed by more ancient brain systems not associated with General Intelligence qualities.

For the rest of this article series, we will ignore control pathways that do not involve the Cortex,  and will focus on direct control output from Cortex layer 5.

Action Selection

It is widely believed that action selection occurs within the flow of information from Cortex through the Basal Ganglia, a group of deep, centralised brain structures adjacent to the Thalamus. There are a number of theories about how this occurs, but it is generally believed to involve a form of Reinforcement Learning used to select ideas from the options presented by the Cortex, with competitive mechanisms for clean switching and conflict resolution.

A major output of the Basal Ganglia is to the Thalamus; one prevailing theory of this relationship is that the Basal Ganglia controls the gating or filtering function performed by the Thalamus, effectively manipulating the state of the Cortex in consequence. The full loop then becomes Cortex → Basal Ganglia → Thalamus → Cortex (see Wikipedia for a good illustration, or figure 5).

As discussed above, this article will focus on motor output generated directly by the Cortex.

Figure 5: Pathways forming a circuit from Cortex to Basal Ganglia to Thalamus and back to Cortex. Image Source.

Canonical Cortical Circuit

We now have the all the background information needed to define a “Canonical Cortical micro-Circuit” at a cellular level. All the information presented so far has been relatively uncontroversial, but this circuit is definitely our interpretation, not an established fact. However, we will present some evidence to (inconclusively) support our interpretation.

Figure 6: Our interpretation of the canonical cortical micro-circuit. Only a single cortical region or Column is shown. Arrow endings indicate the type of connection - driver, modulator or inhibitor. The numbers 2/3, 4, 5, and 6 refer to specific cortical layers. Each shape represents a set of cells of a particular type, not an individual cell. Self-connections and connections within each set are not shown, but often exist. Shapes T and B refer to Thalamus and Basal Ganglia, not broken down into specific cell layers or types. Data enters the diagram at 4 points, labelled A-D, but does not exit; in general the system forms a circuit not a linear path. Note that shape T occurs twice, because the circuit receives data from only one part of the Thalamus but projects to two areas in forward and backward directions.

Diagram Explanation

We will use variants of the diagram shown in figure 6 to explain our interpretation of cortical function. In this diagram, only a single Cortical region or Column (used interchangeably here) is shown. In later diagrams, we will show 3 hierarchy levels together so the flow of information between hierarchy levels is apparent.

In these diagrams, shapes represent a class of Neurons within a specific Cortical Layer. The numbers 2/3, 4, 5 and 6 refer to the Cortical layers in which these cell classes occur. The shapes labelled T and B refer to the Thalamus and Basal Ganglia (internal cell types and layers are not shown). Arrows on the diagram show the effect of each connection, either driving (providing information or input that causes another cell to become active), modulation (stimulating or inhibiting the activity of a target cell) or inhibition (exclusively inhibiting the activity of a target cell).

If you want more detail on the thalamic end of the thalamocortical circuitry, an excellent source is this paper by Sherman.

There are many interneurons (described in the previous article) that are not shown in this diagram. We chose to omit these because we believe they are integral to the function of a layer of pyramidal cells within a Column, rather than an independent system. Specifically, we suggest that inhibitory interneurons implement local self-organising and local competitive functions (e.g. winner-take-all), ensuring sparse activation of the cell types represented by shapes in our diagram (C2/3, C4, C5, and C6). The self-organising behaviour also ensures that cells within each column optimise coverage of observed input patterns given a finite cell population. Inclusion of the interneurons would clutter the diagram without adding much explanatory value.

We also omit self-connections within a class of cells represented by a shape. These self-connections likely provide context and contribute to learning and exclusive activity within the class, but don’t make it easier to understand circuits in terms of cortical layers and hierarchy levels.

Excitatory Circuit

Figure 7 shows a multilevel version of the cortical circuit, similar to the multi-level figure from Sherman and Guillery (figure 3). We can now understand where the inputs to the circuit come from, in terms of other layers and external Sensors (S) and Motors (M). Note that Motors are driven directly from C5.

Figure 7: The cortical micro-circuit across several levels of Cortex with involvement of Thalamus and Basal Ganglia. The red highlight shows a single excitatory ‘circuit’. See text for details.

The red path in figure 7 shows our excitatory “canonical circuit”: Data flows from the Thalamus to spiny stellate (star-shape in figures) cells in C4 (see source and source), from where it propagates to pyramidal cells in C2/3, and then to pyramidal cells in C5. C6 is known as the multiform layer, but also contains many pyramidal cells of unusual proportions and orientations. C6 cells are driven by C5, and in turn modulate the Thalamus. Note that C6 cells within a region modulate the same Thalamic nuclei that provide input to that region of Cortex.

Inhibitory Circuit

A second, inhibitory circuit exists alongside our excitatory circuit. In addition to providing input to the Cortex via C4, axons from the Thalamus also drive inhibitory Parvalbumin-expressing (PV) neurons in C4 (shown as circles in the diagram). These inhibitory neurons make up a large fraction of all the cells in C4, and inhibit pyramidal cells in C5 (see source or source ).

This means that the input from the Thalamus can be both informative and executive. It is executive in that it actually manipulates the activity of layer 5 within the Cortex, and informative by providing a copy of the signal driving the manipulation to C4. Figure 8 shows our inhibitory circuit. We believe this circuit is of critical importance because it provides a mechanism for the Thalamus to centrally manipulate the state of the Cortex, specifically layer 5 and 6 pyramidal cells. This hypothesis will be expanded in the next article.

Figure 9 catalogues inhibitory cells, notably showing the cells used in our inhibitory circuit.

Figure 8: The inhibitory micro-circuit. The red highlight shows how the Thalamus controls activity in C5 within a Column by activating inhibitory cells in C4. The circuit is completed by C5 pyramidal cells driving C6 cells, which in turn modulate the activity of the same Thalamic nuclei that selectively activates C5. Each shape denotes a population of cells of a specific type within a single Column, excluding ‘T’ and ‘B’ that refer to the Thalamus and Basal Ganglia respectively.

Figure 9: Inhibitory interneurons in the Cortex. Of particular interest are the “PV” cells that are driven by axons from the Thalamus terminating in layer 4 and in turn inhibit pyramidal cells in layer 5. Image source. 

Pathways and the Canonical Circuits

Now let’s look at how pathways emerge from our cortical micro-circuit. Figures 10, 11, 12 show the Feed-Forward Direct, Feed-Forward Indirect and first Feed-Back pathways respectively. We also include another direct, Feed-Back pathway terminating at C6 (figure 13). Feed-back direct pathways terminating at C1, where many fibres are intermingled, are harder to interpret than feedback terminating directly at C6. Pyramidal neurons from many layers have dendrites in C1.

Figure 10: Feed-Forward Direct pathway within our canonical cortical micro-circuit.

Figure 10 highlights the Feed-Forward direct pathway. Signals propagate from C4 to C2/3 and then to C4 in a higher Column. This pattern is repeated up the hierarchy. This pathway is not filtered by the Thalamus or any other central structure. Although activity from C2/3 propagates to C5, it does not ascend the hierarchy via this route: C5 in one Column does not directly connect to C5 in a higher Column, only via an indirect pathway (see below).

Figure 11: Feed-Forward Indirect pathway.

Figure 11 highlights the Feed-Forward Indirect pathway. The Thalamus is involved in this pathway, and may have a gating or filtering effect. Data flows from the Thalamus to C4, to C2/3, to C5 and then to a different Thalamic nucleus that serves as the input gateway to another cortical Column in a different region of the Cortex.

Figure 12: The first of two Feed-Back Direct pathways.

Figure 12 highlights the first type of Feed-Back Direct pathway. This pathway may be more concerned with provision of broader and more abstract (i.e. hierarchically higher) contextual information to be used in the Feed-Forward pathways for better prediction. This suggestion is supported by evidence that axons from C6 via C1 synapse with apical dendrites of pyramidal cells in C2/3, C5 and C6, in hierarchically lower regions.

Figure 13 highlights the second of two Feed-Back Direct pathways. This pathway might be involved in cascading control activity down the hierarchy towards sensors and motors - the next article will expand on this idea. Activity propagates from C6 to C6 directly. C6 modulates the activity of local C5 cells and relevant Thalamic nuclei that drive local C5 cells. Note that connections from a Column to the Thalamus are reciprocal; feedback from C6 to the Thalamus targets the same nuclei that project axons to C4.

Figure 13: The second of two Feed-Back Direct pathways.

Summary

We’ve presented some additional, detailed perspectives on the organisation and function of circuits and pathways within the Thalamo-Cortical system and presented our interpretation of the canonical cortical micro-circuit.

So what’s the point of all this information? What do these circuits and pathways do, and why are they connected this way? How do they work?

It might seem that we’ve stopped short of really trying to interpret all this information and that’s because we are, indeed, holding back. After having spent so much time presenting background information, the next article finally attempts to understand why the thalamocortical system is connected in the ways described here, and how this system might give rise to general intelligence.