Science of Chess: Gestalt Processing in Experts' Brains

In my last Science of Chess post, I started describing some of what we know about the cortical organization of chess from a combination of fMRI experiments and direct stimulation of neural tissue. One of the ideas I brought up in that piece was that understanding how complex functions are supported by different parts of the brain isn't just an exercise in putting together something like a map of where various behaviors "live" in the brain. Instead, I suggested that something else we can gain from studying the cognitive neuroscience of chess (or anything else) are insights into the nature of the complex function we're examining. That is, can we find out what chess is cognitively by determining its neural basis? In the study I want to tell you about here, I think the researchers do a nice job doing something along these lines, so I thought it was a neat set of experiments and analyses to tell you about. Specifically, I think this study helps shed some light on words that comes up a lot in characterizations of expert chess players and their abilities: "chunks" and "chunking."

What does "chunking" mean?

In general, these words (and their closely related cousin "pattern") refer to the grouping of chess pieces, their position on the board, and perhaps some sequences of moves, into a larger-scale configuration. This broad definition (which I imagine some folks may take issue with!) admits a lot of ways to instantiate chunks/patterns, any of which may be more or less relevant to your own subjective experience of surveying a chessboard and organizing what you see there. Here's just one example of a verbal chunk that I bet you find pretty evocative: "Castled King with fianchettoed bishop." My hunch is that this immediately brought a structure like this to mind:

Not too shabby, really - just 5 words conveyed the identity of 6 pieces and the position of each one on the board. I'm not going to try and count bits or anything, but I hope this strikes you as a fairly efficient way to describe part of a complex position. Want another one? If I say "Greek gift" to you, you probably have a sense of both some structure on the board AND some dynamics. Maybe something like this...

This is actually sort of an interesting chunk to think about a little more deeply. I doubt that everyone reading this actually conjured up this entire position immediately after seeing the words "Greek Gift" up above. Instead, I imagine that what may have come to mind was something that included a castled King with an opposing Bishop aiming at the critical h7 square, perhaps with the supporting Knight and Queen in place. If you're a better player than I am, you might also have included some positional information regarding pieces that can spoil the Greek gift for the attacking side! Regardless, what I think is neat here is that these two words have the potential to provoke another fairly rich board position that includes some key structures, but key structures that also accommodate the utility of the sequence of moves beginning with the Bishop capturing the h-pawn.

So look, chunks, patterns, whatever your favorite terminology is for this phenomenon - they seem to be a very useful cognitive tool for organizing and simplifying your perception and memory for chess positions. But what are they? Are they verbal? Are they visual? Are they spatial? To what extent does this process separate players according to ability? There are at least a handful of studies that address these questions (or variations of them) through purely behavioral means, but this is a case where I think a cog. neuro. approach pays some dividends too. With all this in mind, let's see what the researchers behind our target article did and what it might reveal about the neural basis of expert chess play.

Identifying a target node in the cortical chess network

I want to start by retracing some of the steps I took in my last post: Where do we find chess-related activity in the brain using fMRI? Answering this question generally requires us to put participants into a scanner and then ask them to carry out (1) some kind of task that involves chess playing and (2) at least one other task that we will use in a contrast. That is, what we will get out of our fMRI data is a map of the brain, divided up into volume elements called voxels, with a statistical value assigned to each voxel that tells us the extent to which there was significantly more activity in that voxel during the chess condition compared to the other condition. What you get from such an analysis will depend a great deal on what you choose for your chess task and what you choose for your comparison condition, but chess is a sufficiently complex task that it's very likely you will end up with not just one area of interest, but a network of areas that are more active when playing chess than not. I showed you one look at a network like this in the previous article, but here's another from a two-paper series including parallel analyses of the network supporting chess and the network supporting Go. The letters G, B, and R in the figure below refer to the different tasks they asked participants to do in the scanner: "G" refers to a "best-move" judgment given a typical middlegame position, "B" refers to passive viewing of an empty chessboard, and "R" refers to passive viewing of a chessboard accompanied by pieces at random locations. For my money, the G-R comparison is the most interesting to look at because more of the visual contents of the images participants saw in the two conditions are matched. This means that any parts of the brain that are active in response to the presence of the chess pieces will be "subtracted out" in the comparison between conditions, hopefully leaving us mostly with brain areas that are more active when you're making a decision about how to move.

Figure 2 from Atherton et al., 2013a - Contrast between a "best-move" condition and two different control conditions are presented in the top and bottom rows.

What are those areas? Without getting too precise (for now), you can see that we have some active regions in both the parietal lobe and the temporal lobe, consistent with the hypothesis that playing chess probably depends on multiple processes. But now the question shifts from "What areas?" to "Which processes?" and here is where I think things start to get more interesting in most fMRI studies. Once you have a way of identifying part(s) of the brain that you'd like to investigate further, you can ask participants to do many different tasks while you measure the activity inside each region you're interested in. By examining the factors that affect activity in your target area, you can hopefully start to build up some intuitions regarding that area's function, which you can then start linking to other studies and other manipulations of both the observer, the stimuli, and the tasks you might choose to investigate next.

The thing that I think is neat about the paper that is the main focus of this article is that they adopt this strategy by pooling the results from several different studies to give us insights about a particular part of the extended chess network called the Temporal-Parietal Junction (or TPJ). This part of the brain is an interesting target for their research for a few different reasons. First, it makes an appearance in that "G-R" comparison up above, which suggests that this is indeed a part of the brain that participates in determining the best move for a given chess position. Second, and perhaps more importantly, we already have some ideas about what this part of the brain does! This is particularly nice because it helps us start to link the cognitive processing involved in playing chess to other tasks that we understand fairly well. That kind of bridge-building is an important way we can use neural data to understand the nature of complex behaviors and competencies: What sub-processes are engaged during the complex task in question?

Putting visual elements together via Gestalt principles

In the case of the TPJ, we know from studies of neuropsychological patients that this part of the brain contributes to something called perceptual organization. This refers to the process by which your visual system uses small-scale measurements from early stages of vision (stuff like spots of color/brightness contrast or edges) as the basis for making inferences about larger-scale objects and surfaces in your visual world. This is a highly ambiguous problem for your visual system to solve, but arguably also a highly necessary one. Your visual world is filled with a lot of clutter and noise and the large set of measurements you end up with after early stages of the visual system have their turn to describe what's in a scene is frankly kind of a mess. Before you can start recognizing objects or textures (or so the reasoning goes) you need to work out how to separate parts of the image that don't belong together while ensuring that you group parts of the image that do. To convince you that this is something you do, here's a classic two-tone image that you might have seen before that is only interpretable after you've done some perceptual organization to segment and group parts of the image appropriately.

Did you see it? If so, congratulations - you successfully organized the various black splotches and small edges and corners into a Dalmatian. If you didn't see it, I hope the hint that it is indeed a Dalmatian that you're on the lookout for helps a little, but I'm happy to give pointers in the forum later if you need some help spotting it. The point here is that this kind of thing: Working out the right way to bind some parts of an image together while not admitting other parts of the image to the same party is what we think the TPJ plays a crucial role in doing. How does it do so? An important set of ideas in vision science is that your visual system may use a collection of heuristics - rules of thumb for perceptual organization - to try and make inroads on a complex image like that hidden Dalmatian up above. The graphic I used as the title card for this article illustrates a number of these Gestalt Principles, each of which describes a property that parts of an image may have relative to one another that may lead you to group those image elements together into a larger whole. They include things like symmetry: If parts of a scene are symmetrical about some axis, then both those parts of the scene and that axis may be worth linking together into a proto-object. Good continuation is another one: Image elements that are relatable by virtue of smoothness, like edges that line up with one another enough to suggest a continuous line or curve, should also probably be linked together. We don't need to spell all of these out here, of course - suffice it to say that perceptual organization as implemented by the TPJ may depend on some neural version of these heuristics. How do we know that the TPJ is doing this? Because when it is damaged, perceptual organization can go terribly wrong.

Consider the picture below. It's quite messy, because the artist has superimposed several line drawings of simple objects on top of one another. Chances are that you are not fooled by this however - you are probably more than able to use some of the aforementioned Gestalt principles (or other types of perceptual organization) to recover the hammer, the cleaver, the vase, and the press that are part of this compound image. This is not what a patient with damage to the TPJ is likely to see when presented with an image like this, however. Instead, they may experience an impairment called simultagnosia. This is a condition in which patients are unable to perceive more than one object at a time, especially under conditions like this that require a great deal of grouping and segmentation. This specific impairment of visual function provides some of our strongest evidence regarding what visual work the TPJ is involved in and this is what makes it an interesting candidate for a more thorough examination of the extended chess network. What is TPJ doing there? Does the activity we see in this region in that "G-R" contrast way up above mean that perceptual organization of some kind, grouping and segmenting chess pieces or fragments of the board, is a key sub-process in playing the game?

Is TPJ activity (and thus, "chunking" via perceptual organization) a marker of chess expertise?

That question is what the authors of the target article set out to answer and there are a bunch of things about what they did that I quite like. First, let me lay out the overarching strategy they use for synthesizing the results of multiple different experiments comparing Expert chess players' brain activity to that of Novices. The big idea here is to use a non-chess task as a means of identifying the TPJ functionally, followed by independent assessments of how the TPJ responds to different chess-related tasks in a group of Expert players relative to a group of Novice players. The first part of the analysis, finding the TPJ without using anything related to chess, is what we might call a localizer. In general, this refers to a set of experimental conditions that we use solely to identify the parts of the brain (the specific voxels in our fMRI measurements of brain activity) we will focus on later. Using a localizer that is distinct from the scans and/or experimental conditions we will use to compare activity across groups or across stimuli and tasks is an important way to avoid circular reasoning in fMRI analysis. Here, I think it's especially powerful because the task the authors used to identify the TPJ seems quite unrelated to chess, making it all the more interesting if we find differences here as a function of chess expertise.

You can see that localizer task in the panel marked "A" in the omnibus figure below. This panel shows you different versions of a visual task in which a large-scale shape is made up of small-scale shapes. The large circle at right, for example, is made up of many small circles, while the large, tilted square in the middle is composed of small squares. You can probably tell that neither of these shapes is perfect, however: there are little irregularities in some of the elements that disrupt the uniform appearance of the large shape. To be specific, in both of these images, one-fifth (or 20%) of the elements in each image are disrupted. This turns out to be enough for you to still see the large shape, but a larger proportion of erroneous elements makes it much worse - take a look at that 80% figure at the far right of Panel A, for example. There are still lots of small tilted squares in there, but they aren't arranged in a coherent enough way for you to see the large figure.

A section of Figure 1 from Rennig et al. (2013) depicting their global shape perception conditions used to localize TPJ.

Their idea (in one of their previous studies included in this article's synthesis) was that a comparison of neural activity based on this manipulation of the global coherence of their stimuli might reveal the TPJ. Specifically, determining which voxels were more active in the 20% condition than in the 80% condition might reveal the parts of the brain that participate in organizing those small shapes into something larger according to principles of perceptual organization. Lo and behold, that very contrast yields two regions of interest (or ROIs) - one in the left hemisphere, and a matching one in the right.

Figure 2 from Rennig et al. (2013) in which we can see the ROIs in the left and right hemisphere resulting from a 20%-80% image coherence contrast as described above.

Now armed with these brain regions, which we identified without any reference to chess (!!), the authors can now start asking questions about chess expertise. Remember at this point the idea is to stick with these voxels, which we know do a specific kind of visual work (binding small parts of a scene together into larger wholes), and see if they respond differently in Expert players vs. Novices when they consider chess positions in different ways. The authors consolidate data from 4 previous studies in which they asked Experts and Novices to do a bunch of different chess-related tasks in the magnet, depicted below in the rest of their omnibus design figure from the manuscript.

The remainder of Figure 1 from Rennig et al. (2013) depicting a ton of different chess-related tasks presented to Experts and Novices in separate studies predating this analysis.

As you can see, there is a lot of different stuff here. In some cases, we're primarily presenting participants with examples of chess positions in canonical (upright) orientations or inverted orientations, presumably to measure how specific chess-related activity is to the visual arrangement players typically see. In other cases, the authors are asking participants to evaluate the presence of checking attacks, or count the number of threats on the board. In every one of these original experiments, there are also control conditions intended to serve as a means of separating more complex processes related to game play from more general visual and cognitive mechanisms. What's with the faces in the first task, for example? Faces are known to be a stimulus class that's processed far more effectively when upright than inverted (something that is not universally true), so this gives us a nice benchmark for any differences we see in upright and inverted processing of chessboards by Expert players and Novices. Suppose Experts had a much bigger effect of board orientation than Novices - is that specific to chess, or some reflection of a broader difference between these groups of players? If it's the latter, we should also expect to see an Expert/Novice difference when they see faces. In the other studies you can see that varying the complexity of the task participants are asked to complete with the chessboard serves as a means of control - including conditions where you simply count the number of Knights or Bishops, or work with the scrambled board configurations we've seen in previous Science of Chess posts, each offer a chance to see how constrained functional differences related to chess expertise are. Are Expert brains just generically responsive to chessboards, regardless of what they're doing with the pieces? Or rather, do Expert brains (and Expert TPJ regions in particular) exhibit more activity for complex chess judgments only, suggesting a specific role for the TPJ and perceptual organization in high-level play?

What follows in the manuscript are a whole bunch of bar graphs like the one below. In this case, the authors are showing off how much brain activity they measured in the different TPJ ROIs (how's that for some acronym density?) using data from the task that included upright and inverted chessboards and faces. Specifically, the y-axis here is something called percent signal change, which refers to how much stronger the signal we measure with fMRI was in each condition (higher numbers mean more neural activity). What we're looking for here is the following: Do Expert TPJs exhibit more activity than Novices'? If so, is that difference specific to chess or something that we see across both chess and faces?

Adapted from Rennig et al.'s (2013) Figure 3, in which percent signal change in the left TPJ is compared across conditions in their Experiment 1.

To walk you through what you're seeing the key thing to notice is that those bars labeled "Expert" are a LOT higher on the left side of the figure, where the authors are showing us what they measured when participants saw chessboards. By itself, that's potentially kinda neat, but makes you wonder immediately if there is something else contributing to the group difference: Are these Experts different in some very general way from the Novices, for example, or were they tested under slightly different conditions? Such concerns are alleviated by the right side of the figure, however, in which the difference between Experts and Novices essentially vanishes when both groups are looking at faces. What this means is that there is not some general difference in TPJ activity between these groups, but instead, a difference that is pretty large and easily measurable when both groups are looking at chessboards*.*

I'll spare you the full rundown of bar graphs, but there is a similar story to be told when we look at the data from their Experiment 4, in which participants either counted the number of threats against White/Black in a position , counted the Knights/Bishops in the position, or simply counted ALL of the pieces on the board. Again, look for where we do and don't see those differences between Experts and Novices percent signal change.

Adapted from Rennig et al.'s (2013) Figure 6, in which percent signal change in the left TPJ is compared across conditions in their Experiment 4.

Much like in the first experiment, Experts have much more activity than novices when counting threats and when counting minor pieces, but NOT all the time. Counting up all of the pieces (a task that critically lacks any need to identify any of the pieces on the board at all!) erases the Expert advantage for percent signal change, suggesting (again) that TPJ is more active in Experts when they're doing chess-related processing but not otherwise.

So what's a chunk again?

Where exactly does this leave us with regard to understanding the chess network in the brain of a human player? What I'd say we learned from all this is something interesting and pretty specific about what separates Experts from Novices in terms of visual cognition. By identifying the TPJ as a place to look for differences in brain activity related to expertise, these researchers were really testing a hypothesis about something that makes an expert an expert: Does perceptual organization play a role in expert chess play? This matters because we know a good bit about what perceptual organization is and how it works from many, many vision science studies, meaning that the authors' affirmative answer to this question links chess ability (partly) to a bunch of visual processes we feel like we understand well. I'd argue that this helps us start thinking about what exactly the chunks and patterns involved in playing chess well might be: A "chunk" could be some intermediate-to-large scale segment of the board, bound together according to Gestalt-like rules for grouping and segmentation.

This is potentially a powerful idea and immediately makes it easy to start asking questions about how the Gestalt principles apply to the patterns of pieces we see over-the-board. Do principles like symmetry, closure, and figure-ground play a role in how chess positions are perceived? Does something like good continuation operate over arrangements of pieces (e.g. pawn chains) or over directions pieces might go (e.g., the arrows players like Hikaru draw on the board while playing). Perhaps more importantly, how is a chessboard segmented into "objects" of larger scale by expert players? All this brain science stuff always brings me back to these kinds of bread-and-butter vision questions, but your mileage may vary. Ultimately, I hope this was a nice way to see how fMRI analyses can be used not just to label parts of the brain as belonging to a chess network, but to explore proposals regarding the visual and cognitive processes being carried out in specific parts of the chess player's brain.

I'm not entirely sure what I'd like to write about next, so be on the lookout for...well, kind of anything really. I might say some more about cognitive neuroscience by way of EEG, but I'm also thinking a little bit about some more data science-y things related to chess cognition. If you have questions or topics you'd like to see in Science of Chess posts, please feel free to let me know in the forum!

Support Science of Chess posts!

Thanks for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Atherton, M., Zhuang, J., Bart, W. M., Hu, X., & He, S. (2003). A functional MRI study of high-level cognition. I. The game of chess. Brain research. Cognitive brain research, 16(1), 26–31. https://doi.org/10.1016/s0926-6410(02)00207-0

Bilalić, M., Langner, R., Ulrich, R., and Grodd, W. (2011b). Many faces of expertise: fusiform face area in chess experts and novices. J. Neurosci. 31, 10206–10214. doi: 10.1523/JNEUROSCI.5727-10.2011

Bilalić, M., Langner, R., Erb, M., and Grodd, W. (2010). Mechanisms and neural basis of object and pattern recognition: a study with chess experts. J. Exp. Psychol. Gen. 139, 728–742. doi: 10.1037/a0020756

Chen X, Zhang D, Zhang X, Li Z, Meng X, He S, Hu X. A functional MRI study of high-level cognition. II. The game of GO. Brain Res Cogn Brain Res. 2003 Mar;16(1):32-7. doi: 10.1016/s0926-6410(02)00206-9. PMID: 12589886.

Huberle, E., and Karnath, H.-O. (2012). The role of temporo-parietal junction (TPJ) in global Gestalt perception. Brain Struct. Funct. 217, 735–746. doi: 10.1007/s00429-011-0369-y

Rennig, J., Bilalić, M., Huberle, E., Karnath, H. O., & Himmelbach, M. (2013). The temporo-parietal junction contributes to global gestalt perception-evidence from studies in chess experts. Frontiers in human neuroscience, 7, 513. https://doi.org/10.3389/fnhum.2013.00513