This is how the brain represents words and sentences

A new computer model can predict brain activity for a sentence such as, “The family played at the beach,” even though it has never seen that specific sentence before.

“We found that we can predict brain activity patterns—not perfectly [on average 70% correct], but significantly better than chance,” says Andrew Anderson, a research fellow at the University of Rochester, who led the study.

“We wanted to know if given a whole sentence, can we filter out what the brain’s representation of a word is.”
To build the model, scientists used a broad set of sentences, with many words shared between them. For example: “The green car crossed the bridge,” “The magazine was in the car,” and “The accident damaged the yellow car.”

The team collected data from brain scans taken with functional magnetic resonance imaging (fMRI) from 14 participants as they silently read 240 unique sentences.

“We estimate the representation of a word ‘car,’ in this case, by taking the neural activity pattern associated with all of the sentences which that word occurred in and we decomposed sentence level brain activity patterns to build an estimate of the representation of the word,” says Anderson.

“Using fMRI data, we wanted to know if given a whole sentence, can we filter out what the brain’s representation of a word is—that is to say, can we break the sentence apart into its word components, then take the components and predict what they would look like in a new sentence,” he explains.

The study is published in the journal Cerebral Cortex.

“First, we introduced a method for predicting the neural patterns of words within sentences—which is a more complex problem than has been addressed by previous studies, which have almost all focused on single words,” Anderson says.

“And second, we devised a novel approach to map semantic characteristics of words that we then correlated to neural activity patterns.”

65 attributes of the word ‘coffee’

“Coffee has a color, smell, you can drink it—coffee makes you feel good—it has sensory, emotional, and social aspects,” says senior author Rajeev Raizada, assistant professor of brain and cognitive sciences. “So we built upon a model created by Jeffrey Binder at the Medical College of Wisconsin, a coauthor on the paper, and surveyed people to tell us about the sensory, emotional, social and other aspects for a set of words.

“Together, we then took that approach in a new direction, by going beyond individual words to entire sentences.”

The new semantic model employs 65 attributes—such as “color,” “pleasant,” “loud,” and “time.” Participants in the survey rated, on a scale of 0-6, the degree to which a given root concept was associated with a particular experience.

This is what’s going on in your brain when you read

For example, “To what degree do you think of ‘coffee’ as having a characteristic or defining temperature?” In total, 242 unique words were rated with each of the 65 attributes.

“The strength of association of each word and its attributes allowed us to estimate how its meanings would be represented across the brain using fMRI,” says Raizada.

The model captures a wider breadth of experience than previous semantic models, says Anderson, “which made it easier to interpret the relationship between the predictive model and brain activity patterns.”

The team was then able to recombine activity patterns for individual words, in order to predict brain patterns for entire sentences built up out of new combinations of those words.

The researchers says the study opens a new set of questions toward understanding how meaning is represented in the brain.

“Not now, not next year, but this kind of research may eventually help individuals who have problems with producing language, including those who suffer from traumatic brain injuries or stroke,” says Anderson.

The Intelligence Advanced Research Projects Activity and the National Science Foundation supported the research.