the vision
and memory lab

Computational models of visual memory

Much of our work builds and tests explicit computational models of how visual information is represented and how memory-based decisions get made. Instead of treating memory as a fixed number of stored items, (e.g., all or none "slots" in working memory, or all-or-none recollection in long-term memory), we model the continuous, noisy strength of memory signals, and how observers turn those signals into responses and confidence judgments. As one example, our Target Confusability Competition (TCC) model shows that one psychophysical similarity function can unify measures of visual memory that had looked unrelated, including working memory, long-term memory, and even visual attention and ensemble perception. The same modeling argues we ought to change how memory is measured.

Structure and capacity of visual memory: noisy and hierarchical across time scales

We think working memory and long-term memory, though usually studied apart, rely on the same kind of representation: noisy and hierarchically structured. Your memory of a scene is built from several somewhat separable parts: memories of features like color and shape, the links between those features and whole objects, statistical summaries of the set of objects, the global texture and spatial layout, and semantic knowledge. Knowing any one of these constrains the others, so no part of the representation stands on its own. The same structure can support nearly perfect memory for visual detail in some conditions and very noisy memory in others. It also changes how capacity should be measured, and it predicts a counterintuitive result: visual long-term memory can hold detail at almost the same fidelity as visual working memory.

Knowledge, meaning, and real-world objects

How does prior knowledge change what we can hold in mind? People remember far more about real-world, meaningful objects in working memory than you would predict from how they do with simple colors or oriented lines. Meaningfulness helps even for the basic perceptual features that working memory studies usually focus on. People remember a color better when it sits on a recognizable object than on a scrambled version of that object or a plain shape, even when the color is random and paired with the object at random. We think meaningful objects act as a scaffold that lets observers hold more in mind, and we have been mapping where that help runs out. Semantic labels on their own, for instance, are not enough when the object cannot be mentally reorganized.

Scene representation and ensemble perception

People categorize scenes remarkably fast and accurately. What makes that possible, and what do we actually take away from a single glance at a scene?

We have come at this from several directions. When people remember a display, they store individual items, but they also store summary statistics and the global texture of the whole set. These “ensemble” representations work together with item information, and they matter enough that some classic results about working memory capacity turn out to reflect how people use ensemble statistics rather than a hard limit on items. We have also shown that people implicitly pick up the categorical regularities of real-world experience, like which kinds of scenes tend to follow one another.

We have also looked at the neural and computational basis of scene representation. Earlier fMRI work showed that scene-selective cortex (the PPA) and object-selective cortex (the LOC) play complementary roles, with the PPA tracking a scene’s spatial structure and the LOC its component objects. More recently we took a model-comparison approach and found something we did not expect: a 2D Gabor-wavelet model, which captures only spatial-frequency and orientation information, predicts responses in scene-selective regions (OPA, PPA, and MPA/RSC) better than a model built from explicit 3D-surface structure. Together with our finding that global ensemble texture is critical to rapid scene perception, this suggests that much of what these “scene-selective” regions represent may be carried by relatively low-level image statistics. We are still asking how ensemble and texture perception relate to scene recognition.

Scaling up and applying visual cognition

We take the theories and methods from the lab and ask how they hold up in the real world. Working with expert radiologists, we find that knowing which abnormality is present literally changes what people see, producing a visual hindsight bias for mammograms, and that the high-fidelity memory we study is stronger in experts. Other projects look at how people perceive and remember faces under everyday occlusions like masks and sunglasses, how people read (and reliably misread) data visualizations like bar graphs and scatterplots, and how widely used measures of semantic networks, often tied to creativity, hold up across different tasks.