Reconstruction of Visual Information Through Recordings of Brain Activity in the Visual Cortex

The Gallant group at UC Berkeley has taken on the formative challenge of reconstructing images based on brain activity. They do so by recording the neuronal activity that follows exposure to patterns of pictures and videos; subsequently, statistical analysis is used to generate libraries based on the brain activity triggered a specific visual stimulus. First, images are converted into features, informed by shape and movement, through temporal and spatial gabor filtering. Blood oxygen level-dependent (BOLD) signals, obtained through fMRI, are recorded from the occipitotemporal visual cortex. A linear function is approximated based on the features and the BOLD signals they illicit. Researchers are then able to record BOLD signals and predict the image qualities in novel visual stimuli using the linear function as a reference. The reconstructions in the video below were made through this decoding technique.

Predicting Speech Through Cortical Potentials Using ECoG

Intracranial electrocorticographic (ECoG) recordings work by registering the electrical potentials recorded by electrodes implanted in the cerebral cortex. ECoG has an impressive temporal resolution of several milliseconds; the spatial resolution is dependent on the size of the electrodes, and it is typically between several millimeters to one centimeter.

(Image: Herff et al., Frontiers in Neuroscience. 2015.) 

To decode speech using ECoG, subjects were asked to read out some sentences on a screen; ECoG activity and speech were both recorded. Phones refer to physical segments of speech, such as a, b, ch, f, and these were obtained from the speech recordings. In addition, ECoG data was segmented into 50 ms intervals; importantly, broadband-gamma activity (70-170 Hz) was extracted as these signals are associated with auditory processes and word repetition. The ECoG segments were then labeled with the corresponding phones. To use ECoG as a method of predicting speech, both learned models and language models (ex. liberty is pronounced as l-ih-b-er-t-iy) were used. The authors of the study found an error rate of 25% for words and an error rate of less than 50% for phones.

Based on the two studies described above, we can see it is seemingly possible to decode auditory and visual stimuli through brain activity. Nonetheless, these technologies will require further advancement as the fMRI lacks the necessary temporal resolution; conversely, ECoG has excellent temporal and spatial resolution, but is highly invasive.

Spatial and Time Resolution of Current Measurement Technique

(Image: Nature Neuroscience. 17, 1440–1441 (2014))

To summarize the current field of brain technologies, we can see high resolution is achievable, but usually at a cost. Optogenetics, calcium imaging, and single-cell electrode recordings produce excellent temporal and spatial resolution, but these systems have only been applied to animals. As of yet, there is no apparatus which can measure activity across vast regions of the human brain with the same degree of resolution as we have seen in animal experiments.

EEG and MEG measure the electrical currents and magnetic fields, respectively, produced by neuronal activity; both these techniques have a temporal resolution of several milliseconds. On the other hand, EEG and MEG only have spatial resolutions of a few centimeters. Activity recorded by EEG and MEG can only be roughly localized to specific areas of the brain, and it is impossible to ascertain which specific neurons were activated. Conversely, fMRI has exceptional spatial resolution of several millimeters but lacks temporal resolution. As described above, fMRI indirectly measures neural activity; it does so by measuring changes in blood flow that result from neural activity. Specifically, fMRI measures changes in the magnetic properties of blood due to the presence of oxygenated hemoglobin. This methodology comes with some drawbacks as fMRI only records changes that occur following brain activities (meaning lower temporal resolution); furthermore, fMRI is incapable of capturing rapid activity as blood flow changes take time.

Similar to fMRI, NIRS measures hemodynamic changes in the brain and therefore has the same drawbacks pertaining to temporal resolution. Furthermore, NIRS imaging is generally contained to the frontal lobe as hair interferes with the signal; spatial resolution is also diminished slightly as infrared rays must pass through the skin and skull barrier. In addition, NIRS imaging must account for signals coming from skin blood flow. In spite of these obstacles, NIRS is advantageous because recordings are not affected by body movement. Also, myoelectric signals produced by muscle fiber contractions can interfere with EEG and MEG, but NIRS is not affected. Furthermore, NIRS is considered a more practical option compared to techniques like fMRI which require large-scale apparatuses.

Lastly, as mentioned above, in spite of its excellent spatial and temporal resolution, ECoG is not a good option due to its invasiveness.

Concurrent Connection of More Than One Million Neurons Required by DARPA

The human brain is said to have approximately 8.6 billion neurons. Thus, no current method of measuring brain activity is sufficient to record that level of detail. For years DARPA has been interested in the brain-machine interface, and it recently invested 65 million dollars in the Neural Engineering System Design Program. They aim to record interactions between a million neurons simultaneously. This will dramatically enhance our understanding of neuronal functions and our ability to develop new treatments for disabilities. Among the six groups to receive this funding is Paradromics Inc., which seeks to decode spoken language using neural implants. Paradromics Inc. received 18 million dollars while the remaining funds were divided between research institutes.

New Technologies to Improve Resolution and Usability

Imaging machine with incredible spatial and temporal resolution uses lasers and ultrasonic waves

Single-impulse panoramic photoacoustic computed tomography (SIP-PACT) is an example of a newly developed, non-invasive technology offering good spatial and temporal resolutions.

(Image: Li et al., Nature Biomedical Engineering. 2017.) 

Thus far this technique has only been applied to animals but has been shown to be capable of recording information with a depth penetration of 48 mm, a spatial resolution of 125 μm, and a temporal resolution of 50 Hz. These results could have immense implications for preclinical trials and translational research.  

Magnetic nanoparticles applied to imaging apparatuses to improve spatial and temporal resolution

Weinberg Medical Physics is developing a technology that could capture neuronal activity at a spatial resolution of 30 micrometers, and a temporal resolution of 100 Hz. The technology uses 100-nm sized magnetic particles, which can be inhaled through the nose. Using exterior magnets, the particles can be targeted and used to record neural activity. These particles do not require helium, or strong magnetism and are currently being investigated as a novel imaging option by researchers at the University of Maryland and John Hopkins University. Although these nanoparticles have only been tested in animal subjects, they hold great potential for human patients.

Mary Lou Jepsen and her attempt to image the brain in fine detail using holography

Openwater is a company founded by Mary Lou Jepsen whose goal is to develop a non-invasive neuroimaging device with comparable resolution to fMRI using holography technology.

(Image: https://www.youtube.com/watch?v=BP_b4yzxp80)

Opening up the future of neuroscience and technology with high resolution apparatuses

Thanks to advancements in technology, our ability to conduct science and accurately observe nature has accelerated. In particular, the field of neuroscience was widened by the introduction of the fMRI in the 1990s. But considering how many neurons exist in the brain it is clear there is much more to learn than what our current technologies enable.

For example, presently we can predict categories of visual stimuli (ex. cat, building, plane, etc.) by decoding visual cortex activity through fMRI, but our reconstructions are far from perfect. Perhaps with increased resolution we may one day be able to decode exact shapes and outlines.

BMI, which uses electrode arrays such as ECoGs, has made it possible to move a robotic arm. However, in its current state, BMI does not allow for the same fine motor skills and dexterity of a human arm being controlled directly by the brain. By enhancing our recording electrodes perhaps BMI motor precision can be augmented. Moreover, patients can use visual feedback of the arm moving as a gauge of their brain activity. One day it may be possible to use this visual feedback to modify brain activity and allow for better control. 

Emotion is a complex phenomenon. Fear and anger are predominantly associated with the amygdala, but this brain region is also activated when we experience happiness. Recent work suggests the amygdala may be separated into two sections with each part separately responsible for processing positive and negative emotions. Currently, it remains unclear how these signals would be processed and differentiated. Being able to visualize the brain in a more precise manner would allow us to strive to answer profound questions like, “what is emotion, and how does it manifest from brain activity?”

Paradromics and Openwater aim to produce practical devices without cumbersome cables and large apparatuses. Should these aims come to fruition, we will be able to understand the brain activity behind all our daily activities. For example, we could scan the brain and determine which regions are activated when making a purchase. This could then be expanded on to determine if brain activity is modulated based on preferences or the type of purchase. By increasing the resolution and accessibility of brain imaging technology our understanding of the brain, and its everyday functions, could progress exponentially.

References

  • Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu & Jack L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology. 2011
  • Christian Herff, Dominic Heger, Adriana de Pesters, Dominic Telaar, Peter Brunner, Gerwin Schalk & Tanja Schultz. Brain-to-text: decoding spoken phrases from phone representations in the brain. Frontiers in Neuroscience. 2015; 9
  • Terrence J. Sejnowski, Patricia S. Churchland & J. Anthony Movshon. Putting big data to good use in neuroscience. Nature Neuroscience. 2014.
  • Lei Li et al., Single-impulse panoramic photoacoustic computed tomography of small-animal whole-body dynamics at high spatiotemporal resolution. Nature Biomedical Engineering. 2017.
  • www.weinbergmedicalphysics.com
  • https://www.openwater.cc/technology
  • Joshua Kim, Michele Pignatelli, Sangyu Xu, Shigeyoshi Itohara, & Susumu Tonegawa. Antagonistic negative and positive 1 neurons of the basolateral amygdala. Nature Neuroscience. 2016. doi: 10.1038/nn.4414