基於人類大腦視覺皮層之階層性表徵影像重建

Image Reconstruction from Hierarchical Representations of Human Visual Cortex Activities

吳謦廷(Ching-Ting Wu)

指導教授 ： 盧信銘

國立臺灣大學/管理學院/資訊管理學研究所/碩士(2021年)

https://doi.org/10.6342/NTU202102731

摘要

隨著深度學習演算法的快速進步以及功能性磁振造影(functional Magnetic Resonance Imaging, fMRI)技術的成熟，基於腦部活動變化的刺激目標影像重建任務，是近期在神經計算科學領域中廣泛受到討論的研究議題。近幾年，將類神經網路應用於fMRI-影像成對資料集，影像重建任務得以顯著的進步。此外，許多神經科學研究提出，大腦視覺皮質會以階層性地透過不同皮質區域合作解析視覺刺激，影像重建任務的關鍵在於能夠在演算法中實踐此概念，這個階層性解析視覺訊號的概念，提供了科學家深究大腦內部系統運作機制的窗口。
在本研究中，我們首先跟隨由Fang在2020年所提出的演算法「分治法」，是目前基於腦部活動變化影像重建任務中最先進的方法，最終我們得以重製誤差在百分之十以內的影像重建成果。基於重製的結果，我們更進一步採用K-means分群演算法以更加準確地萃取出圖片中的語意特徵。我們的研究展示了「分治法」在影像重建任務中的可行性與彈性，更重要的是，以不同機器學習模型解析來自不同視覺皮層的視覺刺激，可以顯著的提升影像重建的成果，如此的實作方法呼應到大腦視覺皮層的階層性架構，給予科學家一個探索大腦解析視覺刺激的媒介。

關鍵字

影像重建 ； 功能性磁振造影 ； 分治法 ； K-means分群 ； 階層性架構

並列摘要

Reconstructing stimuli images from brain activity has been a rising topic in the field of computational neuroscience due to the advent of deep learning algorithms and the ad-vancement in functional Magnetic Resonance Imaging (fMRI) techniques. The applica-tion of deep neural networks on larger fMRI-image-paired dataset has demonstrated sig-nificant improvement in brain decoding tasks recently. Moreover, perceptive contents be-ing hierarchically encoded in human brain has been the key to many research break-throughs in the field. This concept provides a new window into the internal mechanisms of brain system for scientists.
In this work, we follow the current state-of-the-art method “divide-and-conquer” pro-posed by Fang (2020). We are able to reproduce reconstruction results within 10% of the state-of-the-art performance. Based on the reproduced work, we further enhance our per-formance by introducing K-means image clustering method that captures more accurate semantic features for reconstructed images. Our experiments demonstrate the feasibility and flexibility of “divide-and-conquer” method in image reconstruction task. Most im-portantly, incorporating fMRI signals from different regions of visual cortex with different decoders improves the image reconstruction performance. Such implementation results map to the concept of hierarchical structure of visual cortex, giving us a medium to ex-plore more about how human brain handles visual stimuli.

並列關鍵字

image reconstruction ； fMRI ； divide-and-conquer ； K-means ； hierarchical structure

參考文獻

Agrawal, P., Stansbury, D., Malik, J., Gallant, J. L. (2014). Pixels to Voxels: Modeling Visual Representation in the Human Brain. JOURNAL.

Attwell, D., Buchan, A. M., Charpak, S., Lauritzen, M., Macvicar, B. A., Newman, E. A. (2010). Glial and Neuronal Control of Brain Blood Flow. JOURNAL, 468(7321), 232-243. https://doi.org/10.1038/nature09613

Bandettini, P. A. (2012). Twenty Years of Functional Mri: The Science and the Stories. JOURNAL, 62(2), 575-588. https://doi.org/10.1016/j.neuroimage.2012.04.026

Beliy, R., Gaziv, G., Hoogi, A., Strappini, F., Golan, T., Irani, M. (2019). From Voxels to Pixels and Back: Self-Supervision in Natural-Image Reconstruction from Fmri. JOURNAL.

Calhoun, V. D., Adali, T., Pearlson, G. D., Pekar, J. J. (2001). Spatial and Temporal Independent Component Analysis of Functional Mri Data Containing a Pair of Task-Related Waveforms. JOURNAL, 13(1), 43-53. https://doi.org/10.1002/hbm.1024