基於視線之顯示器局部解析度提升技術：演算法及積體電路架構設計

Real-Time Resolution Boosting on Gaze-Contingent Displays: Algorithm and Hardware Architecture

呂宏璋(Hung-Chang Lu)

指導教授 ： 簡韶逸

國立臺灣大學/電機資訊學院/電子工程學研究所/碩士(2020年)

https://doi.org/10.6342/NTU202100040

摘要

由於目前頭戴式顯示器(如虛擬實境VR)的像素密度不足，VR頭戴式顯示器旨在提供的完全沉浸式體驗很難實現。在凝視顯示器的概念下，已有些研究試著透過時域上的向上採樣來提升感知分辨率來克服物理分辨率的限制。然而這些基於優化方法的計算成本過高，阻礙了我們對於實時應用的實現。另一方面，卷積神經網路(CNNs)的硬體實現蓬勃發展，啟發並使得我們得以設計一種基於CNN的演算法及相關硬體架構來解決這類問題。
在本論文中，我們提出一種能夠以合理的計算成本提升VR顯示器感知分辨率的框架。本論文提出的感知幀合成網路可以在時域上產生高分辨率的信息，然後通過視網膜的整合過程恢復高分辨率的感知。此外基於人眼對於中央凹外圍的視力下降，我們透過高幀率來提高感知分辨率，並將我們的方法應用於人眼聚焦的區域內。另外我們還提出了在同一幀中幀率混合的方法，使我們在提高感知體驗的同時，沒有產生邊界偽影。最後我們進行了主觀實驗來驗證所提出框架的有效性。
從實驗結果來看，所提出的算法能夠達到提升感官體驗的效果，此外我們還設計的一個硬體架構來滿足應用的實時性需求。

關鍵字

感受解析度 ； 凝視 ； 視線 ； 積體電路架構

並列摘要

Due to the insufficient pixel density of current head-mounted displays, such as Virtual Reality (VR), the fully immersive experiences that VR headsets aim to provide are hard to achieve. With the concept of gaze-contingent display, several works tried to overcome the resolution constraint by boosting the perceptual resolution with temporally up-sampling the content. However, those optimization-based methods are too computationally expensive, preventing real-time application scenarios. On the other hand, the flourishing hardware evolution of Convolutional Neural Networks (CNNs) inspires and enables us to design an CNN based algorithm and the associated hardware architecture to solve this kind of problem.

In this thesis, we propose a framework that can boost the perceptual resolution of VR displays with reasonable computational cost. The proposed perceptual frame synthesis network can generate high-resolution information in the temporal domain, and the high-resolution perception can then be restored by the integration process in retina. In addition, based on the decrease of visual acuity in the peripheral vision, we improve perceptual resolution by increasing frames rates and apply our method only on the focused region. Furthermore, we propose a method to blend mixed frame-rate regions in the same frame, allowing us to improve the perceptual experience only in the focused region without boundary artifacts. Subjective experiments are conducted to verify the effectiveness of the proposed framework.

From the experimental results, the proposed algorithm can achieve the effect of enhancing sensory experience. In addition, we design a hardware architecture to fulfill the real-time demand of application.

並列關鍵字

Gaze ； Gaze-Contingent ； Resolution Boosting

參考文獻

[1] C. A. Curcio, K. R. Sloan, R. E. Kalina, and A. E. Hendrickson, “Human photoreceptor topography,” Journal of Comparative Neurology, vol. 292, no. 4, pp. 497–523, 1990. 1

[2] A. Eisen-Enosh, N. Farah, Z. Burgansky-Eliash, U. Polat, and Y. Mandel “Evaluation of critical flicker-fusion frequency measurement methods for the investigation of visual temporal resolution,” Scientific reports, vol. 7, no. 1, pp. 1–9, 2017. 2, 8

[3] D. A. Robinson, J. Gordon, and S. Gordon, “A model of the smooth pursuit eye movement system,” Biological cybernetics, vol. 55, no. 1, pp. 43–57, 1986. 3

[4] P. Didyk, E. Eisemann, T. Ritschel, K. Myszkowski, and H.-P. Seidel, “Apparent display resolution enhancement for moving images,” in Proceedings of ACM SIGGRAPH 2010, 2010, pp. 1–8. 3, 6, 7, 8, 9, 31, 38

[5] K. Templin, P. Didyk, T. Ritschel, E. Eisemann, K. Myszkowski, and H.-P. Seidel, “Apparent resolution enhancement for animations,” in Proceedings of Spring Conference on Computer Graphics(SCCG), Vinicne, Slovakia, 2011, pp. 85–92. 3, 7, 9