深度和结构相似性引导的四参考视点融合算法
PDF下载 (179)何小梅,章联军,陈 芬,蔡真真,王晓东.深度和结构相似性引导的四参考视点融合算法[J].宁波大学学报(理工版),2022,35(2):96-104.DOI:
HE Xiaomei,ZHANG Lianjun,CHEN Fen,CAI Zhenzhen,WANG Xiaodong.Depth and structural similarity guided blending algorithm with four-reference viewpoints[J].Journal of Ningbo University(Natural Science & Engineering Edition),2022,35(2):96-104.DOI:
| Title: | Depth and structural similarity guided blending algorithm with four-reference viewpoints |
| 作者: | 何小梅, 章联军, 陈 芬, 蔡真真, 王晓东 |
| Author(s): | HE Xiaomei, ZHANG Lianjun, CHEN Fen, CAI Zhenzhen, WANG Xiaodong |
| 关键词: | 深度图像绘制; 虚拟视点融合; 平面相机阵列 |
| Keywords: | depth image based rendering; viewpoints blending; plane camera array |
| 分类号: | TP391.9 |
| 文献标识码: | A |
| 摘要: | 平面相机阵列四参考视点的深度图像绘制(Depth Image Based Rendering, DIBR)方案允许用户全方位身临其境地体验场景, 可有效避免虚拟视点图像边界空洞, 然而该方案引入了较为显著的伪影、背景渗透等失真. 为此, 提出一种深度和结构相似性(Structural Similarity, SSIM)引导的四参考视点融合算法. 首先, 深入分析了针对平面相机阵列的四参考视点DIBR方案中失真产生的原因; 然后, 利用参考视点与虚拟视点间的相对位置关系进行视野错误排除, 并根据恰可察觉失真模型提取融合图像的失真掩膜; 最后, 利用失真区域各视点的深度信息和SSIM进行自适应视点融合, 进而绘制出高质量的虚拟视点图像. 实验结果表明, 本文算法绘制的虚拟视点图像比标准方案在SSIM和沉浸式视频峰值信噪比方面分别提升了0.0018和1.46dB, 比文献方法在主观视觉感知方面更接近于真实图像. |
| Abstract: | Depth image based rendering (DIBR) with four-reference viewpoints for planar camera array allows users to enjoy the scene immersively from all directions. This scheme can effectively eradicate holes distortion at the boundaries of virtual viewpoint images. However, it results in obvious artifacts, background penetration, etc. Therefore, this paper proposes a four-reference viewpoints blending algorithm guided by depth and structural similarity (SSIM). Firstly, the main distortions generated by DIBR are analyzed based on four-reference viewpoints for planar camera array comprehensively. Then, the view errors are eliminated as per the relative position relationship between reference and virtual viewpoints, and distortion masks of the blended image are extracted using the noticeable distortion model. In the end, the depth information and the SSIM in the distorted areas of each viewpoint are utilized to optimize the blending weights, and the high-quality virtual viewpoint image is thus rendered. Experimental results show that the SSIM and immersive video peak signal-to-noise ratio of the virtual viewpoint image rendered by the proposed algorithm reads from 0.0018 to 1.46dB, which is higher than those of the benchmark. In addition, the virtual viewpoint image rendered by the proposed algorithm is more consistent with the ground truth image compared with the state-of-the-arts in terms of visual perception. |
| 参考文献 /References: | [1] Zhang Y, Zhu L, Hamzaoui R, et al. Highly efficient multiview depth coding based on histogram projection and allowable depth distortion[J]. IEEE Transactions on Image Processing, 2021, 30:402-417. [2] Teratani M, Senoh T, Kroon B, et al. Overview of MPEG-I visual test materials[C]//131st MPEG Meeting of ISO/IEC JTC1/SC29/WG11, 2020. [3] Gao X, Li K, Chen W, et al. Free viewpoint video synthesis based on DIBR[C]//2020 IEEE Conference on Multimedia Information Processing and Retrieval, 2020: 275-278. [4] Cheung C H, Sheng L, Ngan K N. Motion compensated virtual view synthesis using novel particle cell[J]. IEEE Transactions on Multimedia, 2021, 23:1908-1923. [5] Senoh T, Tetsutani N, Yasuda H, et al. Proposed View Synthesis Reference Software (pVSRS 4.3) manual[C]// 124th MPEG Meeting of ISO/IEC JTC1/SC29/WG11, 2018. [6] Vijayanagar K R, Kim J, Lee Y, et al. Efficient view synthesis for multi-view video plus depth[C]//2013 IEEE International Conference on Image Processing, 2013: 2197-2201. [7] Lee T C, Chien C L, Hang H M. Virtual view synthesis quality refinement[C]//2016 IEEE 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), 2016:1-4. [8] Wegner K, Stankiewicz O, Domański M. Novel depth- based blending technique for improved virtual view synthesis[C]//2016 IEEE International Conference on Signals and Electronic Systems, 2016:93-98. [9] Sharma M, Ragavan G. A novel image fusion scheme for FTV view synthesis based on layered depth scene representation & scale periodic transform[C]//2019 IEEE International Conference on 3D Immersion, 2019:1-8. [10] Qiao Y, Jiao L, Yang S, et al. Color correction and depth-based hierarchical hole filling in free viewpoint generation[J]. IEEE Transactions on Broadcasting, 2019, 65(2):294-307. [11] 蔡李美, 李新福, 田学东. 基于分层图像融合的虚拟视点绘制算法[J]. 计算机工程, 2021, 47(4):204-210. [12] Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: From error visibility to structural similarity [J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. [13] Liu A, Lin W, Paul M, et al. Just noticeable difference for images with decomposition model for separating edge and textured regions[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2010, 20(11):1648- 1652. [14] Wang X, Wang K, Yang B, et al. Perceptual quality assessment on DIBR synthesized videos with composite distortions[C]//2020 IEEE International Conference on Image Processing, 2020:186-190. [15] Arseneau S, Cooperstock J R. An improved representation of junctions through asymmetric tensor diffusion[C]// International Symposium on Visual Computing, 2006: 363-372. [16] Senoh T, Tetsutani N, Yasuda H, et al. Proposed Depth Estimation Reference Software (pDERS 8) manual[C]// 124th MPEG Meeting of ISO/IEC JTC1/SC29/WG11, 2018. [17] Dziembowski A. [MPEG-I Visual] Software manual of IV-PSNR for immersive video[C]//127th MPEG Meeting of ISO/IEC JTC1/SC29/WG11, 2019. |
| 备注/Memo: | 收稿日期: 2021－07－30. 宁波大学学报（理工版）网址: http://journallg.nbu.edu.cn/ 基金项目: 浙江省自然科学基金（LY20F01000）; 重庆理工大学科研启动基金（2020ZDZ029, 2020ZDZ03）. 第一作者: 何小梅（1997－）, 女, 浙江金华人, 在读硕士研究生, 主要研究方向: 图像和视频信号处理. E-mail: 1367787845@qq.com *通信作者: 陈芬（1973－）, 女, 四川邻水人, 教授, 主要研究方向: 图像和视频信号处理. E-mail: chenfen@nbu.edu.cn 宁波大学学报（理工版）网址：http://journallg.nbu.edu.cn/ |
