IEEE Trans. on Image Process. Probability-Based Rendering for View Synthesis
Bumsub Ham1 Dongbo Min2 Changjae Oh1 Minh N. Do3 Kwanghoon Sohn1
1Yonsei Univ. 2ADSC 3UIUC

 


A comparison of the PBR and the DIBR: with (a) the left reference image and (b) corresponding depth map and matching probability, intermediate views were rendered via (c) the DIBR and (d) the PBR, respectively.

 

Abstract

In this paper, a probability-based rendering (PBR) method is described for reconstructing an intermediate view with a steady-state matching probability (SSMP) density function. Conventionally, given multiple reference images, the intermediate view is synthesized via the depth image-based rendering (DIBR) technique in which geometric information (e.g., depth) is explicitly leveraged, thus leading to serious rendering artifacts on the synthesized view even with small depth errors. We address this problem by formulating the rendering process as an image fusion in which the textures of all probable matching points are adaptively blended with the SSMP representing the likelihood that points among the input reference images are matched. The PBR hence becomes more robust against depth estimation errors than existing view synthesis approaches. The matching probability (MP) in the steady-state, SSMP, is inferred for each pixel via the random walk with restart (RWR). The RWR always guarantees visually consistent MP, as opposed to conventional optimization schemes (e.g., diffusion or filtering based approaches), the accuracy of which heavily depends on parameters used. Experimental results demonstrate the superiority of the PBR over the existing view synthesis approaches both qualitatively and quantitatively. Especially, the PBR is effective in suppressing flicker artifacts of virtual video rendering although no temporal aspect is considered. Moreover, it is shown that the depth map itself calculated from our RWR-based method (by simply choosing the most probable matching point) is also comparable to that of the state-of-the-art local stereo matching methods.

Paper: PDF

Previous Work:

Probabilistic Correspondence Matching using Random Walk with Restart
British Machine Vision Conference (BMVC), 2012. [PDF] [PPT]

Code

SSMP - MATLAB (rar), ver1.0 (2013.12.30), 2451KB

PBR - Available upon request

 

Experimental Results: SSMP

Results for the Middlebury Stereo













Results for (left to right) Tsukuba, Venus, Teddy and Cones: (from top to bottom) left image, (b) ground truth, and (c) depth maps obtained from the SSMP. The proposed method provides high quality depth maps especially around depth boundaries, although only 4-neighborhood is used for inferring matching probability.

Numerical Evaluation


Experimental Results: PBR

Comparison to DIBR with Still Images
















Intermediate views of Cones [32] for (from left to right) the DIBR(NOcc), the DIBR(NOcc+POST1), DIBR(Occ), DIBR(Occ+POST1), and the PBR with (from top to bottom) the AW(1), the AW(7), and the AW(21), respectively. Although the PBR implicitly handle the occlusion and disocclusion regions, it show the sharp transition around the boundaries in contrast to the DIBR. Furthermore, the PBR is more powerful when the low quality MP is given, since it relaxes the errors from local minima. See the Table IV for the object comparison.


Comparison to DIBR with Video Sequences

Vassar sequence [1], rendered from the view point of 0 and 2 to that of 1 by the DIBR [2] and the PBR.

Ground truth (mov, 5mb) Rendering results (mov, 6mb)

BookArrival sequence [3], rendered from the view point of 6 and 10 to that of 8 by the DIBR [2] and the PBR.

Ground truth (mov, 4mb) Rendering results (mov, 5mb)

Poznan sequence [3], rendered from the view point of 4 and 5 to that of 3 by the DIBR [2] and the PBR.

Ground truth (mov, 2mb) Rendering results (mov, 4mb)

GtFly sequence [3], rendered from the view point of 1 and 9 to that of 5 by the DIBR [2] and the PBR.

Ground truth (mov, 4mb) Rendering results (mov, 8mb)

[1] ISO/IEC JTC1/SC29/WG11 "Multiview Video Test Sequences from MERL," ISO/IEC JTC1/SC29/WG11 Doc. M12077, Apr. 2005.
[2] D. Min, D. Kim, S. Yun, and K. Sohn, "2D/3D Freeview Video Generation for 3DTV System," Signal Processing: Image Communication, vol. 24, no. 1-2, pp. 31-48, 2009.
[3] ISO/IEC JTC1/SC29/WG11, "Call for Proposals on 3D Video Coding Technology," ISO/IEC JTC1/SC29/WG11 Doc. N12036, Mar. 2011.

 

References

Citation

[1] B. Ham, D. Min, C. Oh, M. N. Do, and K. Sohn, Probability-Based Rendering for View Synthesis, IEEE Trans. on Image Process., vol. 23, no. 2, pp. 870-884, Feb. 2014.
[2] C. Oh, B. Ham, and K. Sohn, Probabilistic Correspondence Matching using Random Walk with Restart, British Machine Vision Conference (BMVC), Sep. 2012.

BibTex

@article{Ham14tip, author = {Bumsub Ham and Dongbo Min and Changjae Oh and Minh N. Do and Kwanghoon Sohn}, title = {Probability-Based Rendering for View Synthesis}, journal = {IEEE Trans. on Image Process. (TIP)}, year = {2014}, month = {Feb.} }

@inproceedings{Oh12bmvc, author = {Changjae Oh and Bumsub Ham and Kwanghoon Sohn}, title = {Probabilistic Correspondence Matching using Random Walk with Restart}, journal = {British Machine Vision Conference (BMVC)}, year = {2012}, month = {Sep.} }

Acknowledgements

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (NRF-2013R1A2A2A01068338).

 

 

Last updated: Apr, 2014