Bumsub Ham1 | Dongbo Min2 | Changjae Oh1 | Minh N. Do3 | Kwanghoon Sohn1 |
1Yonsei Univ. | 2ADSC | 3UIUC |
A comparison of the PBR and the DIBR: with (a) the left reference image and (b) corresponding depth map and matching probability, intermediate views were rendered via (c) the DIBR and (d) the PBR, respectively. |
Abstract
In this paper, a probability-based rendering (PBR) method is described for reconstructing an intermediate view with a steady-state matching probability (SSMP) density function.
Conventionally, given multiple reference images, the intermediate view is synthesized via the depth image-based rendering (DIBR) technique in which geometric information (e.g., depth)
is explicitly leveraged, thus leading to serious rendering artifacts on the synthesized view even with small depth errors. We address this problem by formulating the rendering process as
an image fusion in which the textures of all probable matching points are adaptively blended with the SSMP representing the likelihood that points among the input reference images are
matched. The PBR hence becomes more robust against depth estimation errors than existing view synthesis approaches. The matching probability (MP) in the steady-state, SSMP, is inferred
for each pixel via the random walk with restart (RWR). The RWR always guarantees visually consistent MP, as opposed to conventional optimization schemes (e.g., diffusion or filtering
based approaches), the accuracy of which heavily depends on parameters used. Experimental results demonstrate the superiority of the PBR over the existing view synthesis approaches both
qualitatively and quantitatively. Especially, the PBR is effective in suppressing flicker artifacts of virtual video rendering although no temporal aspect is considered. Moreover, it is shown that
the depth map itself calculated from our RWR-based method (by simply choosing the most probable matching point) is also comparable to that of the state-of-the-art local stereo matching
methods.
Paper: PDF
Previous Work:
Probabilistic Correspondence Matching using Random Walk with Restart
British Machine Vision Conference (BMVC), 2012.
[PDF]
[PPT]
SSMP - MATLAB (rar), ver1.0 (2013.12.30), 2451KB
PBR - Available upon request
Experimental Results: SSMP
Results for the Middlebury Stereo
|
|
|
|
|
|
|
|
|
|
|
|
Numerical Evaluation
Comparison to DIBR with Still Images
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Comparison to DIBR with Video Sequences
Vassar sequence [1], rendered from the view point of 0 and 2 to that of 1 by the DIBR [2] and the PBR.
Ground truth (mov, 5mb) | Rendering results (mov, 6mb) |
BookArrival sequence [3], rendered from the view point of 6 and 10 to that of 8 by the DIBR [2] and the PBR.
Ground truth (mov, 4mb) | Rendering results (mov, 5mb) |
Poznan sequence [3], rendered from the view point of 4 and 5 to that of 3 by the DIBR [2] and the PBR.
Ground truth (mov, 2mb) | Rendering results (mov, 4mb) |
GtFly sequence [3], rendered from the view point of 1 and 9 to that of 5 by the DIBR [2] and the PBR.
Ground truth (mov, 4mb) | Rendering results (mov, 8mb) |
[1] ISO/IEC JTC1/SC29/WG11 "Multiview Video Test Sequences from MERL," ISO/IEC JTC1/SC29/WG11 Doc. M12077, Apr. 2005.
[2] D. Min, D. Kim, S. Yun, and K. Sohn, "2D/3D Freeview Video Generation for 3DTV System," Signal Processing: Image Communication, vol. 24, no. 1-2, pp. 31-48, 2009.
[3] ISO/IEC JTC1/SC29/WG11, "Call for Proposals on 3D Video Coding Technology," ISO/IEC JTC1/SC29/WG11 Doc. N12036, Mar. 2011.
References
Citation
[1] B. Ham, D. Min, C. Oh, M. N. Do, and K. Sohn, Probability-Based Rendering for View Synthesis, IEEE Trans. on Image Process., vol. 23, no. 2, pp. 870-884, Feb. 2014.
[2] C. Oh, B. Ham, and K. Sohn, Probabilistic Correspondence Matching using Random Walk with Restart, British Machine Vision Conference (BMVC), Sep. 2012.
BibTex
@article{Ham14tip, author = {Bumsub Ham and Dongbo Min and Changjae Oh and Minh N. Do and Kwanghoon Sohn}, title = {Probability-Based Rendering for View Synthesis}, journal = {IEEE Trans. on Image Process. (TIP)}, year = {2014}, month = {Feb.} }
@inproceedings{Oh12bmvc, author = {Changjae Oh and Bumsub Ham and Kwanghoon Sohn}, title = {Probabilistic Correspondence Matching using Random Walk with Restart}, journal = {British Machine Vision Conference (BMVC)}, year = {2012}, month = {Sep.} }
AcknowledgementsThis work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (NRF-2013R1A2A2A01068338).
Last updated: Apr, 2014