epth Map Adjustment for the Improvement of tereoscopic Image Perception Manbae Kim ept. of Computer and Communications Engineering Kangwon Nationa University Chunchon, Repubic of Korea, 2-71 E-mai: manbae@kangwon.ac.kr Abstract - Recenty, a variety of stereoscopic contents have been provided to academic and industria fieds for broadcasting, movies and mobie materias. However, few works have been interested in the adjustment of 3 contents for diverse dispays. For instance, movie contents suited to arge screen frequenty do not deiver the same 3 perception to sma-size screen such as mobie phone, tabuar PCs, etc. For this, this paper presents an adjustment method of stereoscopic contents. 2+epth is one of popuar methods with which stereoscopic images are generated. For this, depth panes are derived based on a depth histogram. By adjusting depth panes, a new depth map is made. hen 2+epth produces a stereoscopic image. Experiments performed on various 2+epth images vaidate that the proposed methods deiver more enhanced 3 depth based on subjective evauation experiments. Keywords: depth map, stereoscopic image, 3 improvement 1 Introduction he advances in stereoscopic video technoogies have ed to an increasing interest in various 3 appications [1, 2]. ignificant amount of research has been carried out to introduce new 3 appications. In genera, stereoscopic images are acquired from two camera sensors. ispaying the images on a 3 monitor, humans can view and perceive 3. In genera, the stereoscopic images are deivered to viewers without any modification or enhancement. Any simiar efforts have not been performed for such a probe yet. Based on this, this paper presents a nove method to enhance 3 perception of the stereoscopic images. he overa aim of the proposed method is to enhance the quaity of viewing experience of the end users [3]. 2+depth map approach is used as the representation format in our approach. patia compexity of depth map is one of the key dimensions by which the perceived quaity and depth perception of stereoscopic image are adjusted. he experimenta resuts demonstrate that the ower the spatia compexity is, the higher the perceived video quaity and depth perception are. he paper is organized as foows: Overa approach is introduced in ection 2. ection 3 presents the agorithm of dividing the depth into depth panes utiizing the spatia compexity of the depth maps. he depth map adjustment agorithm is presented in ection 4. Experimenta resuts are described in ection 5. Finay, ection 6 concudes the paper. 2 Overview of Proposed Method Fig. 1. he bock diagram of the proposed method Fig. 1 shows the overa approach of the proposed method. Given an input depth map, its histogram is anayzed for the separating a depth map into depth panes. he spatia compexity is exaed for the depth panes. hen the depth panes undergo the adjustment for the variation of 3 perception. Combining the depth panes, an output depth map is made. A stereoscopic image is generated by 2+epth method. 3 epth Pane Generation 3.1 patia compexity patia compexity of depth maps is measured by appying standard deviation to the pixe depth vaues in the histogram. he reason behind using the standard deviation for the measurement of spatia compexity is that it is the measure of the dispersion or variabiity of a set of vaues around the mean of that set [4]. hus, if the depth map has high spatia compexity, the standard deviation of the pixe depth vaues is expected to be high. he pixes in the depth map detere the distance of the associated coor image pixe to the viewer.
hey take grey vaues ranging from to 255. represents the furthest away pixe from the viewer, whie 255 corresponds to the cosest pixe to the viewer in a 3 scene. Given an MxN depth map, the mean pixe depth is computed by 1 M 1N = 1 µ ( n) (1) NM m= n= ubsequenty, the standard deviation is defined by 3.2 Histogram anaysis 1 M 1N = 1 2 [ ( n) µ ] m= n= NM σ (2) epth map histogram H (i) provides the frequency of the depth vaue i in the depth map, and is defined as foows: Fig 2 shows an exampe of depth threshods with which a depth map is divided into five depth panes. he red bar indicates the depth threshods. Fig. 2 he separation of a depth map into depth panes by depth threshods (coored in red) he mean and standard deviation of pixe depth vaues of each depth pane are computed by where M 1 N 1 1 H ( i) = δ [ i, ( n)] (3) NM m= n= 1,if a = b δ ( a, b) =, otherwise ( j) j R = No.of pixes in R µ and 2 [ ( j) µ ] j R σ = (7) No.of pixes in R As we, a cumuative histogram C(i) of a histogram H(i) is defined as where C(255) = NM. i C( i) = H ( h) (4) h= 3.3 epth pane generation he histogram separation using Gaussian mixture mode has been studied in some appications [5]. his method might work for a coupe of objects and background. On the contrary, for images containing mutipe objects and background, this method may not work we. herefore, we use a simpe, but efficient method utiizing the cumuative histogram. he foowing condition is used. C ( i k) C( i + k) < (5) where is a threshod vaue and k is a user-defined parameter. i vaues satisfying the above condition are chosen as depth threshod separating a depth map. uppose that (L+1) depth threshods (e.g., i, i 1,, i L ) are acquired. L depth panes are then generated. hen the range of the th depth pane is defined as [ R =, max ] = [ i 1, i ], {1,..., L } (6) 4 epth Map Adjustment he bock diagram of Fig. 3 shows the depth map adjustment agorithm proposed in this paper. Given depth panes, standard deviation measuring the spatia compexity is computed for each depth pane. he source standard deviation σ is the sum of depth pane standard deviations. If a target standard deviation σ is detered ( σ < σ ), the depth range of depth panes are reduced unti σ is ess than σ. As a resut, the distance between two depth panes are widened and the 3 depth difference between them becomes more visibe. Finay, a stereoscopic image can be generated from 2+epth approach. For each depth pane, the standard deviation σ is computed using Eq. (2). hen σ is the sum of L depth pane standard deviations. σ = L = 1 σ o reduce the spatia compexity, we define a target standard deviationσ as foows; σ (8) σ = τ (9) where τ is a user-defined parameter at [, 1]. In the experiments, τ is set to be.9,.8,.7, and.6.
Unti σ is achieved, the depth range of depth panes are reduced. he depth panes are sorted according to its standard deviation. he reduction of depth range starts from a depth pane with the greatest standard deviation with a reduction ratio λ. he foowing equation expains how the range of a depth pane is reduced. For a range [, max ], its depth range is adjusted into [ E, E max ] as foows: E = ( + λ and E = ( 1 λ (1) 1 ) max ) max tep 6: If σ > σ, stop and fina depth panes are acquired. 5 Experimenta Resuts he proposed methods were performed on various 2 images and depth maps. We iustrate the resuts for each test image. he first image is MR breakdance image and depth sequences as shown in Fig. 4. he depth pane adjustment agorithm is impemented by the foowing iterative method: Fig. 4. RGB image and depth map [6] he depth threshods are 38, 45, 7, 149, 21, and 216. Five depth panes are shown in Fig. 5. Fig. 5. Five depth panes Fig. 3 he bock diagram of depth map adjustment method Given L input depth panes, tep 1: σ is computed for each depth pane. ubsequenty, σ is aso computed. tep 2: arget standard deviation σ is set with. tep 3: We sort the depth panes according to σ. he depth adjustment of a depth pane with the greatest standard deviation is processed. = and λ is set to be. or.5. tep 4: he depth range of th depth pane is adjusted. As we, a new standard deviation σ is aso computed. tep 5: If σ < σ, exae whether is ess than L. If < L, = + 1 and go to tep 4. Otherwise is and increase by 1 and λ = λ + λ. When τ is.8, at the first iteration, we can not achieve σ. o, we increased λ by.1. hen in the second iteration, the condition was met at the depth pane 4. A fina σ is 786. For τ =.7, the second iteration was competed with depth pane 3. A fina σ is 685. he fina depth panes are shown in Fig. 6. abe 1. tandard deviation of input and output depth panes. (* denotes the range-changed depth panes) Output standard deviation epth pane σ τ =.9 σ =998 τ =.8 σ =887 τ =.7 σ =776 1 43 43 39 * 39 * 2 8 82 68 * 68 * 3 35 359 289 * 188 * 4 41 298 * 176 * 176 * 5 214 214 214 * 214 * = σ σ 341 996 786 685
We observed the stereoscopic images with a 3 monitor adopting QCQ (oube timuus Continuous Quaity cae) subjective test [7]. At the first stage, origina views were dispayed to five participants. Each participant watched the views for 1 seconds and their new views for the same period, and evauated the effect of the 3 depth. wo test sets were carried out in order to exae the 3 perception improvement. epth perception was then subjectivey judged on a scae of 1 (no improvement), 2 (mid improvement), 3 (average improvement), 4 (good improvement) and 5 (exceent improvement) in terms of 3 perception. Fig. 11 shows two subjective grades with respect to τ as we as σ. As τ decreases, we observe that the perceived quaity is improved. Fig. 8. RGB image and depth map of Baet Fig. 6. epth panes generated byτ. τ =.8 and τ =.7 Fig. 7 shows the histogram of input and output depth maps. As τ becomes smaer, the distance between depth panes increases. herefore, depth difference is more apparent. (c) (d) Fig. 7. he histograms of input depth map and depth maps at τ =.9, (c) τ =.8, and (d) τ =.7 he second test image in Fig. 8 is Baet sequence of MR [6]. Fig. 9 shows newy adjusted depth panes. Fig. 9. epth panes generated according to τ. τ =.8 and τ =.7
8 References (c) (d) Fig. 1. he histograms of input depth map and depth maps at τ =.9, (c).8, and (d).7 Fig. 11 ubjective grades with respect to τ and σ 6 Concusions In this paper, we presented a depth map adjustment method that coud provide the improvement of 3 stereoscopic perception. For this, a histogram of a depth map is used for the extraction of mutipe depth panes. For spatia compexity, standard deviation of each depth pane is exaed. According to the target standard deviation, the depth range of each depth pane is adjusted, thereby making the distance between neighboring depth panes increased. his effect deivers better 3 perception that was vaidated through subjective tests. Our proposed method is neary automatic and is expected to provide a technica contribution to 3 video fied. [1] L. M. J. Meesters, W. A. IJssesteijn, and P. J. H. euntiens, A survey of perceptua evauations and requirements of three-dimensiona V, IEEE ransactions on Circuits and ystems for Video echnoogy, Vo. 14, No. 3, pp. 381-391, 24. [2] Bonde, L., oyen,., and Bore,., 3 stereo rendering chaenges and techniques, Information ciences and ystems (CI), 21 44th Annua Conference on, IEEE, pp. 1-6, 21. [3] C. Fehn, epth-image-based Rendering (IBR), Compression and ransmission for a New Approach on 3- V, Proc. of the PIE, vo. 5291, pp. 93-14, CA, U..A., Jan. 24. [4] J. L. evore, Probabiity and tatistics for Engineering and the ciences, uxbury, 1995. [5] Harimi, A. and Ahmadyfard, A., Image egmentation Using Correative Histogram Modeed by Gaussian Mixtre, IEEE Int Conf. on igita Image Processing, 29. [6] http://research.microsoft.com/vision/interactive VisuaMediaGroup/3Videoownoad/, Microsoft Research [7] E. Lee, H. Heo, and K. Park, he comparative measurements of eyestrain caused by 2 and 3 dispays, IEEE rans. on Consumer Eectronics, Vo. 56, Issue 3, pp. 1677 1683, 21. 7 Acknowedgement his work was supported by the MKE (he Ministry of Knowedge Economy), Korea, under the IRC (Information echnoogy Research Center) support program supervised by the NIPA (Nationa I Industry Promotion Agency). (NIPA- 211-(C19-1111-3)) and by the I R& program of MKE/KCC/KEI. [KI258, igna Processing Eements and their oc eveopments to Reaize the Integrated ervice ystem for Interactive igita Hoograms].