E.-K. Lee and Y.-S. Ho: Generaton of Mult-Vew VdeoUng a Fuon Camera Sytem for 3D Dplay 2797 Generaton of Mult-Vew Vdeo Ung a Fuon Camera Sytem for 3D Dplay Eun-Kyung Lee and Yo-Sung Ho, Senor Member, IEEE Abtract In th paper, we preent a fuon camera ytem combnng one tme-of-flght depth camera and two vdeo camera to generate mult-vew vdeo equence. In order to obtan the mult-vew vdeo ung the fuon camera ytem for 3D dpla we capture a tereo vdeo ung a par of vdeo camera and a ngle vew depth vdeo wth the depth camera. After performng a 3D warpng operaton for the depth vdeo to obtan an ntal depth map at each vewpont, we refne t ung egment-baed tereo matchng. To reduce mmatched depth value along object boundare, we detect movng object ung color dfference between frame. Fnall we recompute the depth value of each pxel n every egment ung tereo matchng wth a new cot functon. Expermental reult how that the propoed fuon ytem produce mult-vew vdeo equence wth accurate depth map, epecally along the boundare of object. Therefore, t utable for generatng more natural 3D vew for 3D dplay than prevou wor 1. Index Term 3D dpla depth camera, depth etmaton, mult-vew mage generaton I. INTRODUCTION A 3D vdeo become attractve n a varety of 3D multmeda applcaton, t eental to obtan mult-vew vdeo equence wth correpondng depth map, whch are often called a mult-vew vdeo-plu-depth data [1]. In the near future, conumer wll be able to experence 3D depth mpreon and chooe ther own vewpont n the mmerve vual cene created by 3D vdeo. Recentl the ISO/IEC JTC1/SC29/WG11 Movng Pcture Expert Group (MPEG) ha recognzed the mportance of the mult-vew vdeo-pludepth data for free-vewpont TV (FTV) or 3DTV [2], and ha nvetgated need for tandardzaton on 3D vdeo codng [3], [4]. Moreover, 3D vdeo ytem have been tuded to repreent 3D cene for 3D dplay [5], [6]. Wth repect to the current 3D reearch actvte, t mportant to etmate accurate depth nformaton from real world cene. Although varou depth etmaton method have been developed n the feld of computer von, accurate meaurement of depth nformaton from natural cene tll tme-conumng and problematc. In general, depth etmaton method can be clafed nto two categore: pave depth enng and actve depth 1 Th reearch wa upported by the MKE(The Mntry of Knowledge Econom, Korea, under the ITRC(Informaton Technology Reearch Center) upport program uperved by the NIPA(Natonal IT Indutry Promoton Agenc (NIPA-2010-( C1090-1011-0003)). E.K. Lee and Y.S. Ho are wth the Department of Informaton and Communcaton, Gwangju Inttute of Scence and Technology (GIST) (emal: {elee78, hoyo}@gt.ac.r). enng. The former calculate depth nformaton ndrectly from 2D mage captured by two or more vdeo camera. Typcal example nclude hape from focu [7] and tereo matchng [8]. The advantage of ndrect depth etmaton low prce becaue we can create depth map ung cheap offthe-helf vdeo camera. However, accuracy of the depth map relatvely lower than thoe produced from actve approache n occluon and texturele regon. On the other hand, actve depth enng method uually employ phycal enor, uch a laer, nfrared ray (IR), or lght pattern, to obtan depth nformaton from natural cene drectly. Structured lght pattern [9] and depth camera [10], [11] are major example of thee approache. Neverthele, thee drect depth etmaton tool and ytem are qute expenve for conumer. Therefore, tme-of-flght (TOF) depth camera wth low prce and mall ze have been ntroduced and appled for 3D home game and multmeda envronment [12]. Whle they can capture depth value drectly n real-tme, ther crucal dadvantage are that they produce only low-qualty depth map wth optcal noe. In recent year, fuon camera ytem compoed of multple vdeo camera and one or more TOF camera have been propoed [13], [14]. Zhu et al. [15] preented a calbraton method to mprove depth qualty ung a TOF depth enor. They ued the probablty dtrbuton functon of the depth nformaton meaured by the TOF depth enor and provded a more relable depth map. Lee et al. [16] enhanced the depth reoluton and accuracy by combnng the actual dtance nformaton meaured by the depth camera wth the dparty map etmated by the pave depth enng method. However, the prevou fuon ytem have produced only low-reoluton depth map and focued on generatng depth map of tatc 3D cene. Nowada many reearch nttute and compane are ntereted n development of a fuon camera ytem for 3D conumer devce uch a 3D cellphone, 3D tablet PC, 3D laptop, 3D game conole, etc. Snce forthcomng 3-D multmeda applcaton runnng on thoe devce are expected to ue hgh qualty 3-D vdeo, we need to create mult-vew vdeo data wth hgh qualty depth nformaton. In th paper, we deve a fuon camera ytem wth one depth camera and tereo vdeo camera. The propoed ytem can produce mult-vew mage for dynamc 3D cene by enhancng the low-reoluton depth nformaton meaured by the depth camera. The man contrbuton of our wor to propoe a practcal 3D vdeo generaton oluton for dynamc 3D cene, whch can be applcable to 3D conumer devce. Contrbuted Paper Manucrpt receved 10/15/10 Current veron publhed 12/23/10 Electronc veron publhed 12/30/10. 0098 3063/10/$20.00 2010 IEEE
2798 IEEE Tranacton on Conumer Electronc, Vol. 56, No. 4, November 2010 Image Acquton Preproceng Stereo Matchng Content Dplay Stereo Camera Image Rectfcaton Color Correcton Bloc-baed Color Segmentaton Mult-vew Image Generaton Depth Camera Relatve Camera Calbraton Enhancement Depth Correcton Radal Dtorton Correcton Segment-baed Stereo Matchng Movng Object Detecton Refnement Generaton 3D Vdeo Generaton Stereo Vdeo Generaton Fg. 1. Overall framewor of mult-vew mage generaton for 3D dplay. The remander of th paper organzed a follow. In Secton II, we preent the overall archtecture of the propoed fuon camera ytem. Secton III decrbe preproceng tep for enhancng depth map and Secton IV preent how to generate the multple vdeo equence wth ther correpondng depth map ung the propoed camera ytem. After howng expermental reult n Secton V, we draw concluon n Secton VI. II. SYSTEM ARCHITECTURE The propoed fuon ytem compoed of one depth camera and two vdeo camera. Fgure 1 llutrate the overall framewor to generate tereo depth map ung the fuon ytem. After calbratng each camera ndependentl we perform an mage rectfcaton to adjut vertcal mmatche n multple mage. Then, we apply a color correcton operaton to mantan color contency among tereo mage. To obtan depth map for tereo mage, we perform a 3D warpng operaton onto each tereo camera ung the depth map meaured by the depth camera. The warped depth data ued a an ntal depth at each camera poton. After we egment each tereo mage, we agn the depth value of the warped depth data n each egment a the ntal depth of the egment. In order to mprove the depth accuracy of object boundare, we eparate the movng object ung color dfference between frame. Then, the depth of each egment refned by a color egmentaton-baed tereo matchng method. Fnall we obtan depth map by conductng a pxelbaed depth map refnement ung a propoed cot functon n each egment. Snce all tep are proceed twce, from left to rght and rght to left, we can obtan at leat two depth map n two vew. From the two-vew nformaton, mult-vew mage can be generated from the propoed algorthm. In th paper, we ntroduce a compact and mnmum camera etup for mult-vew mage generaton wth two vdeo camera and one depth camera. However, dependng on applcaton and devce capablte, th ytem can be ealy extended to mult-vew vdeo and mult-vew depth camera. III. PREPROCESSING OF THE FUSION CAMERA SYSTEM If the propoed camera etup bult n 3D devce, the followng tep can be pped n practcal envronment. Once the camera etup fxed n the devce, parameter computed from the preproceng tage are not changed. A. Relatve Camera Calbraton Snce the propoed fuon camera ytem cont of two dfferent type of camera, a depth camera and tereo vdeo camera, t eental to fnd out relatve camera nformaton through camera calbraton [17]. For that, we apply a camera calbraton algorthm to each camera n our camera ytem and obtan projecton matrce for the depth camera and each vdeo camera. P K R t ] (1) [ P K R t ] (2) [ where P the projecton matrx of the depth camera repreented by t camera ntrnc matrx K, rotaton matrx R, and tranlaton vector t. P mean the projecton matrce of the th vdeo camera whch conted of t camera ntrnc matrx K, rotaton matrx R, and tranlaton vector t. We then employ a rectfcaton operaton [18]. The camera have geometrc error becaue they are et manually by hand. In order to mnmze the geometrc error, we fnd the common baelne, and then apply the rectfyng tranformaton to the tereo mage. Conequentl the projecton matrx of vdeo camera are changed a P K R t ] (3) [ where K and R are the modfed camera ntrnc matrx and rotaton matrx of the th vdeo camera, repectvely. Thereafter, we convert the rotaton matrx R of the depth camera nto the dentty matrx I by multplyng nvere rotaton matrx R -1. The tranlaton vector t of the depth camera alo changed nto the zero matrx O by ubtractng the tranlaton vector t. Hence, we can defne new relatve projecton matrce for the tereo camera on the ba of the depth camera a P K [ I O] (4) ~ 1 P K [ R R t t ] (5)
E.-K. Lee and Y.-S. Ho: Generaton of Mult-Vew VdeoUng a Fuon Camera Sytem for 3D Dplay 2799 where P and ~ P ' are fnal projecton matrce of the depth camera and the th vdeo camera, repectvely. After relatve camera calbraton, we reolve the color mmatch problem of tereo mage ung a color calbraton method [19]. The color charactertc of captured mage are uually ncontent due to dfferent camera properte and lghtng condton even the hardware type and pecfcaton of the multple camera are the ame. Thereafter, we perform blateral flterng to reduce optcal noe ncluded n the depth map acqured from the depth camera [20]. B. Depth Calbraton The depth value meaured by the depth camera are very entve to noe. Ther ource are dvere ncludng phycal lmtaton of hardware and pecfc object properte, etc. Therefore, depth data are notceably contamnated wth random and ytematc meaurement error dependent on reflectance, angle of ncdence, and envronmental factor le temperature and lghtng [21]. To reduce thoe error, we employ a depth calbraton method [17]. For depth calbraton n ndoor envronment, we compute the depth of the planar checer pattern wthn the lmted pace by ncreang the dtance from the mage pattern to the depth camera ung our ytem a hown n Fg. 2. To extract the correpondng feature pont n two dfferent type of camera effcentl we ue the color checer pattern. The pattern mage captured n every 5cm dtance. The plane pattern orthogonal to the mage plane. To chec the accuracy of the calbrated depth value, we perform 3D warpng to the tereo camera. Fgure 3(a) the 3D warpng reult ung the acqured depth map and Fg. 3(b) how that of the calbrated depth map ung the LUT. Whle there are many mmatched depth value n Fg. 3(a), mot of them are correctly matched n the boundare of the rectangular box n Fg. 3(b). The other problem that even though the dtance from the depth camera to the object contant, depth nformaton from the depth camera can be dfferent dependng on the object color and lghtng condton. 3D warpng Acqured depth map Projecton to left mage (a) Before Depth Calbraton 3D warpng Calbrated depth map Projecton to left mage (b) After Depth Calbraton Fg. 3. Depth accuracy tet ung acqured depth map and calbrated depth map. (a) Pattern acquton (b) Pattern mage from hybrd camera ytem Fg. 2. Acquton of the planar chec pattern for depth calbraton. Thereafter, we mae a four dmenonal loo-up table (LUT) mappng 3D poton of the multple vdeo camera and the depth value from the depth camera. 3D poton contructed by y poton of the feature pont and the real depth value calculated by the mult-vew mage. Depth accuracy tet ung the acqured depth map and calbrated depth map the real depth value z calculated from the multvew mage by parwe tereo matchng. Snce we have already obtaned camera parameter, the real depth value calculated by K B dn( p py ) (6) D ( p, p ) n where K the focal length of the left camera and B the baelne dtance between two vdeo camera. D n (p x, p y ) the real depth value correpondng to the meaured dparty value d n (p x, p y ) at the pxel poton (p x, p y ) n the checer pattern. x y To analyze the depth entvty of a tatc object n the dynamc cene, we chec the depth value of a blac har, a hown n Fg. 4. We can notce the ncontent depth value change of the tatc object caued by object movement and materal properte. Epecall the depth value of the dar color regon meaured by the depth camera very untable and unrelable. The blac har ha to utan a near-contant depth n the cene; however, the acqured depth value are unpredctable and random. The reaon that dar or blac color aborb lght of all frequence and the depth camera ue near IR ray. 224 161 Acqured depth value 240 230 220 210 200 190 180 170 160 150 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 No. of frame Fg. 4. Depth ncontency for tatc object. Depth A Depth B Depth value of B Depth value of A
2800 Although we perform the depth calbraton to correct the acqured depth map, there are tll lmtaton n the depth value acqured from the depth camera. To obtan the hghqualty mult-vew depth map, we need to refne the acqured depth value ung an effcent tereo matchng algorthm. C. Radal Dtorton Correcton for Depth Image Depth map from the depth camera have a large amount of len radal dtorton. There are two type len dtorton whch are barrel dtorton and pncuhon dtorton. In th cae, the barrel dtorton occurred by the ntrnc problem of the depth camera. Th dtorton caue not only the hape mmatch between the color mage and the correpondng depth mage but alo the error n the reult of ome feature pont baed proceng uch a camera calbraton. In order to avod that tuaton, we have to perform radal dtorton correcton to the obtaned depth mage. In general, there are two man categore of radal dtorton correcton. Method n the frt category ue the pont correpondence between two or more vew. The econd category alo ha lot of approache whch are baed on the dtorted traght lne component n the mage. In the propoed fuon camera ytem, we ue one of the econd approache to correct the radal dtorton n the depth mage [22]. After fndng the curved traght lne component n the captured mage, we etmate the dtorton center and the dtorton parameter. Wth the dtorton nformaton, we can recontruct the mage from the dtorted mage. Fgure 5 how the depth and ntenty mage before and after the correcton. IEEE Tranacton on Conumer Electronc, Vol. 56, No. 4, November 2010 movng D to the world coordnate carred out by P K 1 p (7) where K -1 ndcate the ntrnc matrx of the depth camera. In the bacward 3D warpng, nce rotaton and tranlaton matrce of the depth camera are the dentty matrx I and zero matrx O, we have only to conder t ntrnc matrx. Thereafter, we project the 3D pont P nto the each vew to get t correpondng pxel poton p (u, v ) of the th -vew mage by ~ p P ' P where P ndcate the projecton matrx of the th -vew vdeo camera. Fgure 6 how the reult of 3D warpng ung the acqured depth map. (a) Warped depth mage (8) (b) Matched to color mage Fg. 6. 3D warped depth map. Orgnal Fg. 5. Radal dtorton correcton. Corrected B. Regon Separaton To etmate depth map of tereo vdeo camera ung the warped depth nformaton, we egment the mult-vew mage by a mean-hft color egmentaton algorthm [23]. However, we cannot control the maxmum egment ze becaue there no parameter to control the maxmum egment ze. When we perform the egment-baed tereo matchng, one egment ha one depth value. If the ze of egment too large, we cannot get a mooth depth map. The other wa f the ze of egment too mall, t hard to overcome texturele problem durng the tereo matchng. To olve th problem, we plt one mage nto 1616 bloc egment, o that we can lmt the maxmum egment ze. IV. DEPTH MAP GENERATION A. 3D Warpng of Depth Camera Data We generate ntal depth of the mult-vew mage by performng 3D warpng of the depth value obtaned from the depth camera. Frt, we project pxel of the depth map nto the 3D world coordnate ung the depth value. We then reproject the 3D pont nto each vew. Let u aume that D (p x, p y ) the depth ntenty at the pxel poton (p x, p y ) n the depth map. P (x x, y y, z z ) a 3D pont correpondng to D. The bacward projecton for 16 BLOCK_SIZE = 16 Segment A count_pxel > Th Segment B_1 Segment A_1 count_pxel < Th Fg. 7. Bloc-baed egment mergng. f (count_pxel > Th) color_ndex++ ele f (count_pxel < Th) eg_mergng( );
E.-K. Lee and Y.-S. Ho: Generaton of Mult-Vew VdeoUng a Fuon Camera Sytem for 3D Dplay 2801 Fgure 7 how the procedure of the egment mergng. A bloc can have two or more color egment. Before mergng the egment, we plt the egmented mage nto bloc-baed egment agan. If each egment maller than half ze of the bloc, we merge t nto one egment by earchng adjoned bloc to fnd the ame ndexed egment. If the ze of the merged bloc larger than threhold, the mergng procedure fnhed; otherwe we repeat the ame proce untl mergng condton atfed. The earchng order of connected bloc rght, bottom, left, and top ncludng the dagonal drecton becaue left and top bloc are merged bloc and rght and bottom bloc wll be merged bloc. For example, Segment A dvde nto many bloc-baed egment and Bloc (, j) have two egment: Segment A_1 and Segment B_1. Snce the ze of Segment A_1 maller than the predefned threhold value n Fg. 8, the ame ndexed egment of Segment A_1 the bloc n (+1, j), (, j+1), and (+1, j+1). We merge the current Segment A_1 and the ame ndexed egment n (+1, j) by the earchng order. Before we etmate depth map, we eparate movng object ung color dfference between frame. To extract the movng object n the current frame, we calculate color dfference between the prevou frame n-1 and the current frame n by ung the threhold whch ndcate the current poton foreground or not. We cannot drectly ue the egment-baed movng object detecton becaue hape of each egment can be vared n the temporal doman a hown n Fg. 8. Fg. 8. Segmentaton reult n temporal doman. Snce color egmentaton performed frame by frame, t hard to fnd the ame egment n the temporal doman. Therefore, we ue the Eucldean dtance between frame to extract the movng object a E ( R y n 2 2 2 n1 Rn ) ( Gn 1( Gn( ) ( Bn 1( Bn )) where R, G, and B ndcate the pxel value n RGB color doman. To fnd the movng object, we compute the E n ( at each pxel locaton for all pxel. If we ubtract the RGB value between frame, camera noe can be mxed up. To remove them, we calculate the average RGB value for 33 bloc. If the average larger than the threhold value, we et the center pxel of each 33 bloc a the foreground pxel. Fgure 9 how the reult of movng object for 78 th frame mage n the left camera. (9) Segment-baed Mult-vew Depth Etmaton We defne the ntal depth of each egment a 3D warped depth n the egment; the aumpton that each egment ha one depth value [7]. However, there one problem to et the ntal depth ung warped depth value. The 3D warpng performed from the mall reoluton depth map to the tereo mage n our ytem. Snce there are many error uch a camera calbraton error and depth error acqured from the depth camera, the warped reult not exactly matched wth the tereo mage a hown n Fg. 10. Acqured depth map 3D warpng Warped ntal depth reult Fg. 10. Boundary mmatchng problem. Boundary mmatchng To obtan the accurate ntal depth value, we ue the warped reult a multple ntal depth value for tereo matchng. If we tart the tereo matchng wth the ntal depth, we can reduce the earch range for fndng the matched regon. In addton, dependng on earch range reducton, we can overcome the mmatched problem n the texturele regon. However, f the gven ntal depth the error value, we could fnd wrong area whch ha local mnmum. Therefore, the agnment of the correct ntal depth crucal n ung the depth camera. Becaue there are correct ntal depth around the currently warped poton, whch are not exactly matched wth the orgnal mage, we ncreae the canddate of the ntal depth value to reolve th problem. Fgure 11 how the poton of the ntal depth n two drectonal regon, horzontal and vertcal regon. One or more ntal depth value uually ext n a 1010 area becaue of the dfference of the reoluton. In th cae, we et the horzontal earch regon a 8020 and the vertcal earch regon a 2080. By ung the multple ntal depth, we can et ntal depth for the depthle regon n the boundary of object a hown n Fg. 11. Vertcal earch range Horzontal earch range Current poton for tereo matchng Fg. 9. Movng object detecton ung color dfference between frame. Fg. 11. Set of multple ntal depth value.
2802 Snce tereo matchng meaure the dfference between the correpondng pont of two mage, called a the dpart we convert the ntal depth nto t dparty for tereo matchng by K B IntDp( (10) IntDepth( where IntDp( the converted dparty at the pxel poton ( from the correpondng ntal depth IntDepth(. B and K are the dtance between two vdeo camera and the focal length of the current vdeo camera, repectvely. After performng tereo matchng wth the ntal dpart we convert agan the calculated dparty nto t depth value to produce the depth map. Before performng b-drectonal tereo matchng, we need to et the canddate of the ntal depth value. For determnng the dparty of each egment, we calculate the mean of abolute dfference (MAD) value between the egment n the current vew mage and t matched regon n the left or rght vew mage by a FG _ d ( IntDp) mn(mn( MAD( j), mn( MAD( )) (11) j0 0 where the ndex of the egment, j and mean ndex of the multple ntal depth. a and b are the number of the ntal depth n the horzontal and vertcal regon, repectvely. FG_d (IntDp) the refned ntal depth value from parwe tereo matchng. Search range to etmate dparte of the current vew mage from IntDp-5 to IntDp+5. The dparty wth the mnmum MAD n the earch range choen a the refned ntal dparty of the egment n the current vew mage. Snce the acqured depth map only for foreground regon, there no depth nformaton for bacground area. We defne that the bacground ha no ntal depth or the number of the ncluded ntal depth n the egment le than 10% of the ze of the egment. In etmatng depth of bacground, we et the mnmum and maxmum depth/dparty value. We then fnd the mnmum MAD a the ntal dparty of the current egment n the bacground by max Dp BG _ d ( IntDp) mn( MAD( )) (12) mn Dp where BG_d (IntDp) the dparty for bacground, mndp and maxdp mean mnmum and maxmum dparty earch range for bacground. The dparty wth the mnmum MAD choen a the ntal dparty d (Intdp) of the egment n the current vew mage n by d ( IntDp) mn( FG _ d ( IntDp), BG _ d ( IntDp)) (13) C. Mult-vew Refnement In tereo matchng, depth refnement uually enhance depth accuracy through teraton at the cot of long b IEEE Tranacton on Conumer Electronc, Vol. 56, No. 4, November 2010 proceng tme, lot of memory requrement, and heavy computaton. However, t ha challenge when our target to generate hgh-qualty mult-vew vdeo baed on depth map. We therefore propoe a mplfed depth refnement approach ung the propoed cot functon for the depth map refnement, whch ha the followng feature: low memory conumpton, fat proceng tme, and no teraton tep. In order to enhance the depth map along the boundary of the object, we refne t for two regon: movng regon and tatc regon. We have already defned the movng regon ung color dfference between frame a hown n Fg. 9. If there no varance of a pxel n the tme doman, we aume that pxel tatc. In that cae, we can refer the prevou depth value for the tatc pxel. Otherwe, we jut ue the refned dparty value wthout referrng the prevou one. w f d ) wd fd dd ) f obj _ mov( 1 E( d) w f( d( ) wd fd dd ) wt ft dt ) f obj _ mov( 0 (14) where w, w d, w t are the weghtng factor for depth refnement. f ( d ( the moothne term wth gradent of the refned depth value n th refnement tep. f d ( d d ( the data term for the refned ntal depth value n the egment-baed tereo matchng tep and f t ( d t ( the temporal term for depth value of the prevou frame for the tatc pxel. obj_mov( ndcate the reult of the movng object detecton. If obj_mov( 0, th pxel not moved. Then, we can refer the depth value of the prevou frame. f d ( d d ( mean the mnmum MAD wth the refned ntal depth value n the earch range from IntDp-5 to IntDp+5. f ( d ( the depth dfference wth neghborhood depth n the ame egment and calculated by f d med( a, b, c ) (15) We can calculate the moothne value a hown n Fg. 12. a ( the refned depth dfference at poton between (x - 1, y-1) and (x-1,. b ( the refned depth dfference at poton between (x-1, y-1) and ( y-1). c ( the refned depth dfference at poton between ( y-1) and (x+1, y-1). The functon med( ) tae the medan value among argument to avod the wrong depth electon, o that t mantan depth contnuty along the vertcal and horzontal drecton. If the elected moothne gradent a vertcal drecton, th depth dfference calculated from ( y-1). Otherwe, the depth dfference computed from (x-1,. S a ( S b ( S c ( (x-1, y-1) ( y-1) (x+1, y-1) (x-1, ( Refned depth Current poton Fg. 12. Smoothne defnton wth gradent of the refned depth value.
E.-K. Lee and Y.-S. Ho: Generaton of Mult-Vew VdeoUng a Fuon Camera Sytem for 3D Dplay 2803 V. EXPERIMENTAL RESULTS AND ANALYSIS In order to generate the hgh-qualty depth map, we have contructed a fuon camera ytem wth two camera and one depth camera. The meaurng depth range of our depth camera from 0.50m to 5.00m. The baelne dtance between two vdeo camera 6.5cm. The propoed camera ytem' baelne dtance depend on the phycal volume of our vdeo camera and the depth camera. However, t poble to reduce the baelne between camera n other ytem confguraton. Fgure 13 how the acqured tet equence captured by the fuon camera ytem. The reoluton of our tet tereo mage 1920 1080, and that of the depth map 176 144. From our expermental, the weghtng factor of the cot functon w, w d, w t are 0.3, 0.5, and 0.2 and the threhold value of Eucldean dtance, 10 ued. From Fg. 14 and Fg. 15, we notce that depth for the overlapped regon n foreground were generated uccefull though the boundare of the blac har were noy. In addton, the yellow table expree gradual depth dfference depte the monotonou color of the table. A a reult, we could overcome the two man problem of pave depth enng effcentl depth etmaton on the occluded and texturele regon, ung the depth camera data a the upplementary nformaton. Fgure 16 preent the computed depth map from 30 th to 270 th n every 30 frame. (a) Stereo Image (a) Depth map from Zhu method (b) Depth map from the propoed ytem Fg. 15. Depth comparon wth the prevou wor. (b) Acqured Fg. 13. Tet mult-vew mage and t depth map Fgure 14 how the fnal tereo color mage and ther correpondng depth map for the 1 t frame. To compare the depth qualty of the propoed method wth prevou wor, we have hown the dparty map generated by Zhu method. method for the left mage of the 93 rd frame a hown n Fg. 15. We can oberve that ome regon of the depth map generated by the prevou method have notceable error n concave area. Furthermore, the mmatched dparte n blac har were remarably reduced by the propoed method. Fg. 16. Generated depth map equence. (a) Stereo Image To evaluate the ubjectve qualty of the propoed method, we have ynthezed ntermedate vew wth the computed depth map ung VSRS oftware [24]. A hown n Fg. 17, the generated ntermedate vew ung depth map obtaned by the propoed method are reaonable n the apect of ubjectve qualty. From the aforementoned reult, the propoed approach outperform the prevou method. (b) Stereo depth map Fg. 14. Generated depth map
2804 IEEE Tranacton on Conumer Electronc, Vol. 56, No. 4, November 2010 (a) Zhu method (b) Propoed ytem Fg. 17. Intermedate mage comparaon. Fg. 18. Generated ntermedate mage ung generated depth map. Fgure 18 how the generated mult-vew mage ugn the generated depth map. Table I how the comparon of the proceng tme n the depth refnement tep. Snce each algorthm have dfferent proceng tep to generate the depth map, t hard to meaure the exact proceng tme n the ame condton. Therefore, we compare the proceng tme for the depth map refnement tep. A hown n Table I, the propoed method fater than other wthout the accuracy reducton for depth map generaton. From the reult, t ueful for the hgh-qualty mult-vew vdeo generaton. SEQUENCE TABLE I COMPARISON OF THE PROCESSING TIME Zhu method Proceng tme (ec) Propoed method Café 836.26 337.21 VI. CONCLUSION In th paper, we have preented a new approach to generate depth map correpondng to color mage ung the propoed fuon camera ytem. We have ued depth nformaton acqured by a depth camera to generate the ntal depth map for tereo matchng. We then have generated the fnal depth map ung egmentaton-baed tereo matchng and the propoed cot functon. Expermental reult have hown that our cheme produced more relable depth map and mult-vew mage compared wth prevou method. Wth the propoed fuon camera ytem, we could olve the two man problem n the current pave depth enng, whch depth etmaton on occluded and texturele regon. Fnall we have generated hgh-qualty mult-vew mage from our ytem. Therefore, our propoed ytem could be ueful for varou 3D multmeda applcaton and dplay. REFERENCES [1] P. Kauff, N. Atzpadn, C. Fehn, M. Müller, O. Schreer, A. Smolc, and R. Tanger, Depth map creaton and mage-baed renderng for advanced 3DTV ervce provdng nteroperablty and calablt Sgnal Proceng Image Communcaton, vol. 22, no. 2, pp. 217-234, Feb. 2007. [2] C. Fehn, R. Barre, and S. Patoor, Interactve 3DTV- concept and ey technologe, Proceedng of the IEEE, vol. 94, no. 3, pp. 524-538, March 2006. [3] ISO/IEC JTC1/SC29/WG11 N8944, Prelmnary FTV model and requrement, Aprl 2007. [4] A. Smolc and D. McCutchen, 3DAV exploraton of vdeo-baed renderng technology n MPEG, IEEE Tran. on Crcut and Sytem for Vdeo Technolog vol. 14, no. 3,.pp. 348-356, March 2004.
E.-K. Lee and Y.-S. Ho: Generaton of Mult-Vew VdeoUng a Fuon Camera Sytem for 3D Dplay 2805 [5] S. Km, S. Lee, and Y. Ho, Three-dmenonal Natural Vdeo Sytem baed on Layered Repreentaton of, IEEE Tran. Conumer Electronc, vol. 52, no. 3, pp. 1035-1042, Augut 2006. [6] H. Shn, Y. Km, H. Parl, and J. Par, Fat Vew Synthe ung GPU for 3D Dpla IEEE Tran. Conumer Electronc, vol. 54, no. 4, pp. 2068-2076, November 2006. [7] C. Ztnc, S. Kang, M. Uyttendaele, S. Wnder, and R. Szel, Hgh-qualty vdeo vew nterpolaton ung a layered repreentaton, Proc. of ACM SIGGRAPH, pp. 600 608, Augut 2004. [8] C. Ztnc, S. Kang, M. Uyttendaele, S. Wnder, and R. Szel, Hgh-qualty vdeo vew nterpolaton ung a layered repreentaton, Proc. of ACM SIGGRAPH, pp. 600 608, Augut 2004 [9] D. Scharten, and R. Szel, Hgh-accuracy tereo depth map ung tructured lght, IEEE Computer Socety Conference on Computer Von and Pattern Recognton, vol. 1, pp. 195-202, June 2003. [10] G. Iddan and G. Yahav, 3D magng n the tudo and elewhee, Proc. of SPIE Vdometrc and Optcal Method for 3D Shape Meaurement, pp. 48 55, Jan. 2001. [11] M. Kawata, T. Kurta, H. Kuch, and S. Inoue, HDTV ax-von camera, Proc. of Internatonal Broadcatng Conference, pp. 397 404, Sept. 2002. [12] S. Gotur, H. Yalcn, C. Bamj, A tme-of-flght depth enor - ytem decrpton, ue and oluton, Proc. of IEEE Conference on Computer Von and Pattern Recognton Worhop, vol. 3, pp. 35, June 2004. [13] Y. Km, D. Chan, C. Theobalt, and S. Thrun, Degn and calbraton of a mult-vew TOF enor fuon ytem, Proc. of IEEE Conference on Computer Von and Pattern Recognton, pp. 539 546, June 2008. [14] G. Um, K. Km, C. Ahn, and K. Lee, Three-dmenonal cene recontructon ung mult-vew mage and depth camera, Proc. of SPIE Stereocopc Dplay and Vrtual Realty Sytem XII, vol. 5664, pp. 271 280, Jan. 2005. [15] J. Debel and S. Thrun, An applcaton of Marov random feld to range enng, Proc. of Advance n Neural Informaton Proceng ytem, pp. 291-298, Dec. 2005. [16] J. Zhu, L. Wang, R. Yang, and J. Dav, Fuon of tme-of-flght depth and tereo for hgh accuracy depth map, Proc. of IEEE Conference on Computer Von and Pattern Recognton, pp. 231 236, June 2008. [17] E. Lee, Y. Kang,Y. Jung, Y. Ho, 3D vdeo generaton ung hybrd camera ytem, Proc. of Internatonal Conference on Immerve Telecommuncaton (IMMERSCOM), pp. T5(1-6), June 2009. [18] Z. Zhang, A flexble new technque for camera calbraton, IEEE Tran. on Pattern Analy and Machne Intellgence, vol. 22, pp. 1330-1334, Nov. 2000. [19] Y. Kang, and Y. Ho, Geometrcal compenaton for mult-vew vdeo n multple camera arra Proc. of Internatonal Sympoum ELMAR, vol. 1, pp 83-86, Sept. 2008. [20] N. Joh, B. Wlburn, V. Vah, M. Levo and M. Horowtz, Automatc color calbraton for large camera arra n UCSD CSE Techncal. Report. CS2005-0821, May 2005. [21] J. Cho, I. Chang, S. Km, and K. Lee, Depth mage proceng technque for repreentng human actor n 3DTV ung ngle depth camera, Proc. of 3DTV conference, paper no. 15, May 2007. [22] S. Schuon, C. Theobalt, J. Dav, and S. Thrun, Hgh-qualty cannng ung tme-of-flght depth uperreoluton, Proc. of IEEE Conference on Computer Von and Pattern Recognton on Tme-of-Flght Computer Von, pp.1-8 2008. [23] A. Wang, T. Qu, and L. Shao, A mple method of radal dtorton correcton wth centre of dtorton etmaton, Journal of Mathematc Imagng and Von, vol. 35, no. 3, pp. 165-172 Nov. 2009. [24] D. Comancu and P. Meer, Mean hft: a robut approach toward feature pace analy, IEEE Tran. on Pattern Analy and Machne Intellgence, vol. 24, no. 4, pp. 603 619, May 2002. [25] ISO/IEC JTC1/SC29/WG11 M15377, Reference oftware for depth etmaton and vew ynthe, Aprl 2008. BIOGRAPHIES Eun-Kyung Lee receved both B.S. and M. S. degree n computer engneerng from Honam Unverty (HU), Korea, n 2002 and 2004, repectvely. She currently worng toward her Ph.D. degree n the Informaton and Communcaton Department at the Gwangju Inttute of Scence and Technology (GIST), Korea. Her reearch nteret nclude dgtal gnal proceng, mult-vew vdeo codng algorthm and ytem, mult-vew depth map generaton, 3D televon, and realtc broadcatng Yo-Sung Ho receved both B.S. and M.S. degree n electronc engneerng from Seoul Natonal Unvert Korea, n 1981 and 1983, repectvel and Ph.D. degree n Electrcal and Computer Engneerng from the Unverty of Calforna, Santa Barbara, n 1990. He joned the Electronc and Telecommuncaton Reearch Inttute (ETRI), Korea, n 1983. From 1990 to 1993, he wa wth Phlp Laboratore, Brarclff Manor, New Yor, where he wa nvolved n development of the advanced dgtal hgh-defnton televon (AD-HDTV) ytem. In 1993, he rejoned the techncal taff of ETRI and wa nvolved n development of the Korea drect broadcat atellte (DBS) dgtal televon and hgh-defnton televon ytem. Snce 1995, he ha been wth the Gwangju Inttute of Scence and Technology (GIST), where he currently a profeor n the Informaton and Communcaton Department. H reearch nteret nclude dgtal mage and vdeo codng, mage analy and mage retoraton, advanced codng technque, dgtal vdeo and audo broadcatng, 3D televon, and realtc broadcatng.