A Method of Hand Contour Tracking based on GVF, Corner and Optical flow Ke Du1, a, Ying Shi1,b, Jie Chen2,c, MingJun Sun1, Jie Chen1, ShuHai Quan1

Transcription

1 2nd Intenational Confeence on Machiney, Mateials Engineeing, Chemical Engineeing and Biotechnology (MMECEB 2015) A Method of Hand Contou Tacking based on GVF, Cone and Optical flow Ke Du1, a, Ying Shi1,b, Jie Chen2,c, MingJun Sun1, Jie Chen1, ShuHai Quan1 1 School of Automation, Wuhan Univesity of Technology, Hubei , China 2 WuHan GUIDE INFRARED CO,WuHan,Hubei,430205,China a dukewok@163.com, ba_laly@163.com, cchenjie @163.com Keywods: GVF Snake, SUSAN Cone, Optical flow Abstact.Since bae human hand is highly aticulated and defomable, its tacking is moe challenging than igid object tacking. In this pape, an automatic appoach is poposed to tack hand contou on the basis of gadient vecto flow (GVF Snake), cone detection and spase optical flow. The skin-colo model and motion infomation help to educe the impact of local minimal unde complex backgound. To lowe the sensitivity of GVF Snake to initial contou, a cuvatue-diffeence-based cone detection is poposed to effectively detect enough cones on hand contou and spase optical flow is applied to ensue a obust tacking of these cones even with violent gestue change. The expeiments have shown a pomising hand contou tacking using eal image sequence unde the usual lighting condition in lab. Intoduction In the field of HCI and vitual envionment, hand gestue is egaded as one of the most natual and unconstained communication manne. Since bae human hand is highly aticulated and defomable, its tacking is moe challenging than igid object tacking [1]. Usually, ectangula o ellipse is used to locate human hand egion in taditional hand tacking[2,3], since the contou o fingetips of complicated hand gestue could not be maked, it is difficult to undestand the hand gestue. Fo a bette undestanding, the hand silhouette o contou is always applied to hand gestue ecognition. Kass [4] fistly poposed Snakes, whee the object bounday is descibed as a paamete cuve and is detemined by using an enegy minimization; howeve, the bounday concavities cannot be captued by Snakes. Xu [5] poposed a GVF Snake by using a new extenal foce to enlage the captue ange and enable convege to bounday concavities. As fa as the image with complicated backgound is concened, GVF Snake still could not give a good object contou fo its sensitivity to the local minimal and initial cuve. Many eseaches focused on these issues [6]. Fo a single hand situation, the modified exta-enegies-based algoithms and pediction-based algoithms appeaed [7]. The fome aims to decease local minimal via adding the exta enegies to taditional snake model [8-10]. Exta enegies do help to educe the effect of local minimal, but sometimes esult in the disability of cuve convegence to pecise object bounday. And due to moe computation cost fo exta enegies, these algoithms ae time-consuming. The latte is poposed to lowe the sensitivity to initial contou via a dynamic pediction on the object contou [11-13]. Yang [14] poposed a GVF Snake combined with skin colo egion tacking. The skin colo model could apidly captue pecise hand shapes only fo smooth hand defomation fom fame to fame but the pedictions always fail fo violent changes of hand shape [15]. In this pape, a method is poposed on the base of hand featue and the taditional enegies, and the impact of local minimal on hand egion segmentation is educed by using Gaussian skin model and the cones on hand contou. And we lowe the sensitivity to initial contou by fitting a cuve of these cones as the initial contou in cuent fame fo its similaity to the eal hand contou. In addition, we tack violent changes on hand shape via tacking the selected cones which bette descibe the motion featue in eal-time tacking [16] The authos - Published by Atlantis Pess 542

2 Cheng[17] poposed a GVF Snake method based on the Canny opeato and thinning method to get a consecutive bounday. Fo a bette segmentation pefomance, SUSAN cone detection could be used fo a pecise detection on the points of hand fingetips and thei oots. Howeve, these points ae not enough fo fitting a good hand contou. Cheng[18] and Malagi [19] poposed a modified GVF snake method on the base of SUSAN cone detection. Both the cones on hand contou with high contou cuvatue and SUSAN cones wee used to the hand contou fitting. In this pape, the cones on hand contou will be sceened befoe the tacking accoding to the sign change on fowad and backwad cuvatue diffeence fo a high efficiency. To obustly tack hand contou, the optical flow was intoduced to tack the cones on hand contou between fames [20, 21]. Fo few cones ae concened, the spase optical flow is applied within the motion sub egion to educe the computation [22]. And these cones ae sceened by the cuvatue-diffeence-based cone detection in successive fame. And then GVF Snake adopts the fitting cuve of these cones as initial contou to detect the hand contou. The pape is oganized as follows: Fistly, hand motion egion is located by using skin colo model within the motion sub egion segmentation using backgound subtaction, and then its contou is captued by GVF Snake model. Secondly, the cones on this contou ae detected by using cuvatue-diffeence-based cone detection, and tacked by using spase optical flow. At last, the cones ae intepolated in next fame to fom a closed cuve as the initial cuve of GVF Snake model, and then the hand contou is captued. By epeating these steps, we can obtain a obust and fast tacking on the hand contous in video fame. Initializing the hand contou Colo image segmentation o backgound subtaction often fails to extact the hand egion due to the complicated backgound and vaying illumination. Usually, Gaussian mixtue model is applied to deal with these poblems. With the consideation on the colo featue of human skin, Gaussian mixtue skin model is combined with backgound subtaction to detect the hand egion [32, 33]. Ou wok addesses this poblem by using both Gaussian mixtue skin model and a flood fill method to mak a complete hand egion. Then the contou of maximum egion with coss-sectional aea geate than a given theshold is supposed to be the initial one, and let it convege to the accuate hand contou via using GVF Snake. Extacting the hand egion (a) (b) (c) (d) (e) (f) Fig. 1. Hand egion detection in the 500th fame: (a)oiginal colo image; (b)diffeence image; (c)diffeence image afte emoving noise; (d)skin detection image; (e)flood filled image; (f) hand egion image To meet speed equiement, the fist fame of input video is taken as the backgound, and motion aea is obtained by subtacting the backgound fom next fames (see Fig. 1b), and it can be seen that thee is a lot of noise and shadows. The mophological eosion opeato is adopted to educe noise, as shown in Fig. 1c. To eliminate the inteuptions of shadows and non-skin moving objects, Gaussian mixtue skin model defined by colo statistic is applied [23,24]. N 1 ( ) T 1 1 x µ i i ( x µ i ) 2 P( x skin)= wi e Σ (2-1) 3 1 i= 1 2 (2 π ) Σ 2 i whee N is the numbe of mixtue components defined as 16, x is an RGB colo vecto, and the th contibution of i Gaussian is detemined by a scala weight ω, mean vecto µ, and diagonal i i 543

3 covaiance matix Σ. Two sepaate mixtue models fo skin and non-skin classes ae tained, and a i paticula RGB value is labeled as skin colo if P( x skin) > θ (2-2) P( x skin) whee 0 θ 1. In ode to obtain a good skin detection fo most images, let θ =0.8. This skin colo model sometimes fails fo the existence of holes in hand egion and noise in small egion when the skin egion is too bight o dak (see Fig. 1d fo example). Thus, a flood fill method is poposed to stuff the missing skin aea (see Fig. 1e). In ode to educe the effect of vaying luminance, the YC bc colo space is chosen fo its skin-tone independence on luminance. The flood fill algoithm is as follows: Convet RGB image I RGB ( x,y) to YCbC image IYC ( ) bc x, y and nonlinea tansfom YC bc image IYC C ( x, y) to YC bc image I ( x, y). Fo points ( x, y) of nonzeo value in I ( x, y ) and points b YC C b detected by Gaussian mixtue skin model in Iseed () i, i = 1, 2,3 n, if I( C b )( x,y)+ T1 Iseed ( C b )( i), and I( C b )( x, y)- T2 Iseed (C b )( i), and I( C )( x, y)+ T1 I seed (C )( i), and I( C )( x, y)- T2 Iseed (C )( i), then mak the points I RGB ( x, y) as skin point. T1, T2 ae the minimum and maximum thesholds, espectively. IC ( b), IC ( ) ae values of chomatic component in the YC bc space, espectively. Afte the skin aea detection, the coss-sectional aea sizes ae counted fo all connected domain. If the size of the geatest is geate than a given theshold, the coesponding egion is extacted, othewise thee is no hand image exiting (see Fig.1f). Captuing the initial hand contou Afte the peceding pocess, the elatively complete hand egion is obtained, and now we need to captue the hand contou fo the convenience of hand gestue tacking. The GVF Snake is a kind of cuve X() s = [ xs () ys ()], s [0,1] which moves though the spatial domain of an image to minimize the enegy functional: E= [ α X () s + β X () s ]+ E ( ()) 0 ext X s ds 2 (2-4) Eint = [ α X () s + β X () s ] 0 2 (2-5) whee α and β ae weighting paametes which contol the snake s tension and igidity espectively. The fist tem is the intenal foce, which contols the cuve changes, while the second tem E ext is the extenal foce, which pulls the cuve to desied featues. Diffeent E ext can be constucted in diffeent models. The minimized E of a snake must satisfy the Eule equation αx () s βx () s E ext = 0 (2-6) A new static extenal foce field V takes the place of the taditional extenal foce, V( xy, )=[ ux (, y), v (x,y)], and is deduced by minimizing the enegy functional: E= µ ( ux + uy + vx + vy) + f v f dxdy (2-7) whee f is the edge map of gay level image, and f is the gadient of edge map, the paamete µ is a egulaization paamete. Numeical solution of the GVF field is as follows: ut( xyt,, ) = µ uxyt (,, ) ( uxyt (,, ) fx( xy, ))( fx ( xy, ) + fy ( xy, )) (2-8a) v( xyt,, ) = µ vxyt (,, ) ( vxyt (,, ) f( xy, ))( f ( xy, ) + f ( xy, )) (2-8b) t y x y (2-3) RGB 544

4 In ode to lowe the sensitivity to initial contou and educe calculation cost, we use the hand egion image (see Fig. 2b) as the initial contou of GVF Snake, and then only the GVF field in an exteio ectangula egion(see Fig. 2d), which contains the hand image, needs to be calculated, as shown in Fig. 2d and 2e. (a) (b) (c) (d) (e) Fig. 2. Hand contou initialization:(a)hand contou initialization; (b)hand egion image; (b) hand egion in an exteio ectangula egion; (d)gvf extenal foces in u diection; (e)gvf extenal foces in v diection TRACKING THE HAND CONTOUR Fig.3. Famewok of hand contou tacking Fo the subsequent hand gestue ecognition, the tacking of hand contou is necessay. On the base of the initial hand contou captued above, we poposed an algoithm to obustly tack hand contou, which can also be applied to tack the contou of othe objects. Hand contou tacking in this pape includes fou steps, as follows (the famewok shows in Fig.3): Step 1: Detect cones on the hand contou in pevious fame. Step 2: Tack cones on the base of Lucas-Kanade (LK) optical flow in cuent fame. Step 3: Initialize GVF Snake with the cuve fitted by the cones. Step 4: Captue hand contou. And then, go to the step 1 fo pocessing next fame. In this pat, the detail of above-mentioned method is focused. A. Cone detection Cone ae good featues in eal-time tacking because they ae close to the oiginal object and can eliably match to the object points in the next fame. We need moe points with high cuvatue to insue the accuate and apid captue of the GVF Snake on those shap aeas. In this pape, cuvatue-diffeence-based cone detection is poposed to obtain enough cones. Calculate cuvatue k at evey point of the contou obtained by GVF Snake, and let ki () st be the cuvatue of the i point on the contou sequence: 545

5 xi () yi () xi () yi () ki () = (3-1) ( xi () + yi () ) whee xi (), xi () denote the fist and second deivatives of xi (), espectively; yi (), yi () denote the fist and second deivatives of yi, () espectively. The following is the detail of cuvatue-diffeence-based cone detection. Step 1: Afte hand egion detection, apply Canny edge detection to obtain a binay edge-map. Step 2: Check cones at the edge using SUSAN appoach and save to Sequence list S in ode, the total point numbe is denoted as N. Step 3: Inset the Nth point to the head of the list S and the fist point to the end of the list S. Step 4: Calculate kiat () the evey point in the list S, whee 1<i<N+1and initialize i to 2. Step 5: If i=n+1, end the pogam, else i++. Step 6: If ( ki ( 1) ki ( ) ( ki ( ) ki ( + 1)) < 0, go to step 7, else jump to step 5. Step 7: Save the point Si () as the cone points to Sequence list P, and loop execute step 5. B. Optical flow based cone tacking Optical flow is a good method fo tacking the points between fames, especially in the case of violent changes. But this method calculates all pixels of the whole image. Lucas and Kanade[31] poposed a spase optical flow method which is ealized by pocessing the pixels within a small neighbohood when tacking a point. In this pape, the LK method is used to fast tack the cones on hand contou. LK optical flow is based on the thee hypotheses, i.e. bightness constancy, tempoal pesistence and spatial coheence. The details of the method can be found in the wok [31]. C. GVF Snake on the base of cones and optical flow As we can see, thee ae many convex aeas and concave aeas on the hand contou. Theefoe, GVF Snake might take much iteation to convege to those aeas because of the distubance fom intenal enegy of snake and the initial position of contou. Usually, it is not helpful to egad the hand contou in pevious fame as the initial one in cuent fame. To addess this poblem, we poposed a contou initialization method which intepolates the cones tacked in cuent fame to fom an initial contou. The details ae as follows: Step 1: Detect Pt ( 1) = { P1, t 1, P2, t 1,, Pit, 1, PLt, 1} as the cones on hand contou at time t-1, 0 < i< L and L is the total numbe of the cones. Step 2: Tack Pt ( 1) by LK method to obtain Pt () = { P1, t, P2, t,, Pit,, PLt, }, 0 < i< L. Step 3: Delete abnomal points, denote point Pit, with the coodinate ( x i,t,y i,t ). If d = xit, xi 1, t + yit, yi 1, t + xi+ 1, t xit, + yi+ 1, t yit, > 50, Delete P it, fom Pt (). Step 4: Linealy fit a closed cuve C(t) by the cone points to insue the distance of the adjacent two points less than two-pixel wide. Step 5: Take C(t) as the initial cuve of GVF Snake at the time t, get the hand contou. Repeat step 1 to 5, hand contou is tacked. EXPERIMENTS In this section, we pesent the esult of hand contou tacking. Two video sequences with the size of unde the usual lighting condition in lab ae ecoded to test the efficiency of ou hand contou tacking algoithm. In two expeiments, we take the fist fame as the backgound fo quick motion detection. The two expeiments ae pefomed with the paamete value, α =0.1 and β =0.1, which detemine the tension and igidity of the snake espectively. The fist expeiment is pefomed on the sequence of video1, which contains a fast moving hand with nealy no hand shape change. Figs. 4a, 4b and 4c show the esult of the taditional contou tacking method, which takes the pevious contou as the initial contou in the cuent fame. In contast, Figs. 4d, 4e and 4f show the esult of the tacking method of GVF Snake on the base of cone and spase optical flow we 546

6 poposed. In the expeiment, the ed cuves pesent hand contous and the blue points ae the taced cones fom the pevious fame. (a) (b) (c) (d) (e) (f) Fig.4. Tacking on the video1:(a) (b) (c) the esult of the taditional contou tacking method;(d)(e)(f) the esult of ou appoach In the video1, the hand does not appea until 48th fame, and it completely moves into the camea view in the 70th fame. As shown in Figs. 4a and 4d, the algoithm poposed in section 2 automatically outlines hand contou as the initial one fo tacking. At the 75th fame, Fig. 4b shows the fingetip of index finge with a complex convex-concave shape cannot be captued by the taditional contou tacking, while it can be tacked pecisely in Fig. 4e. Afte that, the hand moves fast downwad. Fig. 4c shows the index finge infomation is lost because of the local minimal of GVF Snake. Fig. 4f illustates an accuate esult of tacking hand contous fo tacking cones fom the pevious fame. The second expeiment is pefomed on the sequence of video 2, which contains the skin-like colo object and a violent hand change. In Fig. 5, the esult hand contou is maked by ed cuve; the blue points ae the taced cones fom the pevious fame; the yellow points ae the cones detected using cuvatue-diffeence-based cone detection. (a)409th fame (b)426th fame (c)449th fame (d)484th fame (e)504th fame (f)544th fame Fig. 5. Hand contou tacking on the video2 Fig. 6. The 3D figue of mass cente tace of video2 The initial contou is fistly detected in the 409th fame based on the coss-sectional aea size, as shown in Fig. 5a. Along with hand moving into the video, the whole hand contou can be pecisely captued with the accuate contou of complex convex-concave shape in Fig. 5b. When the hand gestue changes smoothly, Fig. 5c shows we still can tack them well. And then the hand begins to move towads the pen in skin-like colo, some local minimal come foth. Because the initial contou in the cuent fame is a closed cuve fitted by the tacked cones, the hand contou can be sepaated fom the pen, as shown in Fig. 6d. Subsequently, violent gestue changes occu in the 504th and 544th fames, by using ou method, the hand contous can be obustly tacked as Fig. 5e and 5f illustate. Fig. 6 shows two 3D taces of mass cente fo video2 fom fame 1 to fame 710, whee x, y and nfame epesent x coodinate, y coodinate and the numbe of the fame, espectively. One tace is the eal mass cente tace of video1, manually labeled in magenta. The othe one is the mass cente tace calculated by the tacking method in this pape, pesented in blue. As can be seen, the contou of hand can be obust tacked 547

7 ove 710 fames, and the esults of ou tacking method ae in good ageement with the eal mass cente tace. CONCLUSION In this pape, we have poposed a novel automatic hand contou tacking appoach based on GVF Snake without the complex exta enegies fo high efficiency. The method consists of skin-colo GMM model based the flood fill algoithm, cuvatue-diffeence-based cone detection and spase optical flow. Ou appoach mainly addesses two poblems, one is about the sensitivity of GVF Snake to local minimal, and the othe one is the sensitivity to initial cuve. To educe the impact of local minimal, we segment hand egion by using motion infomation and skin-colo GMM model. Fo its sensitivity to initial contou, we combine GVF Snake with optical flow. The expeiments show a obust tacking of violent change hand gestue and the sensitivity to initial contou is loweed. Both expeimental esults indicate that this algoithm yields accuate and obust hand contou tacking and the system is fully automatic and bootstaps itself. It woks well by the eal-time tacking of the hand contou. Refeences [1] RT. Collins, Mean-shift blob tacking though scale space, Compute Vision and Patten Recognition, Poceedings IEEE Compute Society Confeence on; pp. II vol.232. [2] S. Bilal, R. Akmeliawati, MJE. Salami, AA. Shafie, EM. Bouhabba, A hybid method using haa-like and skin-colo algoithm fo hand postue detection, ecognition and tacking, Mechatonics and Automation (ICMA), 2010 Intenational Confeence on; pp [3] P. Zhigeng, L. Yang, Z. Mingmin, S. Chao, T. Xing, et al, A eal-time multi-cue hand tacking algoithm based on compute vision, Vitual Reality Confeence (VR), 2010 IEEE; pp [4] M. Kass, A. Witkin, D. Tezopoulos, Snakes: active contou models, Poceedings - Fist Intenational Confeence on Compute Vision. London, Engl: IEEE; pp [5] X. Chenyang, JL. Pince, Snakes, shapes, and gadient vecto flow, Image Pocessing, IEEE Tansactions on 1998,7: [6] Z. Zhou, J. Wu, W. Yang, et al, Face contou tacking based on mean shift and GVF Snake, Infomation Technology Jounal, 2013, 12(10): [7] F. Zhang, X. Zhang, K. Cao, et al, Contou extaction of gait ecognition based on impoved GVF Snake model, Computes & Electical Engineeing, 2012, 38(4): [8] Q. Li, Y. Luo, D. Xiao, Video object contou tacking using impoved dual-font active contou, Computational Intelligence. Spinge Belin Heidelbeg, 2006: [9] K. Takaya, Tacking a video object with the active contou (snake) pedicted by the optical flow, 2008 Canadian Confeence on Electical and Compute Engineeing, Vols 1-4; pp [10] R. Wang, Y. Wang, J. Zhou, et al, Active contous with neighbohood-extending and noise-smoothing genealized gadient vecto flow extenal foce, Infomation Technology and Applications (ITA), 2013 Intenational Confeence on. IEEE, [11] M. Niethamme, A. Tannenbaum, Dynamic geodesic snakes fo visual tacking, Compute Vision and Patten Recognition, CVPR Poceedings of the 2004 IEEE Compute Society Confeence on. IEEE, 2004, 1: I-660-I-667 Vol. 1. [12] Y. Rathi, N. Vaswani, A. Tannenbaum, et al, Paticle filteing fo geometic active contous with application to tacking moving and defoming objects, Compute Vision and Patten Recognition, CVPR IEEE Compute Society Confeence on. IEEE, 2005, 2:

8 [13] Z. Zhou, J. Wu, D. Wu, Human Body Contou Tacking Based on GVF-snake and DESO*, Jounal of Computational Infomation Systems, 2013, 9(18): [14] Y. Yang, Z. Wang, M. Zhang, et al, Skin Region Tacking using Hybid Colo Model and Gadient Vecto Flow, Machine Vision and Human-Machine Inteface (MVHI), 2010 Intenational Confeence on. IEEE, 2010: [15] S. Zhao, X. Song, W. Tan, et al, A novel appoach to hand gestue contou detection based on GVF Snake model and skin colo elliptical model, Compute Application and System Modeling (ICCASM), 2010 Intenational Confeence on. IEEE, 2010, 5: V5-381-V [16] J. Shi, C. Tomasi, Good featues to tack, Compute Vision and Patten Recognition, Poceedings CVPR'94., 1994 IEEE Compute Society Confeence on. IEEE, 1994: [17] J. Cheng, R. Xue, W. Lu, et al, Segmentation of medical images with Canny opeato and GVF snake model, Intelligent Contol and Automation, WCICA th Wold Congess on. IEEE, 2008: [18] J. Cheng, C. Liu, Image segmentation with GVF Snake and cone detection, Compute Science and Softwae Engineeing, 2008 Intenational Confeence on. IEEE, 2008, 1: [19] VP. Malagi, GG. SivaSankai, Object ecognition based on GVF and SUSAN in wieless senso netwok, Signal and Image Pocessing (ICSIP), 2010 Intenational Confeence on. IEEE [20] E. Jayabalan, A. Kishnan, Object Detection and Tacking in Videos Using Snake and Optical Flow Appoach, Compute Netwoks and Infomation Technologies. Spinge Belin Heidelbeg, 2011: [21] G. Qiaoyun, X. Xuemei, A. Li, Q. Mo, Computation of eal-time optical flow based on cone featues, Intelligence Infomation Pocessing and Tusted Computing (IPTC), 2010 Intenational Symposium on; pp [22] BD. Lucas, T. Kanade, Iteative image egistation technique with an application to steeo vision, Poceedings of the 7th Intenational Joint Confeence on Atificial Intelligence. Vancouve, BC, Can; pp [23] HL. Ribeio, A. Gonzaga, Hand Image Segmentation in Video Sequence by GMM: a compaative analysis, Compute Gaphics and Image Pocessing, SIBGRAPI'06. 19th Bazilian Symposium on. IEEE, 2006: [24] MJ. Jones, JM. Rehg, Statistical colo models with application to skin detection. Intenational Jounal of Compute Vision 2002,46: [25] SM. Smith, JM. Bady, SUSAN A new appoach to low level image pocessing, Intenational jounal of compute vision, 1997, 23(1):