A Machine Vision based Gestural Interface for People with Upper Extremity Physical Impairments

Transcription

1 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 1 A Machine Vision based Gestual Inteface fo People with Uppe Extemity Physical Impaiments Haiong Jiang, Badley S. Duestock, Juan P. Wachs Abstact A machine vision based gestual inteface was developed to povide individuals with uppe extemity physical impaiments an altenative way to pefom laboatoy tasks that equie physical manipulation of components. A colo and depth based 3D paticle filte famewok was constucted with unique desciptive featues fo face and hands epesentation. This famewok was integated into an inteaction model utilizing spatial and motion infomation to deal efficiently with occlusions and its negative effects. Moe specifically, the suggested method poposed solves the false meging and false labeling poblems chaacteistic in tacking though occlusion. The same featue encoding technique was subsequently used to detect, tack and ecognize uses hands. Expeimental esults demonstated that the poposed appoach was supeio to othe state of at tacking algoithms when inteaction was pesent (97.52% accuacy). Fo gestue encoding, dynamic motion models wee ceated employing the dynamic time waping (DTW) method. The gestues wee classified using a Conditional Density Popagation (CONDENSATION)-based Tajectoy Recognition (CTR) method. The hand tajectoies wee classified into diffeent classes (commands) with a ecognition accuacy of 95.9%. In addition, the new appoach was validated with the one shot leaning paadigm with compaable esults to those epoted in 12. In a validation expeiment, the gestues wee used to contol a mobile sevice obot and a obotic am in a laboatoy chemisty expeiment. Effective contol policies wee selected to achieve optimal pefomance fo the pesented gestual contol system though compaison of task completion time between diffeent contol modes. Index Tems Gestue ecognition, paticle filte, dynamic time waping, CONDENSATION, one shot leaning. A I. INTRODUCTION Ssistive technologies is about finding new ways to engage cutting-edge technologies in suppot of individuals with physical and/o cognitive impaiments. The development of technologies elying on high usability pinciples, exploited new communicational channels such as eye blinking, voice, Manuscipt eceived August, 12. This wok is patially funded by the National Institutes of Health though the NIH Diecto's Pathfinde Awad to Pomote Divesity in the Scientific Wokfoce, gant numbe DP4- GM Haiong Jiang is with School of Industial Engineeing, Pudue Univesity, West Lafayette, IN 4797, USA ( jiang115@pudue.edu). Badley S. Duestock is with School of Industial Engineeing and Weldon School of Biomedical Engineeing, Pudue Univesity, West Lafayette, IN 4797, USA ( bsd@pudue.edu). Juan P. Wachs is with School of Industial Engineeing, Pudue Univesity, West Lafayette, IN 4797, USA ( jpwachs@pudue.edu). hand gestues, sip and puff, and electomyogam (EMG) as effective contol modalities [1]. These channels have led to ingenious intefaces in suppot of the disabled [2], such as intelligent wheelchais systems, home medical alet systems, and assistive obotic contol, to mention a few [3, 4, 5]. These intefaces offe additional degees of mobility and contol which wee not possible pevious to these developments, leading to a highe life quality and sense of independence. Among all these inteaction channels, hand gestues is a valuable altenative since it does not equie to have the use tetheed though cables o sensos, and it only equies leaning a few customized gestues fo a given task. In paticula, uppe extemity gestue contol can seve as an impotant human compute inteaction (HCI) modality fo individuals with quadiplegia who lack hand fine moto skills. Fo instance, uppe limb gestue contol equies less tageting accuacy than joysticks, the mouse, and othe continuous input devices. Likewise, the option to employ eithe continuous o discete input contol modes educes the effot equied fo individuals with quadiplegics to pefom navigational opeations [6]. Unlike voice contol, gestue contol is effective in noisy envionments [7]. In addition, fo most of the cases, individuals with quadiplegic can only use goss moto function instead of fine moto function to pefom cetain tasks [8]. Apat fom othe common modalities, such as keyboad and joystick that equie fine moto contol to hit a key o move and twist a handle, uppe extemity gestue contol only equies goss moto function fo tageting and navigational tasks [9]. Lastly, hand gestue based HCI is unencumbeed because it does not equie the use to diectly contact o wea sensos as sip-and-puff and EMG based systems [1, 11]. While not evey individual with uppe extemity mobility impaiments can use hand gestue contol eliably, fo those who ae able to move thei ams to some degee, gestue-based HCI can be seen as a pomising altenative o complement to an existing contol modality. In ou pevious wok [12], a pototype of gestue ecognition based inteface was developed fo people with uppe extemity mobility impaiments. In the cuent manuscipt, the tacking algoithm was geatly impoved and compaed with five state-of-at algoithms to demonstate a bette tacking pefomance. Futhe, moe expeimental esults wee povided with subjects with uppe extemity mobility impaiments and one shot leaning was employed to allow instant customization of the gestual system. Face and hand tacking unde fequently self-occlusion was modeled as

2 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 2 a multi-object tacking (MOT) poblem. This poblem is challenging since hands ae non-igid objects and thei fom vaies among individuals, while pefoming a cetain candidate gestue. Additionally, since the appeaance of the left and ight hand ae simila fo the same individual, tackes can focus on one hand o exchange positions when the hands ae too close to each othe. In this pape, an integated appoach was poposed to tackle the challenging poblem of tacking unde self-occlusion. A. Related Wok Often, hand gestue ecognition involves segmentation of the hands, tacking them though occlusion, and the classification of hand s dynamic tajectoies and static pose. Fo vision-based eal-time gestue based intefaces fo assistive technologies, obustness is a citical equiement [13] fo its adoption. Fo hand segmentation, a commonly used method is to back-poject the pe-built skin colo histogam model into new video fames. These methods ae likely to fail in tue wold conditions, whee illumination is uncontolled and the backgound is clutteed. Adding depth infomation can elax at some extent the poblem, by utilizing steeo vision [14] o othe depth commodity sensos, such as Kinect [15] o Leap Motion [16]. Face and hands tacking is a special case of MOT poblem. If gestues in the lexicon only cay tajectoy infomation, (the hand shape does not convey exta infomation), classical tacking appoaches can be adopted. Fo example, CAMSHIFT [17] and CONDENSATION [18] have been shown to successfully tack gestues; howeve they ae susceptible to lose the tacked objects when occluded by new objects, o when the scene illumination changes. Anothe widely used technique fo object tacking is paticle filtes [19]. Peez et al. [] integated colo-based appeaance models to a paticle filte famewok to enhance tacking unde complex backgound and occlusion, and then applied the paticle filte famewok to multiple objects tacking. Okuma et al. [21] futhe extended paticle filtes by incopoating a boosting detecto and enabling automatic initialization of potential multiple tagets. One poblem of these techniques is that the inteaction between the tacked objects (and occlusion) was not consideed pat of the main famewok. When the objects inteact one with the othe, occlusions occu fequently. Local motion infomation was incopoated into a colo-based paticle filte famewok by Kistan et al. [22] to solve the self-occlusion poblem though object tacking. Qu et al. [23] combined a joint state space epesentation with colo-based paticle filte and pefomed joint data association in a multi-object tacking scenaio. All the discussed algoithms so fa, attempted to solve the MOT poblem; howeve they pesented limited pefomance when tacking multiple non-igid simila objects. With the advent of Kinect and othe 3D sensos, hand o body tacking techniques in eal-time wee exploited. Eichne et al. [24] pesented a technique to estimate the body layout of humans by using still images. Thei appoach is capable of estimating uppe body pose in highly uncontolled envionment. Futhe, Yang et al. [25] descibed a method to estimated human pose fom static images using body pat models. By using the depth infomation, Shotton et al. [26] poposed a method to pedict 3D positions of body joints fom a single depth image. They solved the pose estimation poblem though a simple pe-pixel classification poblem. A simila method is also used by OpenNI fo human body skeleton tacking. One poblem of these skeleton-based tacking methods is that they wok well when uses ae standing with thei extemities extended, but suffe sudden pefomance degadation fo seated uses with contacted limbs, as often occu with quadiplegic individuals. Only colo and depth infomation captued fom Kinect wee adopted fo hand tacking by Oikonomidis et al. [27]. They pesented a method to tack the full aticulation of two hands that inteact with each othe in an uncontolled manne. This method is effective fo static gestue ecognition; howeve, the computation cost is excessive which affects its eal-time extension fo gestue tacking. One of the most widely used techniques fo gestue ecognition is Hidden Makov Models (HMM) [28, 29, 3]. Common poblems with HMM appoach consist of finding the optimal paametes set (e.g. initial pobabilities) and tajectoy spotting fo gestue tempoal segmentation. Black and Jepson [31] poposed a CONDENSATION-based tajectoy gestue ecognition algoithm that can obtain less sensitive paametes set and achieve obust tacking, yet gestue tempoal segmentation was not fully addessed. Alon, et al. [32] applied the dynamic time waping (DTW) appoach to gestue ecognition and look at sub-gestues composition to solve the tempoal segmentation poblem (also known as spotting ). Inteaction between hands was not specifically tackled. Recently, a new type of challenge was attacted the attention of the gestue ecognition community the One Shot Leaning Challenge [33]. The one shot leaning [34] consists of leaning a gestue categoy by only obseving one instance of that gestue, simila to how humans lean. In this context, Wu et al. [35] adopted the extended-motionhistogam image fo motion featue epesentation and applied it to segment and classify hand gestues. Yang et al. [36] poposed discoveing high level sub-actions by clusteing optical flow in fou dimensions (RGB-D). In ou wok, one shot leaning povides an inteesting test-bed to demonstate the obustness of ou appoach, compaed with the state of the at [37]. In the cuent pape, we also extended ou method to obotic contol. One advantage of using hand gestues to contol obots is that it povides a natual way fo navigational tasks by sending navigational infomation (e.g. left, ight, fowad and backwad commands [38]). B. Outline of Ou Appoach In this pape, an inteaction model was incopoated to the colo histogam based paticle filte famewok to tack hands though inteaction and occlusion. A pocedue was poposed to ceate dynamic motion models by DTW method and classify input gestue tajectoies using the CONDENSATION

3 Time IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 3 A. Foegound Segmentation Depth Colo Depth Theshold Foegound Mask B. Face and Hand Detection and Tacking Face Detection Skin Colo Detection Hand Extaction Detect Face and Hands Paticle Filteing C. Hand Tajectoy Classification Host D. Robotic Contol Policies TutleBot FANUC Input Tajectoies Command Gound Tuth Motion Models Input Tajectoies (CONDENSATION) Fig. 1. System Oveview. algoithm. The system was integated in a simplistic yet obust fashion by combining CONDENSATION algoithm with an inteaction model-based paticle filte, which makes it suitable fo human obot inteaction in assistive technologies. The contibution of this pape is thee-fold: (1) pove the effectiveness of hand gestues as an altenative modality fo individuals with mobility impaiments by both subjective explanation and quantified esults; (2) solve the fequently hand gestue inteaction and occlusion poblem though integation of colo and 3D spatial infomation as an inteaction model; (3) new gestues can be ceated and leaned though the one shot leaning paadigm, leading to an almost effotless taining pocess (a necessay attibute fo subjects with sevee spinal cod injuies). The pape is oganized as follows: In Section II, the achitectue is pesented fo the gestue ecognition system. In Section III, the appoach suggested to tack and ecognize dynamic hand gestues is discussed in details. In Section IV, compaative tests and esults ae pesented, Section V discusses and concludes the pape, and Section VI pesents futue wok. II. SYSTEM ARCHITECTURE The achitectue of the poposed system is illustated in Fig. 1. Eight gestues wee selected to constitute the gestue lexicon which in tun was used to contol the obots. The machine vision based gestual system included fou pats: foegound segmentation, face and hand detection and tacking, hand tajectoy classification, and obotic contol policies. Those pats wee descibed in the following sections. A. Foegound Segmentation In foegound segmentation section, the backgound was uled out fom the captued fames and the whole human body was kept as the foegound. B. Face and Hand Detection and Tacking Face and hand detection was to initialize the position of the face and hands fo the tacking phase. Afte initialization, both face and hands wee tacked though video sequences by paticle filte method. C. Hand Tajectoy Classification Hand tacking esults wee segmented as tajectoies, compaed with motion models, and decoded as commands fo obotic contol. D. Robotic Contol Policies The commands decoded by gestue ecognition esults wee sent to contol the mobile obot and the obotic am. III. GESTURE RECOGNITION A. Foegound Segmentation Initially, the use s body was teated as a foegound object in ode to detect the use s movements. Two steps wee used to segment the foegound (efe to algoithm 1 in TABLE I). In the fist step, the sensed image assessed by a Kinect [15] senso was thesholded using depth infomation. The depth value of each pixel was defined as D(i, j) with i and j indicating the hoizontal and vetical coodinates of the pixel in each fame of the video sequence. An example of a depth image is shown by Fig. 2(a), whee the distance between objects and the depth senso was mapped to intensity levels. The neae the object was to the senso, the lage the intensity was. Two absolute depth thesholds (a low theshold T DL and a high theshold T DH) wee custom set by the use accoding to thei elative distance to the depth senso. T DL was set to no less than a constant which was the minimum distance that can be egisteed by the depth senso (due to its physical limitations). T DH was set to be the maximum distance that can be eached by the use while seated in a wheelchai 1. In this pape, T DL and T DH wee set to be.4m and m to achieve an 1 These values ae selected since they esulted in the best pefomance; othe thesholds can be used and the impact on the oveall pefomance is likely to be negligible.

4 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 4 optimal pefomance fo segmentation. A mask image (Fig. 2(b)) was geneated by keeping the pixels with a depth value between the two thesholds while discading the othes. In the second step, the egion (blob) with the lagest aea (denoted ast SH ) was extacted fom the mask image. All the emaining blobs with an aea smalle than T SH wee discaded (Fig. 2(c)). If the extacted egion contained an object that wee not pat of the use s body, it would be discaded in a late stage since tacking was achieved based on both colo and spatial infomation. TABLE I FOREGROUND SEGMENTATION ALGORITHM Algoithm 1: Foegound Segmentation Input: Low depth theshold T DL; High depth theshold T DH; pixel value of depth Image D(i, j); Output: pixel value of mask image D 1(i, j); pixel value of foegound mask image D 2(i, j). D 1(i, j) = { 1: T DL D(i, j) T DH : othewise T SH = max(aea(b i)) //B i is the ith blob in the mask image D 1 1: D D 2 (i, j) = { 1 (i, j) B i & Aea(B i ) == T SH : othewise spatial infomation was used to tack the face and hands though video sequences. A detailed desciption of the paticle filte algoithm was illustated in [, 41, 42]. The equation of paticle filteing is given as in (1): p(x t Z 1:t ) = k p(z t X t ) p(x t X t 1 ) p(x t 1 Z 1:t 1 )dx t 1 (1) whee X t is the pocess state at time t, Z 1:t = {Z 1, Z t } denotes the set of obsevations fom time 1 to t, p(x t Z 1:t ) and p(z t X t ) expesses the posteio and pio distibution at time t, p(x t X t 1 ) is the tansition pobability of the system at state X t given that the pevious state was X t 1, and k is a nomalization facto to nomalize the sum of all posteio pobability to 1. In the paticle filte algoithm, N weighted paticles can be used to appoximated the posteio as: p(x t 1 Z 1:t 1 ) {X t 1, ω t 1 } N =1, whee ω t 1 denotes the weight of the paticle at time t-1. Afte popagation, the tacke output at time t can be appoximated by the expectation of the pocess state: X t E[X t Z 1:t ] = N ω =1 t X t. Thus, (1) is conveted to (2): p(x t Z 1:t ) k p(z t X t ) N ω t p(x =1 t X t 1 ) (2) (a) (b) (c) Fig. 2. Foegound Segmentation. (a) Depth image; (b) Depth theshold mask; (c) Foegound segmentation mask. B. Face and Hand Detection In this section, the centoids of the face and hand egions wee extacted to initialize the tacking stage. Two 3D histogams - a skin and a non-skin colo histogam wee ceated using Compaq database [39] and HSV colo space to achieve highe obustness fo skin colo detection (efeed to [12] fo a detailed desciption). The mask image obtained fom histogam back-pojection is shown as in Fig. 3(a). To obtain the hand egions without the face, a face detecto [4] was adopted (Fig. 3(c)) to emove the egion fom the taget image. Two lagest blobs in the taget image wee then selected as hand egions (Fig. 3(b)). The centoids of the hands wee obtained by computing the fist moment of the two blobs. This hand detection pocedue was only used to povide automatic initialization to the paticle filte tacking pocedue. Aftewads the hands positions wee continuously tacked by the paticle filte. (a) (b) (c) Fig. 3. Face and hand detection. (a) Skin colo detection; (b) Hand extaction; (c) Face and hand localization. C. Face and Hand Tacking A 3D paticle filte famewok based on colo, depth and The paticles wee initialized by using the centoids of face and hands calculated in section III. B. The paticle filte tacking pocess consists of thee main phases: pedicting, measuing and e-sampling. In the poposed system, fo the pedicting phase, a second ode auto-egessive (AR) model (as in (3)) [, 41] was selected to model the dynamic motion of each paticle: X t = A 1 (X t 1 X ) + A 2 (X t 2 X ) + X + Bν t (3) Whee ν t ~N(, Σ) is a Gaussian distibution with zeo mean and vaiance matix Σ, X is the oiginal paticle coodinate, A 1, A 2, and B ae the optimal paamete matices that can best match the eal motion of the tacked object, X t is the state of the paticle at time t. In this pape, a 3D paticle filte tacking was adopted. The state of paticle at time t is witten as: X t = [x t, y t, z t, s t, x t 1, y t 1, z t 1 ], whee s t is the scale of object at time t, x t, y t, z t ae the 3D coodinates of paticle at time t, and x t 1, y t 1, z t 1 ae the 3D coodinates of paticle at time t-1. Fo the measuing phase, the selection of the obsevation model detemines the weight of the paticles. Many appeaance-based models, such as contou, edge, piece-wise, etc, wee used in object tacking. Colo-based pe-pocessing using HSV space can facilitate the extaction of the afoementioned featues fo face and hands tacking. As explained ealie, the initial phase of the face and hands wee detemined by the combination of depth-based thesholding and image pocessing techniques. The extacted face and hands egions wee used to compute the efeence HSV histogam models (H * f, H * h1, and H * h2 ) fo tacking initialization. Duing the e-sampling phase, each paticle, assigned in the pedicting phase, was eweighted by the obsevation likelihood function. Fo evey hypothesized face

5 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 5 o hand location of a paticle, the candidate histogams wee computed as H f, H h1, and H h2. The Bhattachayya distance [] D was used to measue similaity between efeence and candidate histogams as (4): 1 D i (H, H ) = [1 H i H 2 i ] (4) whee H i = H f, H h1, o H h2 and H i = H f, H h1, o H h2. The obsevation likelihood function can be witten as (5): p(z t X t ) exp ( λ 1 (D i ) 2 ) (5) whee λ 1 measues the vaiance of the HSV histogam. (5) can be e-witten by adding a nomalization facto k to nomalize the sum of all paticles weight to 1, obtaining (6): p(z t X t ) = k exp ( λ 1 (D i ) 2 ) (6) D. Hand Tacking Though Inteaction and Occlusion Colo-based paticle filte tacking was effective fo multiple independent objects tacking when the objects did not inteact o occlude each othe. Howeve, if inteaction o occlusion occus, multiple independent paticle filtes can be used. Standad multi-object tacking (MOT) with inteaction and occlusion suffes fom the false meging and false labeling poblems [23]. The false meging poblem denotes the situation that the tacke shift fom the object being tacked to a diffeent object that has highe obsevation likelihood. Convesely, the false labeling poblem denotes the situation that the objects being tacked exchange thei labels afte inteaction o occlusion occued. In the poposed system, the face and both hands wee tacked. In this pape, two models wee constucted to solve false meging and false labeling poblems sepaately. The fist model was called the Competition Potential (CP) model. The idea of this model comes fom the Joint Makov andom fields (MRF) theoy [42]. The likelihood function fo CP model is defined as ψ 1 (X i,t, X j,t ), which epesented the paiwise inteaction potential of the MRF [43]. The second model is called Motion Consistency (MC) model. The likelihood function fo MC model is defined as ψ 2 (X i,t, X j,t ), which is based on the assumption that a paticle egion that has simila motion infomation to the pevious state of that paticle will have highe pobability than a paticle egion that has distinct motion infomation. Fo CP model, as in [43], we have p(x t X t 1 ) i,j E ψ 1 (X i,t, X j,t ) i,j E ψ 1 (X i,t, X j,t ). The paticle filte function (2) can be ewitten as (7): p(x t Z 1:t ) = k p(z t X t ) i,j E ψ 1 (X i,t, X j,t ) ω t 1 i p(x i,t X i,t 1 ) (7) The likelihood function fo CP model is then defined as: ψ 1 i,t (X i,t, X j,t ) = β 1 exp ( d(x i,t exp ( λ 3 d(x i,t, X j,t 1 ) 2 ) exp ( λ 2,X j,t ) 2 ) d z (X i,t λ 4 X j,t ) 2 ) (8) whee d(x i,t, X j,t ) denotes the 2D Euclidean distance metic between two objects, d(x i,t, X i,t 1 ) epesents a distance metic between the pevious and cuent centoid of object i, d z (X i,t X j,t ) epesents the diffeence of depth value between two objects, and β 1 is a nomalization facto so the sum of all paticles weight is 1. MC model was used to solve the false labeling poblem. The 3D motion infomation was incopoated into the oiginal likelihood function to incease the obustness of the method. We adopted a compact expession of the likelihood function simila as in [23], which integated the magnitude and diection infomation of motion as (9). Instead of using 2D, 3D motion featues ae used to compute the motion infomation of the hand movement. The likelihood function fo the MC model is defined as: ψ 2 (X i,t i,t, X j,t ) = β 2 exp( λ 5 (θ t ) 2 ) exp ( λ 6 (A t A ef,t ) 2 ) (9) whee A t and A ef,t epesent the nom of 3D motion vecto and efeence motion vecto (can be computed by the diffeence of the cuent and the pevious 3D position vecto) of paticle at state t, espectively., θ t is the angle between the 3D motion vecto of paticle and the efeence vecto and β 2 is a nomalization facto to nomalize the sum of all paticles weight to 1. This likelihood function assumes that a paticle egion that has a simila motion to the pevious state will have a lage weight than one with a diffeent motion. When the objects obsevations do not inteact with each othe, the appoach suggested behaves as if multiple independent tackes wee applied to the objects (Fig. 4(a)). Howeve, when the objects obsevations inteact (e.g. patial o complete occlusion occus), the conventional paticle filte famewok is extended (Fig. 4(b), (c)). The decision of when the objects inteact is made based on the intedistance between the hands. When this distance is below a cetain theshold, the system switches to the inteaction model (Fig. 4). To find the optimal theshold, a histogam of the numbe of tacking fame eos at each distance is obtained (Fig. 5) and the theshold is selected so the tacking eos ae minimized when the inteaction model is activated. Note, a damatic decease of eo, at the distance aound 5 pixels at which hand inteaction fequently occued. The theshold T was detemined accoding to the distibution of eos in the histogam. The extension models wee given though (8) and (9). The paametes λ 1, λ 2, λ 3, λ 4, λ 5, and λ 6 in (6), (8) and (9) wee optimized by utilizing a neighbohood seach method [38]. The algoithm fo hand tacking duing inteaction and occlusion is shown by TABLE II.

6 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 6 Object i D(i, j)>t Object j Object i Object j D(i, j)<=t Object i Object j (a) (b) (c) Fig. 4. Dynamic motion analysis. (a) independent object tacking; (b) inteaction model added; (c) objects occlusion occus. to pefom a gestue by paticipants with uppe mobility impaiments. An eight-gestue lexicon (see Fig. 6) was then constucted by analyzing the Bog Scale esults collected fom the subjective ankings and selecting those coesponding to the least equied effot. A detailed desciption of the pocess fo gestue lexicon constuction can be efeed to [45]. Numbe of Eo Tacking Fames Distance Between Left and Right Hand (pixels) Fig. 5. Numbe of eo tacking fames fo each distance befoe and afte the addition of inteaction model. TABLE II HAND TRACKING THROUGH INTERACTION AND OCCLUSION Algoithm 2: 3D Paticle Filte tacking Input: Refeence HSV histogam models H f, H h1, and H h2 ; Optimal paamete λ 1, λ 2, λ 3, λ 4, λ 5, λ 6. Output: Centoids and the associated bounding box of the face and hands. 1. Initialize: //Initialize paticle states and weight fo face and both hands as: x i = x, ω i = 1, whee i = 1,,n n 2. Pedict, Measue and Resample: //Select k; fo i=1,2,3 //(1-face, 2-ight hand, 3-left hand) fo =1 to N x i,t = A 1 (x i,t 1 x i ) + A 2 (x i,t 2 x i, ) + x i, + Bν t //Compute candidate histogams H D i (H, H ) = [1 H H 2 ] //Calculate the weight: ω i,t = k exp ( λd 2 i,t ) end fo Nomalize the weights and esample the paticles Estimate x i,t = N ω =1 i,t x i,t //Check inteaction if inteactions happens fo object i and j fo q=1,,n //compute inteaction likelihood ψ1 and ψ2: Compute ψ q 1 (X q i,t i,t, X q j,t ) and ψ q 2 (X q i,t i,t, X q j,t ) using (8) and (9) //Calculate the weight: ω q i,t = ω q i,t ψ q q 1 ψ i,t 2 i,t end fo Nomalize the weights and esample the paticles. N Estimate x i,t = ω =1 i,t x i,t end if end fo E. Gestue Lexicon Independent Object Tacking With Inteaction Model A gestue lexicon was designed such that uses will physical impaiments can pefom the gestues with minimal effot. These gestues wee found though a seies of inteviews conducted with subjects with uppe mobility impaiments. Bog Scale [44] was used to ank the physical stess equied 1 (a) (b) (c) (d) (e) (f) (g) (h) Fig. 6. Gestue lexicon. (a) upwad; (b) downwad; (c) ightwad; (d) leftwad; (e) counte-clockwise cicle; (f) clockwise Cicle; (g) S; (h) Z. F. Hand Tajectoy Classification Fo each fame in the video sequence, the centoids of the face and hands wee obtained fom the tacking stage. The motion model fo each gestue tajectoy was ceated based on the data collected fom gestues pefomed by ten subjects. Two of the pool of ten subjects wee quadiplegic due to a cevical spinal cod injuy. Even though the tajectoies fo each gestues pefomed by diffeent subjects o the same subject in diffeent instances may look simila, the pecise duation of each sub-tajectoy within the tajectoy wee diffeent. To nomalize the tajectoies (tempoal alignment), dynamic time waping (DTW) was employed [46]. The velocities components in hoizontal, vetical and depth diections of both hands wee selected as the featue components fo each motion model [41]. The pocedue to constuct the motion models is descibed in ou pevious wok [12]. The CONDENSATION (Conditional Density Popagation) algoithm [31] was employed to classify hand gestue tajectoies in the lexicon (as in Fig. 6). It employs a set of weighted samples instead an equation to fit the obseved data. The oiginal algoithm in [31] was extended to wok fo two hands. The oiginal expession S t = (μ, φ, α, ρ) (the state at time t) was extended to: S t = (μ, φ i, α i, ρ i ) = (μ, φ ight, φ left, α ight, α left, ρ ight, ρ left ) (1) whee, μ is the index of the motion models, φ is the cuent phase in the model, α is an amplitude scaling facto, ρ is a time dimension scaling facto, and i {ight hand, left hand}. The gestues in the lexicon (as in Fig. 6) wee spotted using a est position gestue (when the subjects put thei hands on the am est (neual position) with no hands movement). A dynamic motion model was ceated fo the est position gestue. The segment between two ecognized discontinuous est position gestues is teated as a spotted gestue.

7 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 7 Method False Meging (68 Fames) G. Gestue Customization (One Shot Leaning) One of the objectives of ou pototype follows the came as you ae paadigm [13], whee new gestues can be leaned by the system automatically o by obseving only one instance of it. The eason fo this is to educe the level of effot involved in the taining phase of the system. In this section, we validated ou appoach in the context of one shot leaning to assess the ability of the system to genealize leaning fom vey few obsevations. A Savitzky-Golay smoothing filte [47] was added to smooth the 3D tajectoies duing the ceation of the motion models. Two new gestues (see Fig. 7) wee added to the lexicon to offe anothe degee of navigational contol. (a) (b) Fig. 7. Extended lexicon. (a) clockwise cicle in hoizontal plane; (b) counte-clockwise cicle in hoizontal plane. IV. EXPERIMENTS AND RESULTS A. Expeiment 1: Hand Tacking Pefomance A dataset of 16 videos (4 subjects x 4 activities) was used to evaluate the poposed tacking algoithm. The videos wee captued with a Kinect camea at 3Hz using an image size of Among the 4 subjects, thee wee able-bodied individuals and one was an individual with Cevical-6 level quadiplegia. The 4 activities pefomed by the subjects wee: (a) holding a cup, (b) clapping hands, (c) moving one hand up and down (to occlude the othe hand), (d) otating two hands fowad and backwad (to occlude each othe). The total numbe of fames fo all the videos was 68, while the total numbe of inteactions between the two hands was 157. The total numbe of fames of each video coesponding to each of the fou activities was: 93, 223, 13, and 16, espectively (two sample sequences ae shown in Fig. 17 and 18 in appendix). The gound tuth position of the left and ight hands in each video was povided by manually hand labeling. The local likelihood p(z t i x t i ) was calculated using the 3D colo histogams and two inteaction models as the algoithm mentioned in TABLE II. The pefomance of the poposed method competition potential and motion consistency (CPMC) - was compaed to othe existing methods such as: (i) Makov Chain Monte Calo (MCMC) based paticle filte TABLE III HAND TRACKING PERFORMANCE False Labeling (157 Inteactions) Tacking Accuacy (%) Paticle Numbe Numbe of Body Pats MCMC [41] MI [23] ETH (colo) [24] ETH (depth) [24] Body Pat (colo) [25] Body Pat (depth) [25] Kinect Skeleton [48] CPMC(Poposed) tacking [43], (ii) Magnetic-Inetial based paticle filte tacking [23], (iii) ETH skeleton tacking based on colo o depth fames [24], (iv) Body-pats tacking based on colo o depth fames [25], and (v) Kinect OpenNI SDK skeleton tacking [48]. Fo the method (i), (ii), and the poposed method, 1 paticles wee used fo face and each hand tacking. Fo methods (iii), (iv) and (v), the numbe of body pats (segments of a human body, i.e. hand, head, leg, and pat of the ams) being tacked wee: 6, 26, and 16, espectively. Fo the method (iii), only the uppe body pats wee tacked. 6 pats wee used. Since the focus was on hand tacking, the esults of two hands inteaction fo all the activities ae shown as in TABLE III. The tacking pefomance of these algoithms was evaluated by employing thee metics: false meging, false labeling and tacking accuacy. The false meging is defined as the situation whee the tacke of one hand occupies 8% of the aea of the othe hand. The false labeling is defined as the situation whee the tackes of both hands change positions duing/afte inteaction o occlusion. The tacking accuacy is defined by (11): Tacking Accuacy total numbe of (tue positives + tue negatives) = total nume of tacked fames (11) whee a tue positive is defined as the situation wheeas a taget object is pesent and the tacke was able to find it. Tue negatives ae instances whee the taget object is not pesent and the tacke also ageed that the object was absent [47]. TABLE III shows that the poposed algoithm (CPMC) exhibits the best pefomance fo the inteaction and occlusion conditions among the thee paticle based methods. Thee is a maginal decease in the algoithm speed. When thee is no inteaction o occlusion occuing, CPMC has the same speed as MCMC and MI appoaches. Compaing to the skeleton tacking [24, 48] and body pat tacking method poposed by [25], the poposed method obtained highe tacking accuacy. Since ou tageting use goup is individuals with uppe mobility impaiments, two challenges exist fo hand tacking in the poposed system that could not be tackled vey well by skeleton based o othe aticulated pose tacking method. One challenge is that the uses with uppe extemity mobility impaiments need to sit most of the time on a wheelchai and

8 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 8 pefom limited space hand gestues. Thei hands could be vey close to the body when they pefom the gestues. The tacking method based puely on depth infomation could easily lose tack when the hands ae so close to the toso [24, 25]. The second challenge is that most of the wheelchais have amest, which can be easily confused with human ams and hands. This can explain why the skeleton based tacking method ((iii) and (v)), and the aticulated pose tacking method (iv) does not wok well fo ou dataset. The values of the pefomance metics false meging and tacking accuacy vesus the distance between left and ight hands ae shown in Fig. 8, and Fig. 9. Fom Fig. 8, we can see that the poposed appoach outpefoms method (i), (ii), (iv), and method (iii) (with colo images). Additionally, the esult of the poposed appoach (1 false meging fame) wee vey close to the esults of method (iii) (with depth images) and method (v) (no false meging occued). In Fig. 9, the total numbe of false positive and false negative fames vesus the hands intedistance was pesented fo each method. This figue showed that the poposed appoach outpefomed all the othe state of at algoithms, since it displayed the fewest numbe of false positive and false negative fames among all the algoithms fo nealy all distances. Numbe of False Meging Fames Distance Between Left and Right Hand (pixels) Fig. 8. Numbe of false meging occued vs. hand distance. Numbe of False Positive and False Negative Fames MCMC MI ETH (colo) BodyPat (colo) BodyPat (depth) CPMC (Poposed) MCMC MI ETH (colo) ETH (depth) BodyPat (colo) BodyPat (depth) OpenNI KINECT CPMC (Poposed) Distance Between Left and Right Hand (pixels) Fig. 9. Numbe of false positive and false negative vs. hand distance. B. Expeiment 2: Gestue Recognition Pefomance The motion models wee constucted using the DTW algoithm. The velocities (3D diections) of ight and left hands wee used as the main featue components. The gestue lexicon in Fig. 6 was adopted, and those gestues wee used to ceate spatio-tempoal tajectoies that late wee classified by the gestue-based ecognition system. The system was validated by eight able-bodied subjects and two subjects with quadiplegia due to cevical spinal cod injuies aged aound The ten subjects pefomed all the gestues in the lexicon each ten times (8 gestues x 1 subjects x 1 epetitions). Ten sessions wee used fo coss validation fo each gestue (k-fold with k=1). In each session, 7 obsevations (8 gestues x 9 subjects x 1 epetitions) wee used fo taining and 8 gestues (8 gestues x 1 subject x 1 epetitions) wee used fo testing. This coss validation esulted in an aveage accuacy of 95.9%. A confusion matix was computed and shown by Fig. 1 (with a tempoal window size of w=19). Confusions occued when the subjects pefomed a gestue mistakenly in a single diection o not enough motion was exhibited as expected in othe diections. Othe cases of misclassification occued when two gestues shaed simila sub-tajectoies (i.e, counte clock and S gestues). Up Down Left Right Clock Counte S Z Up Down Left Right Clock Counte S Z Fig. 1. Confusion matix with window size of w=19. The ecognition pefomance fo the CONDENSATION algoithm with ou taining pocedues (CONDENSE) was compaed to fou othe existing state-of-the-at ecognition algoithms: (i) Basic motion [5]; (ii) Motion based PCA [51]; (iii) Dynamic time waping (DTW) [52], and (iv) Hidden Makov Model (HMM) [28]. Afte applying each gestue ecognition method to ou data set, the esults shown in TABLE IV wee obtained. The confusion matices fo the diffeent methods ae shown in Fig. 11, Fig. 12, Fig. 13 and Fig. 14 espectively. Method (i), (ii), and (iii) used motion infomation to ecognize hand gestues, while (iv) and the CONDENSATION method ecognized hand gestues by extacting and classifying hand tajectoies. The compaison esults demonstate a high ecognition accuacy fo the tajectoies classification based method. HMM based ecognition method can get compaable esults as the method used in ou pape. TABLE IV GESTURE RECOGNITION PERFORMANCE Method Basic PCA DTW HMM CONDENSE Accuacy (%)

9 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 9 Up Down Left Right Clock Counte Fig. 11. Confusion matix fo Basic motion method. Up Down Left Right Clock Counte S Z Fig. 12. Confusion matix fo PCA method (with 12 pincipal components). Up Down Left Right Clock Counte S Z Up Down Left Right Clock Counte S Z Fig. 13. Confusion matix fo DTW method. Up Down Left Right Clock Counte S S Z Fig. 14. Confusion matix fo HMM method. C. Expeiment 3: One Shot Leaning Pefomance One instance (one epetition by a subject) fo each gestue in the lexicon was used fo taining and emaining obsevations wee used fo testing. Ten sessions wee used fo coss validation fo each gestue (k-fold with k=1). In each session, 1 obsevation (1 gestues x 1 subjects x 1 epetitions) was used fo taining and 74 gestues (8 gestues Up Down Left Right Clock Counte S Z Up Down Left Right Clock Counte S Z Up Down Left Right Clock Counte S Z Z x 9 subject x 1 epetitions and 2 gestues x 1 subjects x 1 epetitions) wee used fo testing. This coss validation esulted in an aveage accuacy of 82.78%. A confusion matix was computed and is shown by Fig. 15 (with a tempoal window size of w=19). The ecognition accuacy found is compaable to those epoted in the ChaLean Competition [33] in 12 (fouth place in the competition). Up Z Down Left Right Clock Counte S Clock_H Counte_H Up Down Left Right Clock Counte S Z Clock_H Counte_H Fig. 15. Confusion matix fo one shot leaning (window size w=19). D. Expeiment 4: Robotic Contol Pefomance Since the fist emotely diven obotic am developed by Goldbeg et al. (1995) [53] fo gadening tasks (Telegaden), thee has been an extensive wave of emote labs enabling uses the ability to pefom lab expeiments without the need to physically attend them. Some examples include a wok-cell with a 6 axes obot fo conducting expeiments emotely [54]; a LEGO mobile obotic platfom fo expeimenting with autonomous navigation [55]; and a emote laboatoy on obotics was developed at Univesity of Siena called TeleTab [56]. A chemisty laboatoy based expeiment was pefomed by five subjects including two individuals with quadiplegia due to a cevical spinal cod injuy and thee able-bodied individuals. In the laboatoy case study expeiment, a mobile obot was contolled by the gestue algoithm to tanspot a beake to a position nea a obotic am. The obotic am was activated by the opeato to add a eagent to the beake and then, the mobile obot was bought back to its oiginal position. The gestues (a)-(h) (fom the lexicon in Fig. 6) wee used and mapped to the commands: change mode, obotic am action, go fowad, go backwad, tun left, tun ight, stop and enable obotic am. The two obots wee contolled by thee modes; discete, continuous and hybid mode (discete plus continuous mode). In discete mode, fo each issued command, the mobile obot moved a fixed incement of distance. While in continuous mode, the mobile obot esponded to a given command, until the stop command was issued. To switch between the discete and continuous mode one distinctive gestue ( upwad ) was used. In the expeiment, the discete, continuous and hybid (continuous plus discete) contol modes wee each tested five times by all subjects. The esulting aveage task completion times wee 241.8, and seconds, fo the discete, continuous and hybid mode, espectively (Fig. 16). Fom the esults, the completion time of discete mode took longe time

10 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 1 than the continuous. Continuous and hybid modes equie commands to be issued only when the obot needs to change diections o stop, theefoe fewe opeations wee equied fo continuous and hybid modes than fo discete mode fo the task obseved. Aveage Task Completion Time(sec) *** Discete Continuous Hybid Contol Mode Fig. 16. Aveage task completion time, unpaied t-test, p<.1. V. DISCUSSION & CONCLUSION Mean SD eo SEM eo A machine vision based gestual inteface was developed fo individuals with uppe extemity physical impaiments. Since skin and non-skin colo histogam models wee used to initialize the face and hands centoid, the pefomance of the system may be affected when the uses wea shot sleeves. In addition, it is expected that uses will be seated within the woking distance to ange specified by the Kinect senso. An inteaction model was incopoated into the colo-based paticle filte famewok fo hand tacking. When thee was no inteaction between the face and hands, multiple independent paticle filtes tacked the uses movements. When inteaction was pesent, the multiple independent paticle filte tackes wee combined with an inteaction model to solve false meging and false labeling poblems. A compaison between ou poposed appoach (CPMC), and five state of the at algoithms demonstated that ou appoach can achieve obust pefomance fo hand tacking though inteaction and occlusion conditions. The poposed tacking stategy can obtain significantly bette pefomance than the othe thee methods fo both false mege and false labeling poblems in hand tacking though inteaction and occlusion. Yet, impovements ae still need fo false labeling solving. A taining pocedue was poposed to obtain motion models fo each gestue in the lexicon. The CONDENSATION algoithm with the poposed taining pocedue was used and compaed with 4 othe ecognition algoithms to classify bimanual gestues. Results showed that HMM based ecognition methods may delive compaable esults to ou method. Thus, highe ecognition could be achieved by using tajectoies classification based method. The gestue ecognition algoithm designed was found to each a ecognition accuacy of 95.8%. One shot leaning was applied in this pape to customize gestues and educe the numbe of epetitions equied to tain/teach the system to a minimum (one obsevation). The esults obtained wee compaable to the state of at one shot gestue ecognition algoithms pesented in the ChaLean Challenge [33]. A laboatoy task expeiment was conducted, a typical biomedical lab pocedue with the help of two obots, which wee contolled though a gestual inteface. Subjects with uppe extemity physical impaiments can successfully use the machine vision based gestual inteface to contol the two obots. It was found that the poposed gestual inteface was obust enough to suppot the completion of this task fo subjects with uppe extemity mobility impaiments. In addition, thee modes of opeation wee compaed: discete, continuous and hybid. Results showed that the continuous mode equied the least aveage task completion time, while the discete contol mode equies the most. Theefoe, the authos ecommend to use continuous contol mode in geneal, and to use discete mode only when the obot is vey nea to the taget, fo pecise location and manipulation. VI. FUTURE WORK Futue wok fo this pape may include: (1) develop moe effective and obust algoithms to solve false mege and false labeling poblems of hand tacking though inteaction and occlusion. (2) extend the laboatoy task to incease the pool of paticipating uses. Ideally, uses with physical impaiments can paticipate and povide feedback about the usability, leaning and adaptability to the inteface suggested. APPENDIX The video sequences of two activities fo hand tacking though inteaction ae shown as in Fig. 17, and 18. Fame 467 Fame 474 Fame 479 Fame 489 Fame 491 Fame 495 Fig. 17. Hand tacking sequence fo clapping hands activity. Fame 228 Fame 236 Fame 24 Fame 255 Fame 257 Fame 26 Fig. 18. Hand tacking sequence fo moving one hand up and down activity.

11 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 11 ACKNOWLEDGMENT This wok is patially funded by the National Institutes of Health though the NIH Diecto's Pathfinde Awad to Pomote Divesity in the Scientific Wokfoce, gant numbe DP4-GM REFERENCES [1] Ahsan, M. R., EMG Signal Classification fo Human Compute Inteaction: A Review. Euopean Jounal of Scientific Reseach, pp (9). [2] Jacko, J. A., Human-Compute Inteaction Design and Development Appoaches. In: 14th HCI Intenational Confeence, pp (11). [3] Moon, I. H., Lee, M., Ryu, J. C., and Mun, M., Intelligent obotic wheelchai with EMG-, gestue-, and voice-based inteface. In: Intelligent Robots and Systems, pp (3). [4] Waltes, M., Macos, S., Sydal, D. S., and Dautenhahn, K., An Inteactive Game with a Robot: People s Peceptions of Robot Faces and a Gestue-Based Use Inteface. In: 6th intenational confeence on advances in compute-human inteactions, pp (13). [5] Bdiczka, O., Langet, M., Maisonnasse, J., Cowley, J. L., Detection Human Behavio Models fom Multimodal Obsevation in a Smat Home. IEEE Tansactions on Automation Science and Engineeing, pp (9). [6] Cook, M.C. and Mille Polga, J. (Eds.) (8) Cook & Hussey s Assistive Technologies: Pinciples and Pactice 3 d Edition, Missoui, Mosby Elsevie. [7] Muthy, G. R. S. And Jadon, R. S., A Review of Vision based Hand Gestue Recognition. Intenation Jounal of Infomation Technology and Knowledge Management, vol 2, No. 2, pp (9). [8] Debuse, D., Gibb, C., and Chandle, C., Effects of hippotheapy on people with ceebal palsy fom the uses pefspective: A qualitative study, vol. 25, No. 3, pp (9). [9] Steba, J. A., Roges, B. T., Fance, A. P., and Vokes, D. A., Hoseback iding in childen with ceebal palsy: effect on goss moto function. Developmental Medicine & Child Neuology, vol 44, No. 5, pp (2). [1] Kitto, K. L., Development of a low-cost sip and puff mouse. In: Poceedings of 16th Annual Confeence of RESNA. pp (1993). [11] Yin, Y. H., Fan, Y. J., and Xu, L. D., EMG and EPP-Integated Human- Machine Inteface Between the Paalyzed and Rehabilitation Exoskeleton. IEEE Tansactions on Infomation Technology in Biomedicine, vol 16, No. 4, pp (12). [12] Jiang, H., Wachs, J. P., Duestock, B. S., Facilitated gestue ecognition based intefaces fo people with uppe extemity physical impaiments. In: Poceedings in Patten Recognition, Image Analysis, Compute Vision, and Applications. Lectue Notes in Compute Science, pp (12). [13] Wachs, J., Kölsch, M., Sten, H. and Edan, Y., Vision-Based Hand Gestue Applications: Challenges and Innovations. Communication of the ACM, Cove Aticle, vol 54, No. 2, pp: 6-71 (11). [14] Li, Z., and Javis, R., A multi-modal gestue ecognition system in a Human-obot inteaction scenaio. In: Poceedings of the IEEE Intenational wokshop on obotic and sensos envionments, pp (9). [15] Suma, E. A., Lange, B., Rizzo, A., Kum, DM., and Bolas, M., FAAST: The Flexible Action and Aticulated Skeleton Toolkit. IEEE Vitual Reality, pp (11). [16] Leap Motion: [17] Badski, G.R., Compute vision face tacking as a component of a peceptual use inteface. In: Wokshop on applications of compute vision, pp (1998). [18] Isad, M., and Black, A., CONDENSATION: Conditional density popagation fo visual tacking. J. Intenational Jounal of Compute Vision, Vol. 29, pp (1998) [19] Maskell, S., Godon, N., and Clapp, T., A tutoial on paticle filtes fo online nonlinea/non-gaussian Bayesian tacking. IEEE tansaction on signal pocessing, pp (2). [] Peez, P., Hue, C., Vemaak, J., and Gangnet, M., Colo-based pobabilistic tacking, LNCS, Spinge, Heidelbeg, vol. 235, pp (2). [21] Okuma, K., Taleghani, A., Feitas, N., Little, J. J., and Lowe, D. G., A boosted paticle filte: multitaget detection and tacking. Euopean Confeence on Compute Vision (4). [22] Kistan, M., Pes, J., Kovacic, S., and Leonadis, A., A local-motionbased pobabilistic model fo visual tacking. Patten Recognition. vol. 42, No. 9, pp (9). [23] Qu, W., Schonfeld, D., and Mohamed, M., Real-time distibuted multiobject tacking using multiple inteactive tackes and a magnetic-inetia potential model. IEEE tansactions on Multimedia, vol. 9, No. 3, pp (7). [24] Eichne, M., Main-Jimenez, M., Zisseman, A., and Feai, V., Aticulated Human Pose Estimation and Seach in (Almost) Unconstained Still. ETH Technical Repot (1). [25] Yang, Y., and Ramanan, D., Aticulated Pose Estimation with Flexible Mixtue of Pats. IEEE Confeence on Compute Vision and Patten Recognition (CVPR), pp (11). [26] Shotton, J., Fitzgibbon, A. W., Cook, M., Shap, T., Finocchio, M., Mooe, R., Kipman, A., and Blake, A., Real-time human pose ecognition in pats fom single depth images. In: IEEE Confeence on Compute Vision and Patten Recognition (CVPR), pp (11). [27] Oikonomidis, I., Kyiazis, N., and Agyos, A. A., Tacking the aticulated motion of two stongly inteacting hands. IEEE Confeence on Compute Vision and Patten Recognition (CVPR), pp (12). [28] Bilal, S., Akmeliawati, R., Shafie, A. A., and Salami, M. J. E.: Hidden Makov Model fo human to compute inteaction: a study on human hand gestue ecognition. Atificial Intelligence (11). [29] Mines, B. W., Basi, O. A., and Kamel, M. S., Undestanding hand gestues using appoximate gaph matching. IEEE Tansactions on Systems, Man and Cybenetics, Pat A: Systems and Humans, vol. 35, no. 2, pp (5). [3] Xu, Z., Xiang, C., Li, Y., Lantz, V., Wang, K., and Yang, J., A Famewok fo Hand Gestue Recognition Based on Acceleomete and EMG Sensos, IEEE Tansactions on Systems, Man and Cybenetics, Pat A: Systems and Humans, vol.41, No.6, pp (11). [31] Black, M. J., and Jepson, A. D., A pobabilistic famewok fo matching tempoal tajectoies: CONDENSATION-based ecognition of gestue and expessions. ECCV [32] Alon, J, Athitsos, V., and Yuan, W., A Unified Famewok fo gestue Recognition and Spatiotempoal Gestue Segmentation, IEEE Tansactions on Patten Analysis and Machine Intelligence, vol. 21, pp (9). [33] Guyon, I., Athitsos, V., Jangyodsuk, P., Escalante, H. J., and Hamne,

12 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS-PART A: SYSTEMS AND HUMANS 12 B., Results and Analysis of the ChaLean Gestue Challenge (12). [34] Fei-Fei, L., Fegus, R., and Peona, P., One-shot leaning of object categoies, IEEE Tansactions on Patten Analysis and Machine Intelligence, vol. 28, No. 4, pp (6). [35] Wu, D., Zhu, F., and Shao, L., One shot leaning gestue ecognition fom gbd images. In: CVPR12 wokshop on gestue ecognition (12). [36] Yang, Y., Saleemi, I., and Shah, M., Discoveing Motion Pimitives fo Unsupevised Gouping and One-shot Leaning of Human Actions, Gestues, and Expessions. IEEE Tansactions on Patten Analysis and Machine Intelligence (12). [37] Guyon, I., Athitsos, V., Jangyodsuk, P, Hamne, B., and Escalante, H. J., ChaLean gestue challenge: Design and fist esults, IEEE Compute Society Confeence on Compute Vision and Patten Recognition Wokshops (CVPRW), pp. 1-6 (12). [38] Wachs, J., Sten, H., and Edan, Y., Cluste Labeling and Paamete Estimation fo the Automated Setup of a Hand-Gestue Recogntion System, IEEE Tansactions on Systems, Man and Cybenetics, vol. 35, no. 6, pp (5). [39] Jones, M. J., and Rehg, J. M., Statistical colo models with application to skin detection, In: IEEE Compute Society Confeence on Compute Vision and Patten Recognition, vol. 1, pp (1999). [4] Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple featues. In: Intenational Confeence on Compute Vision and Patten Recognition, pp (1). [41] Hess, R., Fen, A.: Disciminatively Tained Paticle Filtes fo Complex Multi-Object Tacking. In: IEEE Compute Society Confeence on Compute Vision and Patten Recognition, pp ( 9). [42] Yu, T. and Wu, Y.. Collaboative tacking of multiple tagets. Compute society confeence on compute vision and patten ecognition (CVPR). 4. [43] Khan, Z., Balch, T. and Dellaet, F., An MCMC-based paticle filte fo tacking multiple inteacting tagets. Technical epot. 3. [44] Bog, G., Psychophysical bases of peceived exetion. Medicine & Science in Spots & Execise, vol. 14(5), (1982). [45] Jiang, H., Duestock, B. S., and Wachs, J. P. Integated gestue ecognition based inteface fo people with uppe extemity mobility impaiments, In: Poc. of the 4th Intenaiton Confeence on Applied Human Factos and Egonomics (12). [46] Aach, J., Chuch, G.M.: Alignment gene expession time seies with time waping algoithms, J: Bioinfomatics, vol. 17, no. 6, pp , Oxfod Univesity Pess (1). [47] Savitzky, A., and Golay, M. J. E., Smoothing and difeentiation of data by simplified least squaes pocedues, Analytical Chemisty, vol. 36, no. 8, pp (1964). [48] OpenNI: [49] Pingali, G., and Segen, J., Pefomance evaluation of people tacking systems, In: Poc. IEEE Wokshop on Application of Compute vision, pp (1996). [5] Guyon, I., Athitsos, V., Jangyodsuk, P., and Escalante, H. J., The ChaLean Gestue Dataset (11). [51] Escalante, H. J., and Guyon, I., Pincipal motion: Pincipal motion: Pcabased econstuction of motion histogams. Technical epot, ChaLean Technical Memoandum (12). [52] Sohn, M. K., Lee, S. H., Kim, D. J., Kim, B., and Kim, H., A compaison of 3D hand gestue ecognition using dynamic time waping. In: Poceedings of the 27th Confeence on Image and Vison Computing, pp (12). [53] Goldbeg, K. The Robot in the Gaden: Teleobotics and Telepistemology in the Age of the Intenet. Mit Pess.. [54] Safaic, M. Tuntic, D. Hecog, and G. Pacnik. Con-tol and obotics emote laboatoy fo engineeing ed-ucation. Intenational Jounal of Online Engineeing (ijoe), 1(1), 5. [55] Causi, F., Casini, M., Pattichizzo, D., and Vicino, A. Distance leaning in obotics and automation by emote contol of LEGO mobile obots. In Poc. Int. Conf. on Robotics and Automation, pages , New Oleans, USA, Apile 4. [56] Casini, M., Chinello, F., Pattichizzo, D., & Vicino, A. (8, July). RACT: A emote lab fo obotics expeiments. In Poceedings of the 17th IFAC Wold Congess. Seoul (Koea). BIOGRAPHIES Haiong Jiang eceived he B.S. degee in contol science and engineeing fom Habin Institute of Technology, in 8. He M.S. degee in contol science and engineeing fom Habin Institute of Technology Shenzhen gaduate school, in 1. She is cuently woking towad he Ph.D. degee at the School of Industial Engineeing, Pudue Univesity, USA. He pimay eseach inteests include gestue ecognition and assistive technology. Badley S. Duestock eceived a B.S. degee in Biomedical Engineeing at the School of Intedisciplinay Engineeing in 1994 and Ph.D. degee in Neuobiology at the College of Veteinay Medicine in 1999 fom Pudue Univesity, West Lafayette, IN, USA. He was a postdoctoal eseach associate at the Cente fo Paalysis Reseach at Pudue Univesity. He is an associate pofesso of Engineeing Pactice in the Weldon School of Biomedical Engineeing and School of Industial Engineeing at Pudue Univesity. He is the Diecto of Institute fo Accessible Science. His eseach inteests focus on the estoation of functional impaiment though epai of cental nevous system damage o development of assistive technologies and accessible design. Juan P. Wachs is an Assistant Pofesso in the School of Industial Engineeing at Pudue Univesity. He is the diecto of the Intelligent Systems and Assistive Technologies Lab (ISAT) and he is affiliated with the Regenstief Cente fo Healthcae Engineeing. He completed a postdoctoal taining at the Naval Postgaduate School s MOVES Institute in the aea of compute vision, unde a National Reseach Council Fellowship fom the National Academics of Sciences and he was awaded the Ai Foce Young Investigato Awad 13. His eseach inteests include machine and compute vision, obotics, teleopeations, human obot inteaction, and assistive technologies. Juan Wachs is a membe of IEEE and the Opeation Reseach Society of Isael (ORSIS). He has published in jounals including IEEE Tans. Systems, Man, and Cybenetics, Jounal of Ameican Medical Infomatics, Communications of the ACM, and the Jounal of Robotic Sugey. He eceived his M. Sc. and Ph.D. in Industial Engineeing and Management fom the Ben-Guion Univesity of the Negev.