Gait Recognition Using WiFi Signals

Transcription

1 Gait Recognition Using WiFi Signals Wei Wang Alex X. Liu Muhammad Shahzad State Key Laboratory for Novel Software Technology, Nanjing University, China Department of Computer Science and Engineering, Michigan State University, USA Department of Computer Science, North Carolina State University, USA Abstract In this paper, we propose WifiU, which uses commercial WiFi devices to capture fine-grained gait patterns to recognize humans. The intuition is that due to the differences in gaits of different people, the WiFi signal reflected by a walking human generates unique variations in the Channel State Information (CSI) on the WiFi receiver. To profile human movement using CSI, we use signal processing techniques to generate spectrograms from CSI measurements so that the resulting spectrograms are similar to those generated by specifically designed Doppler radars. To extract features from spectrograms that best characterize the walking pattern, we perform autocorrelation on the torso reflection to remove imperfection in spectrograms. We evaluated WifiU on a dataset with 2,8 gait instances collected from 5 human subjects walking in a room with an area of 5 square meters. Experimental results show that WifiU achieves top-1, top-2, and top-3 recognition accuracies of 79.28%, 89.52%, and 93.5%, respectively. ACM Classification Keywords H.1.2 User/Machine Systems; I.5. Pattern Recognition Author Keywords Gait Recognition; Device-free Sensing. INTRODUCTION Motivation and Proposed Approach With the success of WiFi industry, commercially available WiFi devices can not only achieve high speed and low cost, but can also measure small changes in WiFi signals and thus sense the changes in their surrounding environments caused by moving objects such as humans [3, 31]. In this paper, we propose WifiU, which uses Commercial Off-The-Shelf (COTS) WiFi devices to capture fine-grained gait patterns so that we can recognize humans. WifiU consists of two WiFi devices, one for continuously sending signals, which can be a router, and one for continuously receiving signals, which can Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. UbiComp 16, September 12-16, 216, Heidelberg, Germany 216 ACM. ISBN /16/9... $15. DOI: WiFi signal WiFi signal reflection WiFi router Laptop Signal analysis (a) Application scenario Walking route (5.5m) Table 7.7 m sender 1.6 m receiver Table (b) Data collection environment Figure 1. WifiU system overview be a laptop. Figure 1 shows an overview of our WifiU system. In WifiU, the specific information that the receiver measures is the Channel State Information (CSI) of each received WiFi frame. With a human walking around, as a human is mostly made of water, the WiFi signal reflected by the human body generates unique, although small, variations in the CSI measurements on the receiver due to the well-known multi-path effect of wireless signals. These variations in CSI allow us to use signal processing techniques to obtain gait information such as walking speed, gait cycle time, footstep length, and movement speeds of legs and torso. As it is well known that humans have quite unique gait [8, 15], the gait patterns that the WiFi receiver obtains can be used to recognize the walking human subject. Fundamentally, WifiU recognizes humans based on who they are because WifiU extracts unique biometrics information from WiFi signals and uses it to perform human recognition. WifiU enables many potential applications that require user identification. For example, a smart building can recognize the user using WifiU, while the user is walking along the corridor. Based on the user identity, it can automatically open the door when he/she approaches his/her office. Similarly, smart home applications can use WifiU to adjust background music or ambient temperature based on who is at home. Compared with traditional gait recognition systems, which use cameras [17], floor sensors [18], or wearable sensors [8] to capture gait information, WifiU is easier to deploy and has better coverages. From the deployment perspective, WifiU does not require any special hardware (such as floor sensors) and the human subject does not require to wear any hardware (such as accelerometers). WiFi devices are ubiquitous and most homes/offices are covered by WiFi signals. The hard- 6.5 m

2 ware that we experimented with in this work, namely Net- Gear JR61 WiFi router and ThinkPad X2 laptop (with Intel 53 WiFi NIC), are COTS devices and we do not require any hardware modification to these devices. Furthermore, unlike cameras, WifiU does not require lighting and works in dark just as well as in light. Technical Challenges and Solutions The first challenge is to profile gait patterns using CSI dynamics. Extracting gait information from CSI signals is difficult because the signal reflections of different body parts are mixed together in the CSI waveform. As different human body parts move at different speeds while walking, the radio signal reflections from different body parts have different frequencies. To separate the radio signal reflections from different body parts, we convert the CSI waveforms (of two dimensions: time and amplitude) to spectrograms in the timefrequency domain (of three dimensions: time, frequency, and amplitude). We apply spectrogram enhancement techniques to reduce signal noises. The resulting spectrograms give us detailed human gait information similar to those generated by Doppler radars [22, 27]. The second challenge is to extract features from spectrograms that best characterize the walking pattern of a human. There are various metrics (such as walking speed and gait cycle time) that can be used to characterize human walking patterns. However, obtaining these measurements from a spectrogram is challenging because we need to precisely measure the time of gait cycles. For example, to measure gait cycle time, i.e., the time between two consecutive instances for the right heel hits the ground [33], we need to determine the time that the right heel hitting the ground, which is non-trivial. To accurately measure gait cycle time, we perform autocorrelation on the contours of the torso reflection to remove small imperfection in spectrograms. We also generate a spectrogram signature to distinguish persons with similar walking speeds and footstep lengths. Summary of Experimental Results We implemented WifiU on COTS laptop and WiFi router. We conducted experiments on our gait database that contains more than 2,8 gait instances collected from 5 human subjects walking in a typical laboratory with an area of 5 square meters as shown in Figure 1(b). We anonymized all collected data to protect privacy. Over the 5 subjects, WifiU achieves recognition accuracies of 79.28%, 89.52%, and 93.5% for top-1, top-2, and top-3 candidates, respectively. Limitations and Privacy Implications The current implementation of WifiU has two limitations. First, the recognition system is suitable only for confined spaces, such as a corridor or a narrow entrance, because the users must walk on a predefined path in a predefined direction. Second, WifiU can extract the gait pattern only when there is a single user walking on the predefined path. Our future work is to explore the possibility of using multiple devices to separate gait signals from multiple users. From the perspective of privacy, hackers can potentially use WifiU to identify victims without being detected. Consider the scenario where a burglar attempts to know who is at home by eavesdropping the WiFi signal emitted by the WiFi router in the victim s house. As WiFi signals can penetrate through obstacles such as furniture, wooden doors, and walls [3], the burglar only needs to passively measure the CSI of the signal outside the house without decoding the content of the WiFi packets. Therefore, it is hard for the victim to prevent privacy breaches. While protecting privacy breach is not the focus of this paper, we hope that this work will bring this privacy issue to the research community and inspire future work. RELATED WORK Gait Based Human Recognition Prior gait based human recognition schemes mostly use information collected from the following three types of sensors: cameras, floor sensors, and wearable sensors [8]. Camera based schemes first generate a sequence of human silhouettes from the video frames by separating the moving objects from the background, and then use the sequence of human silhouettes to perform recognition [17]. An example camera based scheme achieves an accuracy of 91% on the USF HumanID Database with 122 subjects [29]. Floor sensor based schemes use force sensors under the floor to detect human footsteps [18]. An example floor sensor based scheme achieves an accuracy of 9% 97.5% for identification a single target person [28]. Wearable sensor based schemes use the accelerometers equipped in smart phones and wearable devices to collect gait information and extract features for human recognition [6, 19]. An example wearable sensor based scheme achieves an accuracy of 9% over 6 human subjects [24]. Activity and Gait Recognition using Radar Signals As a human body can reflect wireless signals, human movement can be analyzed using the Doppler shift of the reflected signal or the time-of-flight measurements obtained through Frequency-Modulated Continuous-Wave (FMCW) radars. The micro-doppler signature, which describes the velocity of different body parts, can be used for human activity recognition [16,21,22]. Using the Boulic walking model, it is possible to estimate parameters from the micro-doppler signature and use these parameters to rebuild the animation of the walking process [27]. Tahmoush et al. built a radar based gait recognition system that uses the stride rate as features and achieve recognition accuracy of 8% over 8 human subjects [25]. Adib et al. used FMCW Radar to capture a human figure behind the wall and to identify the user with an accuracy of 88% over 15 human subjects [1]. Compared to radar signals, WiFi signals are much easier and cheaper to obtain, but have much narrower bandwidth (2 MHz compared to 1.79 GHz of the FMCW radar used in WiTrack [2]) and thus have much lower time-resolutions. This makes human gait recognition using WiFi signals much harder than using radar signals. Consequently, the techniques in this paper are fundamentally different from those used in FMCW radars. Activity Recognition using WiFi Signals CSI measurements have been used for applications such as human activity recognition [31, 32], fall detection [12], and even object localization [23]. E-eyes utilized CSI histogram to achieve in-place and walking activity recognition with an

3 accuracy of 92% [32]. WiFall used anomaly detection algorithm on CSI values to detect falling [12]. PinLoc gathered location dependent CSI value distributions to localize a target with accuracy of several centimeters [23]. MRW used CSI or Received Signal Strength (RSS) values to monitor the presence of human movement behind a concrete wall [4]. CARM used CSI-speed model to extract human movement speeds from CSI signal [31]. However, these speed features are still not accurate enough for gait pattern recognition. In comparison, we perform time-frequency analysis on CSI values to obtain spectrograms, and then to measure human walking metrics, such as gait cycle time, torso and leg speed, and footstep length from the spectrograms. Note that there are schemes that use special hardware (such as USRP [2] and directional antennas [13, 3]) to get fine-grained measurements from WiFi signals; in comparison, we only use COTS WiFi devices. In recent work parallel with ours, Zeng et al. built a system called WiWho that also uses WiFi CSI signals to recognize human gaits [34]. In WiWho, the human subject was asked to walk on a path with a distance of 1 meter parallel to the Line- Of-Sight (LOS) path between the WiFi sender and receiver. The recognition accuracy of WiWho is 92% to 8% for 2 to 6 human subjects. Compared to WiWho, WifiU uses advanced signal processing algorithms to extract CSI variations so that it can recognize gaits at a distance of more than 6 meters to the LOS path. Moreover, WifiU extracts more elaborated gait features from CSI signals. WIFI SIGNAL PROCESSING In this section, we first collect CSI measurements from WiFi signals using COTS WiFi devices. Second, we use the Principal Component Analysis (PCA) technique to extract the principal components from the correlated CSI measurements so that the uncorrelated noises in different subcarriers are reduced. Third, we use Short Time Fourier Transform (STFT) to convert PCA components into spectrograms. Fourth, we apply frequency domain denoising algorithms (such as noise floor subtraction, spectrogram superimposition, and 2- dimensional filtering) to further enhance the spectrogram. CSI Data Collection WifiU collects CSI measurements on the receiving end of a WiFi link between two WiFi devices. For each pair of a sending antenna and a receiving antenna, we obtain CSI values from 3 OFDM subcarriers used by 82.11n [14]. Thus, we get = 18 CSI values for each received 82.11n frame when the sender has 2 antennas and the receiver has 3 antennas. The sequence of CSI values for each subcarrier for a given pair of sending/receiving antenna is called a CSI stream. As our system sends 2,5 WiFi frames per second, we collected 2,5 CSI values for each of the 18 CSI streams in one second. We removed the impact of Carrier Frequency Offset (CFO) by using only the amplitude of the CSI values while ignoring the CSI phase, as described in our earlier work [31]. Denoising CSI Measurements The CSI values describe how the phase and magnitude of the wireless signal change when the signal travels from the sending antenna to the receiving antenna over a subcarrier. When there is no moving object around, the magnitude of CSI remains relatively constant. However, a moving human can significantly distort the CSI magnitude because human bodies are good reflectors of wireless signals. When a human moves, the wireless signal reflected by his body will go through a different path with different length, as illustrated in Figure 1(a). Governed by the principle of superposition of waves, signal reflected by human may add constructively or destructively with WiFi signals traveled through other paths, e.g., the Line- Of-Sight (LOS) path. Whether these WiFi signals are added constructively or destructively is determined by the relative phase differences between these signals [26]. The phase of the signal changes by 2π when the length change of the signal path is equal to the signal wavelength. Whenever the human moves by half of the wavelength, the path length of the human reflected signal will change by the amount of the signal wavelength. Consequently, we will observe a full cycle of magnitude changes in CSI values according to the principle of superposition of waves. The factor of 2 in the path length change is due to the round trip path travelled by the reflected WiFi signal. The signal wavelength λ for 5 GHz WiFi signal band is cm. This implies that we can detect small movements of a few centimeters by observing the magnitude changes of CSI values. For a walking human who moves at the speed of 1 meter per second, we observe cycles of magnitude fluctuations in CSI values, given the wavelength of cm, as each cycle represents the person moves by the distance of half-wavelength. This provides a very detailed measurement of the human walking speed. However, the CSI measurements obtained from commercial WiFi cards contain noises from various sources such as interference coming from nearby devices, transmission power adaptation at the sender, and imperfect clock synchronization [1]. Figure 2(a) shows the magnitude of a raw CSI stream (Stream A) captured while a human is walking around. Although we can observe the fluctuations in CSI values caused by the moving human subject, these fluctuations are irregular due to environmental noises. Thus, we must denoise CSI measurements before we extract human gait information. Existing CSI denoising schemes, such as low-pass filters [3], do not work well for our purpose because CSI streams contain high-level impulse and burst noises. Figure 2(a) shows an example CSI stream with impulse noises at the time of 1.55 seconds as pointed by an arrow. Figure 2(b) shows the lowpass filtering result of CSI Stream A after passing through a Butterworth filter with a cutoff frequency at 15 Hz. We can still observe small residual fluctuations, as pointed by an arrow, after low-pass filtering, due to the wide bandwidth of impulse noises. Figure 2(b) plots another CSI Stream B, which is measured on the same sender/receiver antenna pair but with a subcarrier frequency about 1 MHz higher than that of CSI Stream A. Zooming into the waveform segments separated by the four vertical reference lines, we observe that the valleys for Stream B always appear earlier than that of Stream A. This indicates that variations in CSI streams have different phases. Different CSI streams often have different phases because they differ in their subcarrier frequency

4 CSI amplitude Impulse noises Time (seconds) CSI amplitude Residual noises 35 Stream A 3 Stream B Time (seconds) Time (seconds) (a) Raw CSI waveforms (Stream A) (b) CSI waveforms after low-pass filtering (c) CSI waveforms after PCA denoising Figure 2. CSI waveforms for a person walking along a straight path as in Figure 1(b) PCA amplitude and thus their radio wavelengths also differ. The phases of these CSI streams can differ by π and their waveforms may even cancel each other. Therefore, directly adding these CSI streams is also not a good choice for noise removal. We use Principal Component Analysis (PCA) based denoising algorithm to remove noises in the CSI signal [31]. Phase differences in CSI streams can be characterized by their cross correlations. PCA can automatically discover the correlation between CSI streams and recombine CSI streams to extract components that represent the variation caused by human activity. To remove the noise, we extract the first twenty PCA components from the 18 CSI streams and discard the rest, which are mostly noisy components. The optimal number of PCA components is determined by experiments shown in the experimental results section. The PCA denoised CSI waveforms are much smoother than those generated by low-pass filtering and preserve the details in CSI amplitude changes. Figure 2(c) shows the second PCA component derived from the raw CSI streams in Figure 2(a). We observe that there are 17 peaks for the.5 second between seconds in Figure 2(c). This gives 34 cycles/sec fluctuations in CSI, matching the walk speed of 1m/s of the human subject. Spectrogram Generation It is still difficult to extract human gait information from a PCA denoised CSI waveform because the signal reflections of different body parts are mixed together in the waveform. Different human body parts move at different speeds during different phases in walking. A gait cycle of human walking has two phases: the stance phase and the swing phase [33]. The stance phase starts at the time that the right heel touches the ground. During the stance phase, the left leg moves forward. At the end of the stance phase, the right toe leaves the ground. In the swing phase, the human body is supported by the left leg and the right leg is swinging forward. Thus, the right leg moves at a higher speed than the torso and the left leg is almost static during this phase [27]. As different body parts move at different speeds, the radio signal reflections of different body parts have different frequencies. CSI frequency is determined by human moving speeds as f = 2v/λ, where v is the moving speed, f is the frequency of the CSI waveform, and λ is the radio signal wavelength. For example, a moving speed of 1 m/s can be converted to the frequency of Hz when the radio wavelength λ is 5.79cm. As the radio signal reflections of different body parts have different frequencies, the CSI fluctuations of different frequencies are separable in the frequency domain. Thus, to separate the radio signal reflections of different body parts of different moving speeds, we convert the PCA denoised CSI waveforms into the time-frequency domain. We use the Short-Time Fourier Transform (STFT) technique to transform the waveforms to spectrograms so that CSI waveforms can be analyzed in the time-frequency domain. The spectrogram uses a sliding window to cut a small piece of samples from the waveform and then performs Fast Fourier transform (FFT) on the waveform. The spectrogram has three dimensions: time, frequency, and FFT amplitude. The window size for FFT determines the tradeoffs between frequency and time resolution of STFT. With a larger window size, the STFT has higher frequency resolution but lower time resolutions. CSI measurements for human walking have frequency of 3 4 Hz and changes in tens of milliseconds. Thus, we choose to use an FFT size of 124 samples and sliding window step size of 32 samples because it gives suitable frequency resolution of 2.44 Hz and time resolution of 12.8 ms to trace the changes in walking signals. Figure 3 shows an example spectrogram in a heat map, where hotter color represents higher FFT amplitude. To build the heat map for the spectrogram, we use the first 8 FFT magnitude of each chunk to represent the energy in the frequency range of 146 Hz. Spectrogram Enhancement After a spectrogram is generated, we apply spectrogram enhancement techniques to further reduce the noise in the spectrogram. First, we get the energy level for each chunk by summing the magnitude of the first 8 FFT points. We ignore chunks with energy level lower than an empirical threshold so that silent periods are removed. We normalize the FFT magnitudes for each remaining chunk by dividing them with the energy level of the chunk. Second, we apply frequency domain denoising method by subtracting the noise floor from the spectrogram [7]. The noise floor level is estimated through a long-term mean over the spectrograms, e.g., average magnitude over 4 seconds. If the resulting magnitude becomes negative after subtraction, we set it to zero. Third, we superimpose the spectrograms generated by the first twenty PCA results by adding up the magnitude of corresponding time-frequency blocks. Fourth, we apply a 2- dimensional Gaussian low-pass filter, with a size of 5 and σ of.8, on the resulting spectrogram. Finally, we obtain a high quality spectrogram that gives detailed information for the walking process of a human subject. FEATURE EXTRACTION In this section, we present how gait features can be extracted from the CSI spectrogram. The gait features include walking speed, gait cycle time, footstep length, movement speeds of torso and legs, and spectrogram signatures.

5 We detect the start of walking based on the fact that moving humans generate higher variances in CSI measurements than background noises. When there are no human movements, the variances in CSI measurements are mostly caused by noises. Since the noise level changes slowly over time, we use a dynamic thresholding algorithm to track the noise level. We first calculate the variance var(t) for a segment with 5 samples at time t in the second PCA component derived in Section 3. We do not use the first PCA component because it may contain a large amount of noise [31]. We update the noise level estimation N(t) using an Exponential Moving Average algorithm during the silent period: N(t) = (1 α n )N(t 1) + α n var(t) (1) where the coefficient α n is set to.1. We set the detection threshold as three times the noise level N(t) so that a movement is detected when the variance of the PCA result substantially deviates from the average noise level. Once the movement is detected, we use the torso speed to find the period that the person walks with a steady speed. Figure 3. CSI Spectrogram with human walking. Understanding CSI Spectrogram We observe that CSI spectrograms give us similar information as the expensive Doppler radars [16, 22, 27]. Thus, we can extract human gait patterns from CSI spectrograms using insights from Doppler radars. In particular, we can observe the moving speed changes of different body parts, e.g., torso and legs, from the spectrogram. Figure 3 shows the spectrogram obtained in a scenario where a human walks on a straight line for a distance of 5.5 meters. Since the torso has a larger reflection area than other body parts, the signal energy reflected from the torso is the strongest component in the spectrogram. We observe a high energy ( hot colored) band, which corresponds to torso reflection, in the spectrogram in Figure 3. From the frequency change in the torso reflection in Figure 3, we learn that the subject starts walking at the time of.5 seconds, speeds up, and then walks with a steady speed between the time of seconds, finally he slows down and stops at the time of 6 seconds. During the walking process, the torso reflection has a frequency range of Hz, which corresponds to m/s moving speed. We observe that the torso moving speed fluctuates regularly for about twice per second. This type of regular fluctuations in torso speed is also observed in Doppler radar results [27]. The blue flames in the frequency region of 4 6 Hz are the reflections of swinging legs [25]. The magnitude of leg reflections are weak due to the much smaller reflection area of legs. Both the regular speed changes in leg and body reflection indicate that the gait cycle, which contains two footsteps, has a duration of about 1 second. With these basic understandings of the spectrogram, we are ready to extract useful information about the walking pattern. Detecting Start of Walking Estimating Gait Cycle Time We now estimate gait cycle time, which is defined as the time duration between two consecutive events that the right heel touches the ground. Although we can roughly count steps from Figure 3, we need a robust estimation scheme that works for various walking styles. In WifiU, we use the upper contour for the torso reflection to estimate the gait cycle time. The torso contour frequency f tc (t) is defined as: { } F( f,t) f tc (t) = max f f max F( f,t) > γ (2) where F( f,t) is the magnitude of FFT at time t and frequency f. The summation of f max F( f,t) gives the total energy in the frequency range of f max. Thus, f tc (t) is the highest frequency that has an energy ratio larger than γ for the FFT result at time t. We set the threshold γ as 5% in our system. After smoothing the resulting f tc (t) with a low-pass filter, we get the torso contour speed v tc (t) as the thick line in Figure 4(a) using the equation v tc (t) = f tc (t)λ/2. Comparing the curve in Figure 4(a) with the spectrogram in Figure 3, we observe that our method reliably tracks the torso speed variations during the walking periods. The reason for choosing the contour rather than the frequency that has maximum energy is due to its robustness to noises. With a small amount of noise, as in the 3 4 seconds period in Figure 3, the frequency with maximum energy, i.e., the hottest frequency for a given time, may change significantly and makes the measurement unreliable and discontinuous in time. We use the autocorrelation of the torso contour curve to robustly estimate gait cycle time, which is calculated as: R(τ) = (v tc (t) µ)(v tc (t τ) µ) (3) t where µ is the average speed of the torso contour. The autocorrelation is taken on the period of steady walking, which is defined as the period where the torso speed (calculated in Section 4.4) is no less than 8% of the maximum torso speed. By taking the autocorrelation, we can find the period of the curve by comparing the curve to a version of itself that is displaced by τ in time. This gives a better estimation than directly searching for peaks on the torso contour curve. Compared to the original torso contour curve in Figure 4(a), it is easier to find the peaks in the autocorrelation result in Figure 4(b). Each peak in autocorrelation means that the contour curve remains similar to the original version when displaced by a time of τ. As the gait cycle contains two footsteps, the torso speed varies twice in each gait cycle. So, the torso contour curve is similar when displaced by L/2, where L is the gait cycle time. In Figure 4(b), the first peak in the autocorrelation appears at the point τ =.57 second; thus, we obtain an estimation for the gait cycle time L = 2τ as 1.14 seconds. Estimating Torso and Leg Speeds We estimate the torso and leg speed using the percentile method developed for Doppler radars [27]. The percentile

6 Speed (m/s) % percentile 95% percentile Torso contour Time (Seconds) Autocorrelation value τ (seconds) (a) Speed estimation (b) Autocorrelation of torso contour Figure 4. Cycle time and movement speed estimation. L/2 L/2 L/2 Autocorrelation of torso contour Peaks for autocorrelation CDF Estimation difference (seconds) Figure 5. Estimation difference: CSI vs. accelerometer at a given frequency f is defined as: P( f,t) = f F( f,t) f max F( f,t), (4) where P( f,t) is the cumulated percentage of energy for frequencies lower than f compared to the total energy of the FFT result at time t. The torso movement speed is estimated through the lowest frequency values that satisfy P( f,t) 5%, and the leg speed is estimated by the lowest frequency values that satisfy P( f,t) 95%. We choose the two threshold values 5% and 95% as they were used in prior work on human movement modeling using Doppler radar signals [27]. The estimation result is shown in Figure 4(a). The torso speed estimated through the percentile method has smaller variance than the torso contour method used in cycle estimation. The smaller variance has both advantages and disadvantages. The advantage is that a smaller variance gives more stable torso speed estimations so that we can reliably detect the walking period when the torso speed is stable and falls within the normal walking speed range. The disadvantage is that no periodical fluctuations can be detected on the torso speed curve obtained by the percentile method. Thus, it is suitable to use this method for walking speed estimation, but not for cycle time estimation. This percentile method also allows us to estimate leg movement speed. We can see the periodical speed up of legs in Figure 4(a). The leg speeds are useful for human recognition. But, we do not estimate gait cycle time based on leg speeds because the leg reflections are weak and sometimes buried in the noise. Accuracy and Distinguishability The gait cycle time measurements that we obtained from WiFi signals have comparable accuracy to those obtained from wearable sensors. We use the accelerometer in a smartphone as a reference to verify the accuracy of the gait cycle time obtained from spectrograms. We attach the smartphone on the ankle of the subject so that the accelerometer can get a clear measurement on gait cycles. Both the accelerometer and the CSI measurements are gathered from 1 walking instances of two different subjects. For each walking instance, we compare the estimated gait cycle time obtained from CSI with that obtained from the accelerometer. The average difference between the two methods is 11.9 ms. As the spectrogram has a time resolution of 12.8 ms, this difference is very small. The robustness of our measurement method is also satisfactory. Figure 5 shows the CDF for the measurement difference. For 8% of the cases, the difference between the two methods is smaller than 5 ms and the maximal difference is less than 9 ms, which is smaller than the differences in cycle Subjects Height (cm) Cycle (s) Speed (m/s) A (.4).972 (.6) B (.3).844 (.4) C (.5).984 (.6) D (.3) 1.91 (.4) E (.1).832 (.8) Table 1. Measurements of testing subjects. time of different persons. We choose to use accelerometers to estimate the cycle times instead of estimating them from videos of users for two reasons. First, accelerometers have a higher time resolution (5 to 1 samples per second) compared to videos (24 to 3 frames per second) [17]. Second, accelerometers attached to ankles provide reliable gait cycle time measurements [9]. We use the measurements for five subjects to illustrate that our measurement accuracy is good enough for human recognition. We asked five human subjects to walk in their natural way for 5 times along a line of of 5.5 meters. The measurement results are summarized in Table 1. For gait cycle time and torso speed, the first value in the table is the mean over the 5 walks and the value following in the parenthesis is the standard deviation. We see that people actually walk in different ways that can be captured by WiFi signals. Their average walking speed ranges from.832 m/s to 1.91 m/s, while the standard deviation of the walking speed for the same person is very small (less than.8 m/s) compared to the differences between persons. We also have similar observations in their gait cycle time distributions. Note that the walking speed could be slightly lower than the normal speed for users, as they only walk for a distance of 5.5 meters in the experiments. More interestingly, even for people with apparently similar speed, e.g., human subjects A and C, we still can reliably separate them using gait cycle time and torso speed. Figure 6 shows the scatter plot for all walking samples from the five subjects. Note that the X-axis is the cadence, defined as the gait cycles the person takes per second (1/L). We observe that the five human subjects are clearly separable in Figure 6 and their samples are scattered around the two straight lines. The reason for this type of distribution is that people tend to have constant footstep lengths during walking. For example, subject C walks at speeds between meters per second (m/s), which spans a quite large speed range. The distribution in Figure 6 indicates that the walking speed and cadence either both are high or both are low. This implies that the human subject keeps his footstep length nearly constant, but has small variations in his cadence so that his speed is linearly related to the cadence. Therefore, the few outliers for subject E in the upper right corner are possibly not caused by measurement errors. It is more likely that the human sub-

7 Energy.1.5 Subject A Torso speed (m/s) 5 1 Freq (Hz) A B C D E Cadence (cycles per second) Figure 6. Distribution of cadence and torso speed. Energy.1.5 Subject B Energy 5 1 Freq (Hz).1.5 Subject C Energy 5 1 Freq (Hz).1.5 Subject D Energy 5 1 Freq (Hz).1.5 Subject E 5 1 Freq (Hz) Figure 7. Spectrogram signatures for five human subjects on stage 2 of the half-gait-cycle. ject changes his walking speed for some tests, as the walking speed actually increase linearly with the cadence for these outliers. As tall people tend to have longer step lengths, we also give the heights of these five subjects in Table 1. The samples for subject C and E appear on a line with different slope in Figure 6 than those for subject A, B and D because they are taller and have longer step lengths than others. To understand the relationship between footstep length and height, we further run the footstep length estimation on a larger data set that contains 49 persons, using the product of torso speed and cycle time as the estimator of the step size. The Pearson correlation coefficient between the height and the footstep length of the subject is.45 and the p-value is.39. This shows that there is actually a positive correlation between the estimated footstep length and the height of the subject. Note that due to the variance in human behavior, there are subjects that varies footstep length significantly during the test and some subject have non proportional step size to their heights. Nevertheless, the footstep length still can serve as a useful feature for gait based recognition. Spectrogram Signatures We propose a new set of features, called spectrogram signatures, to further describe the gait patterns in detail. As humans may walk with similar speed and cadence, simply using the two metrics of gait cycle time and torso speed may not sufficiently recognize an individual among a large candidate set. The spectrogram in Figure 3 actually provides more information than the gait cycle time and torso speed. For example, the small green downward spikes appeared in almost every cycle, e.g., at 1.7 and 2.3 seconds, are possibly caused by the lower leg or the foot of the supporting leg that has lower speed than the torso [27]. Such detailed information enables us to characterize human gait patterns. To capture human gait patterns from spectrograms, we use the distribution of reflected energy on predetermined frequency points to serve as signatures of the spectrogram. These energy distributions give an overview on how different body parts are moving at a given stage of walking. Thus, they help to capture the detailed walking patterns of a human subject. In other words, the energy distribution can serve as the signature for a human gait pattern. We extract the spectrogram signature as follows. First, we use the cycle time measured in the previous section to help us to find peaks on the torso contour curve. We then cut the spectrograms into half-gait-cycles using these peaks. Each halfgait-cycle contains the process of swinging one leg, which could be the left or the right leg. We further divide the halfgait-cycle into 4 stages with equal length in time, which represent the different stages for leg swinging. On each stage, we calculate the normalized energy on 4 frequency points by averaging the FFT magnitudes during the stage. The shape of energy change in the frequency domain serves as the spectrogram signature for the subject. The spectrogram signature can serve as a fingerprint of the gait pattern. Figure 7 plots signatures on stage 2 for five subjects. For each subject, we plot the signature curve for 5 walking samples on the same graph. We observe that the signature curves for the same subject are clustered together so that they appear to be a thick curve, while different subjects have very different signatures. These signatures are determined by the gait patterns and other complex factors such as the height and size of the person. Therefore, they can serve as unique characters for a human. In WifiU, we extract a set of 17 features for each walking sample, including gait cycle length, estimated footstep length, the maximum, minimum, average, and variance for torso and leg speeds during the gait cycle, and spectrogram signatures for the 4 stages in the half-gait-cycle on 4 frequency points. TRAINING AND CLASSIFICATION In this section, based on the features that we extract from WiFi signals, we use machine learning techniques to build a gait recognition system. The first step for human recognition is enrollment. In this step, we collect gait instances of the target human subject, which will be used as training data. Our experimental results show that the number of gait instances that WifiU needs is around 4. Although the target human subject needs to walk for a distance of 5 6 meters for 4 times in the enrollment phase, we can collect training data when the individual is using traditional identification mechanisms (such as access tokens) to identify himself to reduce the data collection effort. Using the data gathered in enrollment phase, WifiU trains a gait model for the target human subject, which can be either used for single user identification or multiple user recognition applications. The single user identification application answers the question whether the subject is the target person or a stranger and the multiple user recognition application answers the question who the subject is among the given set of users. We use the LibSVM tool [5] with the Radial Basis Function (RBF) kernel in the training. The optimal values for parameters ν and γ for the RBF kernel are selected through grid search. To train a gait model for single user identification, we build a classifier that can classify gait instances into two classes, one

8 for the target person and one for others. In the training process, we use the gait instances from 7 9 benchmark persons as training data for the negative class. In real deployment, gait instances of benchmark persons can be drawn from a standard database. The reason for using benchmark persons is that these instances are helpful to determine the decision boundary for the target person and improve the identification accuracy. Once the gait model is trained, the classifier can calculate the fitness probability that an unknown gait instance belongs to the target person. We treat gait instances with fitness probability higher than a given threshold as instances belonging to the target person. The classifier can identify gait instances for persons that are not seen in the training phase, as their gait features have low fitness probability to the gait model of the target person. To recognize a person in a given set of candidates, we need one gait model for each person in the candidate set. For each person, we train a two-class classifier that can separate him/her from all the other candidates in the candidate set. Thus, for a candidate set with M persons, we build M onevs-all classifiers. In the prediction phase, we fit the unknown walking sample into the M classifiers and get the probability of fitness. WifiU selects the models with the k highest fitness probability as the top-k candidates for the testing sample. EXPERIMENTAL RESULTS Data Sets We collected gait patterns from 5 human subject with IRB approval. The human subjects were 36 male and 14 female graduate students, with similar ages in the range of years. We conducted our experiments in a typical lab with an area of 5 square meters. The layout of this lab is plotted in Figure 1(b). The WiFi sender and the WiFi receiver were placed on a table with a height of 8 cm and they were separated by a distance of 1.6 meters. In our experiments, we used a NetGear JR61 WiFi router (of less than 1 USD) that supports IEEE 82.11n protocol as the sender and a Thinkpad X2 laptop with the Intel 53 wireless card (of about 1 USD) as the receiver to collect CSI measurements using the Linux CSI tool [11]. The wireless router was configured to work at 5 GHz band and used a channel bandwidth of 2 MHz. We chose the 5 GHz band, rather than 2.4 GHz band because the wavelength of 5 GHz is shorter, and shorter wavelengths give better resolutions in movement speeds. The settings of the WiFi router, including channel bandwidth, carrier frequency and data rate settings, were similar to those of our campus network where other WiFi devices coexisted in the same channel. We used the omni-directional antennas that come with the router and laptop without any modifications. We used the default transmission power settings of the WiFi devices, so there are no potential harms to human subjects, as our devices fully comply with FCC regulations. We anonymized collected data to protect the privacy of human subjects. The detailed data collection process is as follows. Each subject was requested to walk repeatedly on a straight line with distance of 5.5 m. They were asked to walk in their natural way without intentional speed up or slow down. The CSI values were recorded on the laptop and processed offline. We Detection ratio Distance (meters) Figure 8. Detection ratio vs. operational distances collected 5 6 walking instances for each human subject. With the consent of all human subjects, we also recorded their heights and genders in our anonymized database. Evaluation Metrics We evaluated WifiU from four perspectives: operational distance, effectiveness, robustness, and efficiency. For operational distance, we evaluated the minimum distance between the walking human subject and the WiFi devices. For accuracy, we evaluated identification accuracy for single user identification and top-k accuracy for multiple user recognition. We evaluated the identification accuracy in terms of False Acceptance Rate (FAR) and False Rejection Rate (FRR). The FAR is defined as the rate that a stranger is wrongly classified as the target subject and the FRR is the rate that the true target is wrongly classified as a stranger. Since we can tradeoff between the FAR and FRR by changing the probability threshold for identification, we define the Equal Error Rate (EER) point as the point that FAR and FRR are equal. Top-k accuracy is defined as the percentage of tested instances in which WifiU is correct in declaring that the walking human subject is among the top-k candidates. Note that WifiU reports a ranked list of candidates in the decreasing order of similarity with the walking human subject. For robustness, we evaluated the identification accuracy of WifiU from two perspectives: (1) effect of evolution of human gait with time and (2) effect of difference in apparel and accessories. For efficiency, we focused on evaluating classification efficiency, which is the time that the classifier takes to make the classification decision, and classifier construction efficiency, which is the time for constructing the classifier. Operational Distance Our results show that WifiU can detect a walking human subject at a range as long as 14 meters. Using the movement detection algorithm described in Section 4.2, we measured the detection range of our system in a large open lobby area. Figure 8 shows the detection probability for a walking human at different distances. We observe that our system is able to detect a walking human with an accuracy of 92% at a distance of 14 m and the accuracy quickly reduces to around 5% at the distance of 16 m. However, the range that our system can reliably extract gait information is smaller. In our experiments we found the operational distance that allows us to perform gait recognition is about 6 meters, which is the distance used in our data collection process. Accuracy To evaluate identification accuracy, we first randomly choose 7 subjects from the 5 subjects in the database to serve as the benchmark set. For each target subject not in the benchmark set, we train a SVM classifier that treats the target subject as a class and benchmark subjects as the other class. The

9 CDF FAR FRR Error rate Figure 9. CDF of FAR and FRR rest = 42 subjects are treated as strangers in the evaluation. Note that none of the walking instances of these strangers are seen by the SVM during training. The FAR and FRR are calculated using 1 fold cross validation, where one tenth of the walking instances are used for testing and the remaining are used for training. After getting the individual FAR and FRR for all subjects not in the benchmark set, we repeat the whole process using a new set of benchmark subjects. The final results are averaged over 1 randomly selected benchmark sets. Our results show that WifiU achieves average FAR and FRR of 8.5% and 9.54%, respectively, when setting the probability threshold for identification to.5. In other words, our system determines that the gait sample belongs to the true user when the fitness probability is higher than the threshold of.5. Figure 9 shows the distribution of FAR and FRR on different subjects. For more than 8% subjects, WifiU achieves FAR smaller than 1% and FRR smaller than 15%. By changing the probability threshold in the gait model, we can achieve different tradeoffs between FAR and FRR. Figure 1(a) shows the tradeoff curve for FAR and FRR. At the point where FAR and FRR is equal, we get an EER of 8.6% when we use all the available training data. To evaluate top-k accuracy, in each experiment, we randomly pickup ten human subjects from our walking database. The top-k accuracy for the given candidate set is obtained via 1 fold cross validation over the samples of the ten subjects. We perform this experiment for 1 times and calculate the average top-k accuracies. WifiU achieves top-1, top-2, and top- 3 accuracies of 92.31%, 97.58%, and 98.86%, respectively. Note that the recognition task is considerably harder than the identification task in the previous experiment because in the recognition task, WifiU does not just have to determine whether the test subject is the true user or not, rather has to recognize exactly which user it is. When we increase the size of subject set to 5 subjects, the top-1, top-2, and top-3 accuracies are 79.28%, 89.52%, and 93.5%, respectively. Our results show that 4 training instances are practically large enough to build the classifier as shown by Figure 1(a). We plot the FAR and FRR for training schemes that use all available instances for training and compare them to the schemes that limit the training set size to 2, 3 and 4 instances. We see a diminishing improvement in EER with increasing number of training instances. Furthermore, the training set size of 4 gives nearly the same identification accuracy as the scheme that uses all available training instances, which is normally more than 5 instances in our database. Our results show that using the benchmark set size of 7 and PCA components size of 2 in WifiU are effective. Figure 1(b) gives the performance of WifiU under different benchmark set sizes. We observe that using more benchmark subjects gives lower EER, but the results for having 7, 9, and 11 benchmark subjects are almost the same. Therefore, we choose to use 7 9 benchmark subjects in WifiU. Figure 1(c) gives the performance of WifiU under different number of PCA components used in spectrogram generation. Using 2 PCA components is enough, since adding more components do not further improve the performance of WifiU. Robustness Our results show that evolution of human gait does not significantly affect the accuracy of WifiU. To study the robustness of WifiU against gait evolution, we collected more than 2 walking samples from the same subject over a period of four months and performed 1 fold cross validation on these samples. Figure 11(a) plots FAR vs. FRR of WifiU for this scenario obtained from the 1 fold cross validation. We observe from this figure that WifiU achieves an EER of just 11.3% on these testing samples. WifiU is also robust to small changes in environments, such as moving chairs/tables around the room or slightly changing the location of the Router/Laptop, which actually happened frequently during the four month period of data collection. Our results show that the accuracy of WifiU deteriorates only when the apparel and accessories of the target subject change significantly. To study the impact of change in apparel and accessories on the accuracy of WifiU, we first train WifiU using training samples of a subject wearing winter clothes and wearing spring clothes, and then evaluate its FRR on testing samples obtained when the same subject was wearing different types of clothes i.e., suits or carrying a briefcase. Figure 11(b) plots the FRR for different apparel and accessories options when FAR is set at 11.1%. The FRR increases from 1% in the training set (winter/spring clothes) to at least 25% when the subject is wearing different apparel and accessories. This happens due to the well studied fact that human gait changes when wearing different types of apparel and accessories [8]. Despite the fact that we never trained WifiU on samples with these apparel and accessories, it still provides decent FRR while keeping FAR as low as 11.1%. Efficiency Our results show that WifiU can run in realtime on laptops and desktop computers. We evaluated the efficiency of WifiU on a laptop equipped with an Intel Core i5 CPU running at 2.8 GHz. Table 2 gives the CPU processing time and the standard deviation for processing one second of CSI measurement in our implementation on Matlab. We observe that the total CPU time required to process one second of data is.275 second so that the laptop can easily handle realtime processing for WifiU. The major computational cost (about 71.3%) is in the PCA denoising step because PCA needs to process the raw CSI measurements that have = 45k samples per second. As our PCA process reduces the data size to one ninth of the raw CSI measurements, the spectrogram generation step only takes 26.2% of the total computational time.