The Pennsylvania State University. The Graduate School. College of Engineering CHARACTERIZATION AND CLASSIFICATION OF HUMAN ACTIVITY IN

Transcription

1 The Pennsylvania State University The Graduate School College of Engineering CHARACTERIZATION AND CLASSIFICATION OF HUMAN ACTIVITY IN DIFFERENT ENVIRONMENTS USING RADAR MICRO-DOPPLER SIGNATURES A Thesis in Electrical Engineering by Matthew Zenaldin 2016 Matthew Zenaldin Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Science August 2016

2 The thesis of Matthew Zenaldin was reviewed and approved* by the following: Ram M. Narayanan Professor of Electrical Engineering Thesis Advisor Timothy J. Kane Professor of Electrical Engineering Victor Pasko Professor of Electrical Engineering and Graduate Program Coordinator *Signatures are on file in the Graduate School

3 iii ABSTRACT This thesis presents the results of our experimental investigation into how different environments impact the classification of human motion using radar micro-doppler (MD) signatures. The environments studied include free space, through-the-wall, leaf tree foliage, and needle tree foliage. Results are presented on classification of the following three motions: crawling, walking, and jogging. The classification task was designed how to best separate these movements. The human motion data were acquired using a monostatic coherent Doppler radar operating in the C-band at 6.5 GHz from a total of six human subjects. The received signals were analyzed in the time-frequency domain using the Short-time Fourier Transform (STFT) which was used for feature extraction. Classification was performed using a Support Vector Machine (SVM) using a Radial Basis Function (RBF). Classification accuracies in the range 80-90% were achieved to separate the three movements mentioned.

4 iv TABLE OF CONTENTS List of Figures... v List of Tables... vi Acknowledgements... vii Chapter 1 Introduction and Motivation... 1 Chapter 2 Hardware and Environment Description Experimental setup Environmental description Time-frequency analysis Characteristics of non-ideal I/Q demodulation Chapter 3 Comprehensive Data Display Periodic stationary movmeents Non-periodic stationary movements Object-free movements Walking while carrying an object Multi-target movmeents Variability across human subjects Human gait micro-doppler signature database Chapter 4 Feature Extraction and Classification Feature extraction Classification approach Classification results Chapter 5 Conclusions Bibliography... 51

5 v LIST OF FIGURES Figure 1: Simplified block diagram of the Doppler radar system operating in the C-band at 6.5 GHz. Figure 2: Experimental setup in each environment: (a) through-the-wall, (b) leaf tree foliage, (c) and needle tree foliage. Figure 3: Stationary periodic movements (from left to right): Arm swinging, boxing, and waving. Figure 4: Through-wall stationary periodic movements (from left to right): Arm swinging, boxing, and waving. Figure 5: Stationary non-periodic movements (from left to right): Crouching to standing, sitting to standing, and squaring up to shoot. Figure 6: Through-wall stationary non-periodic movements (from left to right): Crouching to standing, sitting to standing, squaring up to shoot. Figure 7: Free-space forward-moving (starting from top-left and moving clockwise): Walking, walking with arms crossed against chest, limping, crawling. Figure 8: Free-space forward-moving (starting from top-left and moving clockwise): Walking, walking with arms crossed against chest, limping, crawling. Figure 9: Forward moving while carrying an object (starting from top-left moving clockwise): Walking with brick, walking with backpack, holding phone up to ear, corner reflector. Figure 10: Through-wall forward-moving while carrying an object (starting from top-left moving clockwise): Walking with brick, walking with backpack, walking with phone up to ear, walking with corner reflector. Figure 11: Free-space multi-target movements (starting from top-left moving clockwise): Two targets walking in opposite directions, two targets walking in the same direction, two targets running in opposite directions, two targets walking in opposite directions at different speeds. Figure 12: Through-wall multi-target movements (starting from top-left and moving clockwise): Two targets walking in opposite directions, two targets walking in the same direction, two targets running in opposite directions, two targets walking in opposite directions at different speeds. Figure 13: Stationary arm swinging MDS across subjects (from top left going clockwise): Subjects 1, 3, 5, and 6. Figure 14: Crawling MDS across subjects (from top left going clockwise): Subjects 1, 3, 5, and 6. Figure 15: MDS of stationary human standing still (breathing) in free space. Figure 16: MDS of human standing still and breathing heavily in free space. Figure 17: MDS of human standing still and holding breath in free space.

6 vi Figure 18: MDS of sitting in chair breathing heavily. Figure 19: MDS for sitting in chair holding breath. Figure 20: MDS for single step towards, keep back toe on ground. Figure 21: MDS for single step towards, bring both feet together. Figure 22: MDS for two steps toward, keep back toe on ground. Figure 23: MDS for right arm swing cycle. Figure 24: MDS for left arm swing cycle. Figure 25: MDS for both arm swing cycle, same direction. Figure 26: MDS for both arm swing cycle, opposite direction. Figure 27: MDS for raise right arm forward once. Figure 28: MDS for raise left arm forward once. Figure 29: MDS for raise both arms forward once at same time. Figure 30: MDS of raise both arms forward once at different time. Figure 31: MDS for oscillate between raising left and right arm. Figure 32: MDS for single right hand punch towards starting at boxing position. Figure 33: MDS of single left hand punch towards starting at boxing position. Figure 34: MDS of single punch both hands starting at boxing position. Figure 35: MDS of punch while oscillating between hands. Figure 36: MDS of turning around, facing towards. Figure 37: MDS of turning around, facing away. Figure 38: MDS of kicking away as if running. Figure 39: MDS of forward hop. Figure 40: MDS of standing still to sitting. Figure 41: MDS of standing still to crouching. Figure 42: MDS of walking towards and stopping.

7 vii Figure 43: MDS of sitting to standing still. Figure 44: Illustration of features extracted from an experimental spectrogram of stationary arm swinging. Figure 45: Two-dimensional un-normalized feature space comparison for each environment: (a) free space, (b) through-the-wall, (c) leaf tree foliage, and (4) needle tree foliage. The two features used here are mean Doppler frequency and total bandwidth. Figure 46: Spectrograms of three different types of human motions in the free-space environment: (a) crawling, (b) walking, and (c) jogging.

8 viii LIST OF TABLES Table 1: Physical characteristics of human subjects Table 2: List of features useful for free space human activity classification Table 3: List of features useful for through-the-wall human activity classification Table 4: Comparison of feature statistics for walking motion across different environments Table 5: SVM classification rate (10 iterations): 75/15 split Table 6: Confusion matrix for free space data collection Table 7: Confusion matrix for through-the-wall data collection Table 8: Confusion matrix for leaf tree foliage data collection Table 9: Confusion matrix for needle tree foliage data collection

9 ix ACKNOWLEDGEMENTS This work was partially supported by the US Naval Research Laboratory Contract # N C-2038 through ITT Exelis Subcontract # I would first like to thanks my parents for their all their support throughout my life. Next I would like to thank Dr. Ram Narayanan for his guidance and instruction in the course of my career as an undergraduate and graduate student. I would also like to thank the students in the lab who helped me with data collection Travis Bufler, Matthew Bradsema, Joshua Allebach, Sean Kaiser, and Sonny Smith. Lastly, I would like to thank Dr. Timothy Kane for volunteering to serve on my graduate committee.

10 1 Chapter 1 Introduction and Motivation The purpose of this study is to characterize the micro-doppler features of various human activities and to investigate the performance of algorithms developed to classify human motions using radar micro- Doppler signatures in different environments. Micro-motions of radar targets, such as vibrations or rotations, induce additional frequency modulations about the target s mean Doppler frequency. These additional frequency modulations are known as the micro-doppler effect. The four environments included in this study are: (1) free space, (2) through wall, (3) leaf tree foliage, and (4) needle tree foliage. The Short-time Fourier transform (STFT) is used to analyze the signatures in the time-frequency domain. After the spectrogram is computed, features are extracted and used to train a Support Vector Machine (SVM). The analysis and classification of human gait is a popular topic in the literature on micro-doppler signal analysis 1-5. Some researchers have examined the effects of an intervening wall on micro-doppler features in through-the-wall radar applications 6-9. The micro-doppler signatures of humans have also been studied in different environments, such as forests 10. Many strategies have also been proposed for human motion classification including Empirical Mode Decomposition 11, Support Vector Machines 12, artificial neural networks 13, and linear predictive coding 14. In most of these papers, human motions have been usually studied in only one environment, and no comparisons have been made between different types of scenarios. Contrarily, this thesis deals with a much wider range of environments for data collection, including foliage clutter, and investigates the variability in the classification rate for a SVM classifier across each environment. The thesis is organized as follows. A description of the experimental setup and the data collection environments are given in Chapter 2. Chapter 3 outlines the process of feature extraction from the micro- Doppler spectrogram. Lastly, Chapter 4 shows the classification results for the separation of crawling,

11 2 walking, and jogging movements across different environments and for different feature spaces. In Chapter 5, we present concluding remarks and explore possibilities for future research.

12 3 Chapter 2 Hardware and Environment Description 1. Experimental Setup Experimental micro-doppler data were collected from six human subjects using a continuous-wave monostatic Doppler radar operating in the C-band at 6.5 GHz. A simplified block diagram of the radar system is shown in Figure 1. The oscillator (OSC) generates a 6.5-GHz signal at a power level of 17 dbm. The signal is split into two halves in a power divider. One half is amplified in a power amplifier of gain 25 db and transmitted via a pyramidal horn antenna of gain 20 db. The other half is sent to the receiver to downconvert the received signal containing the micro-doppler signal. In order to distinguish between positive and negative Doppler shifts, phase coherent in-phase/quadrature (I/Q) demodulation was implemented. Coherent radar is necessary for separating multiple targets in the time-frequency domain since non-coherent radar cannot distinguish positive from negative Doppler shifts. The received signal is collected by an identical horn antenna, amplified in a low noise amplifier (LNA) of 20 db gain and 1 db noise figure. The amplified signal is downconverted to the I and Q components, which are then sent to the computer for further processing. All data are taken at an aspect angle of zero degrees. The duration of each data file is 2.5 seconds and the subjects move towards the radar during the data acquisition process. Figure 3: Simplified block diagram of the Doppler radar system operating in the C-band at 6.5 GHz.

13 4 2. Environmental Description Data collection was performed in four environments: (1) free space, (2) through-wall, (3) leaf tree, (4) needle tree. Each environment, except for free space, under which data were collected is shown in the Figure 2. Measurements were first performed in an indoor environment under line-of-sight conditions. The antennas were mounted approximately 1.3 m above the ground. For each motion, the subject moved directly toward the radar. Each activity was measured five times per subject. The range to the target varied between 9 m and 0.5 m during the unobstructed measurements. The stand-off distance for through-wall measurements was about 60 cm from the front of the wall. The minimum and maximum distance to the human target stood from the wall ranged from 1.5 m to 3 m. A similar arrangement was performed for every other environment.

14 5 (a) (b) Figure 4: Experimental setup in each environment: (a) through-the-wall, (b) leaf tree foliage, (c) and needle tree foliage. (c) There was a non-negligible wind blowing in the course of outdoor data collection for leaf and needle tree foliage. This wind factor was characterized by intermittent wind bursts of up to 5 10 mph which caused relatively greater noise in the received signal for needle tree data. Wind at this speed is particularly problematic because it is at the same speed typical of human movements of walking and

15 6 jogging. The wind is definitely an important factor that can affect the classification rate for through-foliage clutter, which is observed in the relatively large misclassification rate for needle tree foliage shown later in this thesis. 3. Time-frequency analysis: The goal of time-frequency analysis is to detect what frequencies are present in a signal, how strong they are, and how they change over time. The study of signals with time-varying frequency content has motivated the development of large array time-frequency representations (TFRs). Conventional signal processing tools, such as the Fourier transform, are unsuitable for signals that contain time-varying frequency content and thus other tools for analysis must be sought. For the purposes of micro-doppler signal analysis, it is not enough to only know what frequencies are present in the received signal. The theory behind common TFRs along with their application to non-stationary synthetic data will be examined below. The four TFRs utilized are the short-time Fourier transform, wavelet transform, Wigner-Ville distribution, and the Hilbert Huang transform. The properties of each transform, along with merits and demerits of each time-frequency transformation on the synthetic data, will be discussed. 3.1 Short-time Fourier transform The short-time Fourier transform (STFT) is a linear TFR useful for analyzing piece-wise stationary signals. It is composed of two steps: first, the signal is divided into two time segments and then the spectrum of each segment is obtained via the Fourier transform. This procedure results in 3D representation which displays the evolution of frequency content over time. Mathematically, it is expressed by the following equation: + STFT(t, f) = x(τ)h ( τ t)e j2πfτ dτ where h(t) is the time-window centered at t = 0 which is used to extract time segments. The requires the window to have unit energy:

16 7 + h(t) 2 dt = 1 The STFT is the most popular TFR and because it has low computational complexity and is simple to implement. The STFT is limited by the Gabor-Heisenberg uncertainty principle: resolution in time and frequency cannot be made arbitrarily small simultaneously. t w 1 2 Thus, an inherent limitation for joint time-frequency resolution exists for the STFT. As the length of the window increases, time-resolution is lost and frequency-resolution is gained. As the length of the window decreases, time-resolution is gained while frequency-resolution is lost. 3.2 Continuous Wavelet transform The continuous wavelet transform (CWT) is a linear TFR obtained by decomposing a signal into shifted and scaled versions of a mother wavelet. Mathematically, the CWT can be expressed as, CWT(t, a) = 1 a + τ t x(τ)w ( a ) dτ where a is the scale and w(t) is the mother wavelet. To obtain an admissible representation, w(t) must have zero-mean, i.e. + w(t)dt = 0 The most common mother wavelets are Mexican-Hat, Morlet, or Daubechies wavelets. The CWT, leads to a time-scale representation since it displays the signal time-frequency evolution at different scales. However, there is a direct correlation between scale and frequency. If the central frequency of the mother wavelet w(t) isf 0, the scale a corresponds to the frequency f = f 0 /a. In contrast to the STFT, the CWT is a multi-resolution technique that favors time-resolution at high-frequency and the frequency-resolution at low-frequencies.

17 8 3.3 Wigner-Ville distribution Unlike the previous two TFRs, the Wigner-Ville distribution (WVD) focuses on the decomposition of the signal energy in the time-frequency plane as opposed to decomposition of the signal itself. The WVD is expressed mathematically as follows, + WVD(t, f) = x (t + τ 2 ) x (t τ 2 ) e j2πfτ dτ The WVD is not constrained by the Heisenberg-Gabor inequality. However, the nonlinearity of the WVD introduces cross-term interference. These interference terms can render the time-frequency representation difficult to interpret. To reduce this interference, the analytic signal of the input signal is usually analyzed instead of the signal itself. 3.4 Hilbert-Huang Transform The Hilbert-Huang transform (HHT) is a signal analysis method specifically designed to fill the void for analysis nonlinear and non-stationary data. Unlike the previous methods, HHT is data-driven, which means that it does not use a priori basis functions and instead adapts to the signal. The HHT is composed of two parts: empirical mode decomposition (EMD) and Hilbert spectral analysis (HSA). The EMD process is as follows. First, the extrema (minima and maxima) of the signal are located and connected via spline interpolation to form an envelope. The mean of this envelope is designated as m 1 and then subtracted from the original signal, forming the proto-imf, h 1 = x(t) m 1. The same process is then applied to the proto-imf until the definition of an IMF has been satisfied. h 11 = h 1 m 11. This sifting process is repeated k times until h 1k is an IMF, that is h 1k = h 1(k 1) m 1k, thus the first IMF component is obtained, i.e. IMF 1 = h 1k. Then separate IMF 1 from the original times series by x(t) IMF 1 = r 1. Treat r 1 as the new data and subject it to the same sifting process as above. Repeat this procedure on all the subsequent r i s, i.e. r 2 = r 1 IMF 2,,. r n = r n 1 IMF n. The final result is n x(t) = IMF i (t) + r n (t) i=1

18 9 An IMF is defined by the following criterion: 1. In the whole dataset, the number of extrema and the number of zero-crossings must either equal or differ at most by one. 2. At any point, the mean value of the envelope defined by the local maxima and the envelope defined by the local minima is zero. 4. Characteristics of non-ideal I/Q demodulation: To implement ideal I/Q demodulation, the following requirements must be satisfied: The local oscillator and the transmitter must be identical The I and Q channels must have perfectly matched transfer functions over the signal bandwidth The oscillators used to demodulate the I and Q channels must be exactly in quadrature, that is, 90 degrees out of phase with one another When these requirements are not met, the received signal does not only contain the desired signal component (with a slightly modified amplitude), but also an image component with a different amplitude and a conjugated phase function, as well as a complex DC term. The image component is an error resulting from the amplitude and phase mismatches; the DC component is the direct result of the individual channel DC offsets. A diagram depicting the differences between ideal and non-ideal I/Q demodulation is shown below in Figure 2. a. Correction of I and Q channel imbalances in the time-domain: An approach to the correction of I and Q imbalance in the time-domain will now be presented. The non-ideal effects of I/Q demodulation can be represented below by the following equations: I(t) = Acos(wt + φ) + a Q(t) = A(1 + ε)sin(wt + φ + θ) + b

19 10 where (1 + ε) represents the amplitude imbalance between channels, θ represents the phase imbalance between channels, and a and b represent the DC offset of each channel respectively. After subtracting the DC offset, which is simply the mean value of the signal, the I and Q channels become: I (t) = Acos(wt + φ) Q (t) = A(1 + ε)sin(wt + φ + θ) To compensate for the amplitude and phase imbalance, the following matrix can be setup with the error correction terms c and d. [ I (t) 0 ]= [1 Q (t) c d ] [ I (t) Q (t) ] d = c = tan(θ) 1 (1 + ε)cos (θ) Unlike I(t) and Q(t), I (t) and Q (t) are exactly in quadrature with one another. The task now is to actually relate the c and d terms to the physical signal at hand. After some derivations, it was shown in Ref. X that the c and d terms can be solved for in the following manner: c = (I, Q) sqrt( I 2 Q 2 (I, Q) 2 ) I 2 d = sqrt( I 2 Q 2 (I, Q) 2 )

20 11 where (I, Q) represents the inner product operation and I represents the norm of I. These correction terms can now be processed to counteract the imbalance errors plaguing the I/Q system.

21 12 Chapter 3 Comprehensive Data Display 1. Periodic stationary movements Figures 3 and 4 show typical MDS of the first three stationary movements in a free-space (FS) and through-wall (TW) environment, respectively. They are arm swinging, boxing, and waving. For arm swinging, the human subject stands still and swings his/her arms as if walking. For boxing, the human subject alternates between right hand and left hand punches in the direction of the radar. For waving, the human subject stands still and gestures back and forth in the direction facing the radar. All motions have an average Doppler shift of 0 Hz, as expected, since the human is stationary. What distinguishes each movement is the total Doppler spread, or total bandwidth, the swing rate, and the sequence in which positive and negative frequency peaks occur. Waving has a smaller Doppler spread of roughly 100 Hz of the three movements, while the other two have a total bandwidth of about 200 Hz. For arm-swinging, positive and negative shifts occur simultaneously, whereas in the other movements, there is a delay between the occurrence of one and the other. In other words, for arm swinging, the positive and negative shifts occur in parallel (simultaneously) but the positive and negative shifts in boxing and waving occur sequentially (one after the other). The overall structure of the signatures remains unchanged for through-wall MDS, although the lower SNR is observed. Fading effects are not evident in these signatures since they are stationary.

22 13 Figure 3: Free space stationary periodic movements (from left to right): Arm swinging, boxing, and waving. Figure 4: Through-wall stationary periodic movements (from left to right): Arm swinging, boxing, and waving. 2. Non-periodic stationary movements Figure 5 and 6 shows non-periodic stationary movements in the free-space and through-wall environments, respectively. The movements are crouching to standing, sitting to standing, and squaring up to shoot. For crouching-to-standing, the human subject begins in a crouching position and then stands up and remains still. For sitting to standing, the human subject begins in metal chair and then stands up and remains still. For squaring up to shoot, the human subject begins standing still, takes a single lunge towards the radar system, and then raises both arms upward. Based on Figures 6 and 7 there are clearly distinctive differences between each motion. Crouching-to-standing only contains torso movement whereas sitting-to-standing requires a thrust forward out of the chair. This thrust manifests itself as a triangular peak in the middle of the movement s spectrogram. Sitting to standing contains a lunge forward that is noticeably absent in the crouching to

23 14 standing case. Squaring up to shoot involves the human target lunging forward and then raising both arms toward the radar. It is very similar to sitting to standing but one can also see the human raising their arms, indicated by the vertical peak near the end of the signature. The motion squaring up to shoot can be thought of as a sequence of two motions lunging forward and then raising arms. Again, despite the attenuation from free-space to through-wall environments, the overall structure of the MDS remains unchanged. Fading effects are again not evident in these signatures since they are stationary. Figure 5: Free space stationary non-periodic movements (from left to right): Crouching to standing, sitting to standing, and squaring up to shoot. Figure 6: Through-wall stationary non-periodic movements (from left to right): Crouching to standing, sitting to standing, squaring up to shoot.

24 15 3. Object-free movements Figure 7 and 8 shows object-free forward-moving movements in free-space and through-wall environments, respectively. The movements are walking, walking with arms crossed, limping, and crawling. For walking, the human subject begins at a distance of about 7.5 meters and walks toward the radar. The same is the case for walking with arms crossed. For limping, the human subject walks toward the radar while dragging one foot to simulate being wounded. For crawling, the human subject crawls in a baby-like manner towards the radar. Note how the stride rate is exaggerated for the limping motion as opposed to non-limping motion. There is a noticeable dragging effect that occurs in limping which does not occur for any other walking motion. Thus, extracting stride rate as a feature for classification may be useful to separate walking motion from limping, which can potentially be used in a military context for detecting wounded persons. The stride rate disappears for crawling and we notice a more-or-less flat Doppler shift with small arm movements atop the main shift. Crawling is characterized by a small Doppler shift with periodic peaks that indicate arm movement and it has the lowest Doppler shift of each movement. Each walking activity has approximately the same torso Doppler frequency, as expected, which is about 50 Hz. This Doppler shift corresponds to a speed of 1.15 m/s, which is how fast humans typically walk. Again, these MDS signatures are nearly identical to their free-space counterparts, differing only in amplitude. Fading effects are now obvious in the TW signatures. The change in RCS over the duration of the MDS indicates fading and may yield insight into whether the target is approaching or receding from the radar system. Besides the attenuation and fading from FS to TW, the overall structure of the MDS remains unchanged in each environment.

25 16 Figure 7: Free-space forward-moving (starting from top-left and moving clockwise): Walking, walking with arms crossed against chest, limping, crawling. Figure 8: Free-space forward-moving (starting from top-left and moving clockwise): Walking, walking with arms crossed against chest, limping, crawling.

26 17 4. Carrying an object Figures 9 and 10 show movements for a human target walking toward the radar while holding the following objects: cylindrical brick, weighted backpack, cell phone up to ear, and corner reflector. Each movement consists of a human subject walking towards the radar in a casual manner while holding onto some particular object. Observe how walking with the corner reflector emphasizes the torso movement of the human subject. The corner reflector can be used as a surrogate for any object with a high RCS that may be carried by a human target. A feature that measures the ratio of the torso amplitude to the arm/leg amplitude may be used to discriminate regular walking from walking with an object with a high RCS. Observe how walking with the cell phone eliminates the arm movement of one arm (the arm which is holding the phone up to the ear), but movement of one arm is still visible. Walking while holding the brick has a similar spectrogram with the walking case, but it has a slightly narrower frequency spread. In general, carrying different objects does not make a noticeable difference in the spectrograms here except for the corner reflector. Besides the attenuation and fading from FS to TW, the overall structure of the MDS remains unchanged in each environment. Fading effects are now obvious in the TW signatures.

27 18 Figure 9: Free space forward moving while carrying an object (starting from top-left moving clockwise): Walking with brick, walking with backpack, holding phone up to ear, corner reflector. Figure 10: Through-wall forward-moving while carrying an object (starting from top-left moving clockwise): Walking with brick, walking with backpack, walking with phone up to ear, walking with corner reflector.

28 19 5. Multi-target movements Figures 11 and 12 show multi-target movements in free-space and through-wall environments, respectively. The movements are two targets walking in opposite directions, two targets walking towards the radar, two targets running in opposite direction, and two targets walking in the opposite direction at different speeds. Multi-target movements present a challenge because the presence of several people simultaneously in the radar field of sight could involve interferences. From the figures, we can see that it is possible to separate multiple targets that are walking in opposite directions if the total bandwidth of each target s movement does not interfere with the other. If multiple targets are walking in the same direction, however, the multiple targets will appear as one target. It gives no indication at all that there are multiple targets on the scene. Fading effects are present in both the free-space and through-wall MDS. Total Doppler bandwidth may be useful for distinguishing two targets. Doppler frequencies would thus be mixed and the received signal could not be easily separated anymore. It was found that while through-wall propagation affected the magnitude response of the Doppler spectrogram in the form of attenuation and fading, it only introduced very minor distortions on the actual Doppler frequencies from the body parts. Besides the attenuation and fading from FS to TW, the overall structure of the MDS remains unchanged in each environment.

29 20 Figure 11: Free-space multi-target movements (starting from top-left moving clockwise): Two targets walking in opposite directions, two targets walking in the same direction, two targets running in opposite directions, two targets walking in opposite directions at different speeds. Figure 12: Through-wall multi-target movements (starting from top-left and moving clockwise): Two targets walking in opposite directions, two targets walking in the same direction, two targets running in opposite directions, two targets walking in opposite directions at different speeds.

30 21 6. Variability across human subjects The degree of variability in MDS across subjects depends on the movement being performed. Table 1 lists the height and weight of the human subjects on whom data was collected. For movements that only consist of arm movements such as stationary arm swinging or waving there tend to be less variability across subjects than movements that involve the whole body such as crawling. Height and weight have a greater impact on the MDS for movements that involve the entire body than they do for stationary movements. Figures 13 and 14 show the variability of human MDS across different subjects for stationary arm swinging and crawling, respectively. Despite there being slight variations in the MDS across subjects for stationary arm swinging, the overall structure of the MDS remains the same. Some of the heavier subjects give greater torso responses which can be seen from the increased amplitude in the MDS compare the torso response of heavier subject (subject 6) with that of a lighter subject (subject 5). Unlike stationary arm swing, there appear to be significant structural changes in crawling across subjects. Some subjects have a more uniform and periodic crawl (subjects 1 and 5) than the others (subjects 3 and 6). Thus it is important to recognize what features remain the same across subjects when designing classifiers. TABLE 1: PHYSICAL CHARACTERISTICS OF HUMAN SUBJECTS Subject Identifier Height (m) Weight (kg)

31 22 Figure 13: Stationary arm swinging MDS across subjects (from top left going clockwise): Subjects 1, 3, 5, and 6 Figure 14: Crawling MDS across subjects (from top left going clockwise): Subjects 1, 3, 5, and 6

32 23 In this thesis, we present the radar MDS of various human activities in free-space and throughwall environments. The spectrograms show rather interesting and distinct signatures depending on the activity. The following experiments were used to analyze the different effects due to through-wall transmission on human MDS. It was found that while through-wall propagation affected the magnitude response of the Doppler spectrogram in the form of attenuation and fading, it only introduced very minor distortions on the actual Doppler frequencies from the body parts. We also see that it is possible to separate multiple targets that are moving in opposite directions under the proper circumstances. If the targets are moving at different speeds, then they can be separated no matter what direction each target is heading towards. But multiple targets can only be separated when they are traveling in the same direction at different speeds. This may change significantly depending on aspect angle since a target at oblique incidence only gives a radial component velocity which may cause the appearance of the target to be slower than it actually is. The extracted feature values are listed in Tables 2 and 3 for free space and through-wall scenarios, respectively. The mean and the ±1 standard deviation values for each feature are shown. The standard deviations account not only for the variability within each subject but also the variability between different subjects. It is observed that the selected features for each activity are mostly consistent under both free space and through-wall scenarios, although a few cases show significant differences. The differences may be possibly be attributed the effects of multiple scattering between the human and the back surface of the wall. We believe that these feature values would be useful in classification of human activities using radar micro-doppler signatures. While our derived features are based upon a 6.5-GHz transmit frequency, the feature values indicated in Tables 3 and 4 can be scaled to other transmit frequencies since the Doppler frequency is generally proportional to the actual transmit frequency. For example, we obtained a mean cadence frequency (Feature 5) value of Hz for walking (see Table 3), which when scaled from our transmit frequency of 6.5 GHz to a GHz transmit frequency comes to Hz ( ). This value is virtually identical to a cadence frequency of 2 Hz obtained using a GHz system in

33 24 Ref. 15. Similarly, the mean value of the average Doppler frequency (Feature 1), Hz, for walking, when scaled to a transmit frequency of 2.4 GHz is computed as Hz ( ), which is quite close to the value of Hz listed in Ref. 16.

34 25 TABLE 2: LIST OF FEATURES USEFUL FOR FREE SPACE HUMAN ACTIVITY CLASSIFICATION Activity Mean Doppler frequency (Hz) Total bandwidth (Hz) Doppler offset (Hz) Bandwidth without MDS (Hz) Cadence frequency (Hz) Period (Hz) Stationary arm swinging ± ± ± ± ± Stationary boxing ± ± ± ± ± Stationary waving ± ± ± ± ± Sitting to standing ± ± ± ± Crouching to standing ± ± ± ± Squaring up to shoot ± ± ± ± Walking ± ± ± ± ± Walking with arms crossed against chest ± ± ± ± ± Limping ± ± ± ± ± Crawling ± ± Walking with brick ± ± ± ± ± ± ± ± Walking with corner reflector ± ± ± ± ± Walking with backpack ± ± ± ± ± Walking with phone up to ear ± ± ± ± ±

35 26 TABLE 3: LIST OF FEATURES USEFUL FOR THROUGH-WALL HUMAN ACTIVITY CLASSIFICATION Activity Mean Doppler frequency (Hz) Total bandwidth (Hz) Doppler offset (Hz) Bandwidth without MDS (Hz) Cadence frequency (Hz) Period (Hz) Stationary arm swinging ± ± ± ± ± Stationary boxing ± ± ± ± ± Stationary waving ± ± ± ± ± Sitting to standing ± Crouching to standing ± ± ± ± ± ± ± Squaring up to shoot ± ± ± ± Walking ± ± ± ± ± Walking with arms crossed against chest ± ± ± ± ± Limping ± ± ± ± ± Crawling ± ± ± ± ± Walking with brick ± ± ± ± ± Walking with corner reflector ± ± ± ± ± Walking with backpack ± ± ± ± ± Walking with phone up to ear ± ± ± ± ±

36 27 7. Human gait micro-doppler signature database Human gait micro-doppler signature database The spectrograms shown below represent the beginning of an attempt to build a database of human gait MDS. We plan to fill this database with over 100 movements for multiple human subjects. These movements may be classified into categories such as fundamental gait movements, transition movements, communicative hand gestures, common warfare movements, and so on. Examples of fundamental gait movements may include a single step towards the radar or a single arm-swing cycle while examples of transition movements may include crouching-to-standing or standing-to-sitting in a chair. In future research, we will think of ways to discriminate signatures. In this report however we will only display the signatures. The following MDS shown in Figures were recorded for a wide variety of activities in an outdoor environment using a continuous-wave X-band radar system operating at 6.5 GHz. The spectrogram is used simply because it is convenient to implement and gives a good rough overview of the signature. The optimal time-frequency representation for classification of human gait MDS is still a topic we are pursuing. All the signatures shown below were taken from subject #2 from Table 1.

37 28 Figure 15: MDS of stationary human standing still (breathing) in free space. Figure 16: MDS of human standing still and breathing heavily in free space.

38 29 Figure 17: MDS of human standing still and holding breath in free space. Figure 18: MDS of sitting in chair breathing heavily.

39 30 Figure 19: MDS for sitting in chair holding breath. Figure 20: MDS for single step towards, keep back toe on ground.

40 31 Figure 21: MDS for single step towards, bring both feet together. Figure 22: MDS for two steps toward, keep back toe on ground.

41 32 Figure 23: MDS for right arm swing cycle. Figure 24: MDS for left arm swing cycle.

42 33 Figure 25: MDS for both arm swing cycle, same direction. Figure 26: MDS for both arm swing cycle, opposite direction.

43 34 Figure 27: MDS for raise right arm forward once. Figure 28: MDS for raise left arm forward once.

44 35 Figure 29: MDS for raise both arms forward once at same time. Figure 30: MDS of raise both arms forward once at different time.

45 36 Figure 31: MDS for oscillate between raising left and right arm. Figure 32: MDS for single right hand punch towards starting at boxing position.

46 37 Figure 33: MDS of single left hand punch towards starting at boxing position. Figure 34: MDS of single punch both hands starting at boxing position.

47 38 Figure 35: MDS of punch while oscillating between hands. Figure 36: MDS of turning around, facing towards.

48 39 Figure 37: MDS of turning around, facing away. Figure 38: MDS of kicking away as if running.

49 40 Figure 39: MDS of forward hop. Figure 40: MDS of standing still to sitting.

50 41 Figure 41: MDS of standing still to crouching. Figure 42: MDS of walking towards and stopping.

51 Figure 43: MDS of sitting to standing still. 42

52 43 Chapter 4 Feature Extraction and Classification 1. Feature extraction The following features are extracted from the micro-doppler spectrogram 17,18 : (1) Signal energy, (2) Average Doppler frequency, (3) Total bandwidth, (4) Doppler offset, (5) Bandwidth without micro- Doppler, (6) Standard deviation (STD) of lower frequency envelope, and (7) Standard deviation (STD) of upper frequency envelope. The extracted features are shown in Figure 45. Different combinations of these seven features can be used for classification depending on what features best suit the classification task at hand. The optimal combination of features for optimal classification rates are not the subject of this thesis. Instead, we use three test cases of features based on the results in Ref. [19-21]. For optimal classification performance, it is necessary to choose the features intelligently depending on the classification task. For the purposes of the particular classification task of separating crawling, walking, and jogging, only a subset of the features mentioned above are actually used for classification. Figure 44: Illustration of features extracted from an experimental spectrogram of stationary arm swinging.

53 44 To compare the distribution of features across environments, scatter graphs of features from each environment are presented in Figure 46. Table 4 shows the numeric distribution of all extracted features using the mean and standard deviation. The feature space displayed in these graphs does not represent the actual features used in our classifier. (a) (b) (c) (d) Figure 45: Two-dimensional un-normalized feature space comparison for each environment: (a) free-space, (b) through-the-wall, (c) leaf tree foliage, and (d) needle tree foliage. The feature along the x-axis is mean Doppler frequency while the feature along the y-axis is the total bandwidth.

54 45 The feature statistics are shown across environment in Table 4. This table includes all 7 features extracted from the spectrogram. The increase in standard deviation indicates the noise added in both the outdoor foliage environments and through-the-wall. This spread in standard deviation is representative for crawling and jogging as well, but out of interest of space we chose to only include a table for walking. There is a significant increase in the standard deviation for outdoor tree foliage. The standard deviations are included. The standard deviation shows how much the average. There is no big increase in Doppler offset, or in the standard deviation. There is a very large increase for bandwidth, however. TABLE 4: COMPARISON OF FEATURE STATISTICS FOR WALKING MOTION ACROSS DIFFERENT ENVIRONMENTS Environment Energy Avg. Total BW Doppler BW without STD lower STD upper (arbitrary Doppler (Hz) offset MDS (Hz) (Hz) units) (Hz) (Hz) (Hz) Free space ± ± ± ± ± ± ± Through-the wall ± ± ± ± ± ± ± Needle tree foliage ± ± ± ± ± ± ± Leaf tree foliage ± ± ± ± ± ± ±

55 46 2. Classification Approach Support vector machines (SVMs) are a supervised learning algorithm used to analyze and classify data. Given labelled training sets, SVMs can determine an optimal hyperplane to use as a decision boundary in order to separate data points between multiple classes. The hyperplane is maximal when it optimizes the distance between a decision boundary and the nearest data points between the two classes without any error. The margin is defined as the minimum distance between the training vector and the hyperplane. An input vector that lies on this plane is defined as the support vector. SVMs are designed to solve binary classification problems. In order to extend SVMs to multiclass problems, like the problem at hand, there are two approaches that can be used. The first approach is oneversus-one and the second approach is one-versus-all. In this work, one-versus-all classification is used. Classification of new instances for the one-versus-all case is done by a winner-takes-all strategy, in which the classifier with the highest output function assigns the class. After initial testing, we chose a radialbasis kernel for the SVM. Appropriate values for the parameters, λ, and a penalty parameter, C, being used by the SVM were determined through a cross-validation using all available target sequences. We implement a classifier to separate the following three human movements: crawling, walking, and jogging towards the radar at an aspect angle of 0 degrees. A spectrogram for each movement is shown in Figure 5. After our human motion micro-doppler dataset is generated from experiments, we split the data into training and test sets. We use a 75/15 distribution for the dataset. Once the data are split, the features are scaled within the range of 0 to 1. The same scaling factors used on the training data are then used to scale the testing data. Scaling is necessary for data which may be orders of magnitude higher and thus dominate smaller terms. We also implement 6-fold cross validation on the training set. Cross validation divides the training set into N partitions in which N 1 partitions are used to train the model and the remaining partition is used for testing. The classification results depend heavily on both the number of features used and what specific features are used. To illustrate this point, we include the SVM classification results for three cases of

56 47 features. The classification results for different feature spaces are shown here. There are three cases of features investigated: Case 1: (1) Average Doppler frequency, (2) Total bandwidth Case 2: (1) Doppler offset, (2) Bandwidth without MDS, (3) Total bandwidth Case 3: (1) Average Doppler frequency, (2) Total bandwidth, (3) Doppler offset, (4) Bandwidth without MDS. Figure 46: Spectrograms of three different types of human motions in the free-space environment: (a) crawling, (b) walking, and (c) jogging. It is not known a priori which set of features will result in the highest classification rate and therefore they must be experimentally determined. However, it was shown in other works that the most effective features were the torso frequency, total bandwidth, Doppler offset, and bandwidth without MDS, which result in the greatest separability for moving human targets 5. It was also shown that the classification rate increases asymptotically as the number of features increases.

57 48 3. Classification Results We train a Support Vector Machine (SVM) with the feature vectors mentioned in Section 4.1 using a radial basis kernel (RBF). We then validate the classifier s accuracy on unused data (i.e., not used in the training) and observe the classification results, which are shown below in Table 2. The features introduced in Section 3 are extracted from the measured spectrograms. An accessible format for the results of classification is the confusion matrix. The features used are the same as the features for Case 3. These features were used since they gave the highest classification rate across the board for each environment, as can be seen in Table 5. Confusion matrices are shown in Tables 6-9 for each data collection environment averaged over 10 iterations. TABLE 5: SVM CLASSIFICATION RATE (10 ITERATIONS): 75/15 SPLIT Features Free Through- Leaf Needle used space the-wall tree foliage tree foliage Case % 72.22% 83.56% 53.88% Case % 77.44% 88.00% 80.63% Case % 87.78% 97.89% 81.12% For this particular classification task, free space gave the highest classification rate at 98.11%. The second best classification task occurs for leaf tree foliage at 97.89%. The classification rates are highest for the data collection in free space, as one would expect. Due to windy conditions for needle tree data collection, the classification rates were the lowest. The importance of the number of features can be seen here since for Case 1, the classification rate for needle tree data is very low but significantly increases as the number of features used increases for Case 2 and 3. A wind factor resulted in consistently higher misclassification rates for the needle tree. It is shown that increasing the number of features generally increases the classification rate as well. And in windy conditions it can be seen that for two features (Case 1), the needle tree classifier is almost unusable at 46% classification accuracy.