Robust Respraton Detecton from Remote Photoplethysmography MARK VAN GASTEL, 1,* SANDER STUIJK, 1 AND GERARD DE HAAN 1,2 1 Department of Electrcal Engneerng, Endhoven Unversty of Technology, PO Box 513, 56MB, Endhoven, The Netherlands 2 Phlps Research, Hgh Tech Campus 36, 5656AE, Endhoven, The Netherlands * m.j.h.v.gastel@tue.nl Abstract: Contnuous montorng of respraton s essental for early detecton of crtcal llness. Current methods requre sensors attached to the body and/or are not robust to subject moton. Alternatve camera-based solutons have been presented usng moton vectors and remote photoplethysmography. In ths work, we present a non-contact camera-based method to detect respraton, whch can operate n both vsble and dark lghtng condtons by detectng the respratory-nduced colour dfferences of the skn. We make use of the close smlarty between skn colour varatons caused by the beatng of the heart and those caused by respraton, leadng to a much mproved sgnal qualty compared to sngle-channel approaches. Essentally, we propose to fnd the lnear combnaton of colour channels whch suppresses the dstortons best n a frequency band ncludng pulse rate, and subsequently we use ths same lnear combnaton to extract the respratory sgnal n a lower frequency band. Evaluaton results obtaned from recordngs on healthy subjects whch perform challengng scenaros, ncludng moton, show that respraton can be accurately detected over the entre range of respratory frequences, wth a correlaton coeffcent of.96 n vsble lght and.98 n nfrared, compared to.86 wth the best-performng non-contact benchmark algorthm. Furthermore, evaluaton on a set of vdeos recorded n a Neonatal Intensve Care Unt (NICU) shows that ths technque looks promsng as a future alternatve to current contact-sensors showng a correlaton coeffcent of.87. 16 Optcal Socety of Amerca OCIS codes: (17.17) Medcal optcs and botechnology magng system; (17.147) Blood or tssue consttuent montorng; (28.28) Remote sensng and sensors; (17.366) Lght propagaton n tssues References and lnks 1. C. Seymour, J. Kahn, C. Cooke, T. Watkns, S. Heckbert, and T. Rea, Predcton of crtcal llness durng out-of-hosptal emergency care, J. Am. Med. Assoc. 34(7), 747 754 (1). 2. F. Q. Al-Khald, R. Saatch, D. Burke, H. Elphck, and S. Tan, Respraton rate montorng methods: a revew, Pedatr. Pulmonol. 46(6), 523 529 (11). 3. P. H. Charlton, T. Bonnc, L. Tarassenko, D. A. Clfton, R. Beale, and P. J. Watknson, An assessment of algorthms to estmate respratory rate from the electrocardogram and photoplethysmogram, Physol. Meas. 37(4), 61 626 (16). 4. W. Karlen, A. Garde, D. Myers, C. Scheffer, J. Ansermno, and G. Dumont, Estmaton of respratory rate from photoplethysmographc magng vdeos compared to pulse oxmetry, IEEE J. Bomed. Health Inform. 19(4), 1331 1338 (15). 5. F. Adb, H. Mao, Z. Kabelac, D. Katab, and R. C. Mller, Smart homes that montor breathng and heart rate, n Proceedngs of the 33rd Annual ACM Conference on Human Factors n Computng Systems, pp. 837 846 (15). 6. J. Fe and I. Pavlds, Thermstor at a dstance: unobtrusve measurement of breathng, IEEE Trans. Bomed. Eng. 57(4), 988 998 (1). 7. E. Greneker, Radar sensng of heartbeat and respraton at a dstance wth applcatons of the technology, n Radar 97 (Conf. Publ. No. 449), pp. 15 154 (1997). 8. M. Bartula, T. Tgges, and J. Muehlsteff, Camera-based system for contactless montorng of respraton, n Engneerng n Medcne and Bology Socety, 13 35th Annual Internatonal Conference of the IEEE, pp. 2672 2675 (13). 9. A. Henrch, F. van Heesch, B. Puvvula, and M. Rocque, Vdeo based actgraphy and breathng montorng from the bedsde table of shared beds, J. Ambent. Intell. Humanz. Comput. 6(1), 17 1 (15). 1. F. Zhao, M. L, Y. Qan, and J. Z. Tsen, Remote measurements of heart and respraton rates for telemedcne, PLOS ONE 8(1), e71384 (13).
11. W. Verkruysse, L. O. Svaasand, and J. S. Nelson, Remote plethysmographc magng usng ambent lght, Opt. Express 16(26), 21434 21445 (8). 12. M.-Z. Poh, D. J. McDuff, and R. W. Pcard, Advancements n noncontact, multparameter physologcal measurements usng a webcam, IEEE Trans. Bomed. Eng. 58(1), 7 11 (11). 13. Y. Sun, S. Hu, V. Azorn-Pers, S. Greenwald, J. Chambers, and Y. Zhu, Moton-compensated noncontact magng photoplethysmography to montor cardorespratory status durng exercse, J. Bomed. Opt. 16(7), 771 771 9 (11). 14. F. Bousefsaf, C. Maaou, and A. Prusk, Contnuous wavelet flterng on webcam photoplethysmographc sgnals to remotely assess the nstantaneous heart rate, Bomed. Sgnal Process. Control 8(6), 568 574 (13). 15. P. Leonard, N. R. Grubb, P. S. Addson, D. Clfton, and J. N. Watson, An algorthm for the detecton of ndvdual breaths from the pulse oxmeter waveform, J. Cln. Mont. Comput. 18(5 6), 39 312 (4). 16. K. H. Chon, S. Dash, and K. Ju, Estmaton of respratory rate from photoplethysmogram data usng tme frequency spectral estmaton, IEEE Trans. Bomed. Eng. 56(8), 54 63 (9). 17. S. G. Flemng and L. Tarassenko, A comparson of sgnal processng technques for the extracton of breathng rate from the photoplethysmogram, Int. J. Bol. Med. Sc. 2(4), 232 236 (7). 18. A. Skdar, S. K. Behera, and D. P. Dogra, Computer vson guded human pulse rate estmaton: A revew, IEEE Rev. Bomed. Eng. 9, 91 15 (16). 19. C. G. Caro, The mechancs of the crculaton (Cambrdge Unversty Press, 12).. G. de Haan and V. Jeanne, Robust pulse-rate from chromnance-based rppg, IEEE Trans. Bomed. Eng. 6(1), 2878 2886 (13). 21. G. de Haan and A. van Leest, Improved moton robustness of remote-ppg by usng the blood volume pulse sgnature, Physol. Meas. 35(9), 1913 1926 (14). 22. M. Hülsbusch and V. Blazek, Contactless mappng of rhythmcal phenomena n tssue perfuson usng PPGI, Proc. SPIE 4683, 11 117 (2). 23. L. F. Corral Martnez, G. Paez, and M. Strojnk, Optmal wavelength selecton for noncontact reflecton photoplethysmography, 22nd Congress of the Internatonal Commsson for Optcs 811, 81191 7 (11). 24. S. Prahl, Optcal absorpton of hemoglobn, Oregon Medcal Laser Center, http://omlc.og.edu/ spectra/hemoglobn/ndex.html 15 (1999). 25. C. Tomas and T. Kanade, Detecton and trackng of pont features, Tech. rep., Int. J. Comput. Vson (1991). 26. J. Sh and C. Tomas, Good features to track, n Computer Vson and Pattern Recognton (CVPR). Proceedngs CVPR 94., 1994 IEEE Computer Socety Conference on, pp. 593 6 (1994). 27. E. Tur, M. Tur, H. I. Mabach, and R. H. Guy, Basal perfuson of the cutaneous mcrocrculaton: measurements as a functon of anatomc poston, J. Invest. Dermatol. 81(5), 442 446 (1983). 28. L. M. Nlsson, Respraton sgnals from photoplethysmography, Anesth. Analg. 117(4), 859 865 (13). 29. M. Lewandowska, J. Rumnsk, T. Kocejko, and J. Nowak, Measurng pulse rate wth a webcam - a non-contact method for evaluatng cardac actvty, n Computer Scence and Informaton Systems (FedCSIS), 11 Federated Conference on pp. 45 41 (11). 3. M. van Gastel, S. Stujk, and G. de Haan, Moton robust remote-ppg n nfrared, IEEE Trans. Bomed. Eng. 62(5), 1425 1433 (15). 31. L. A. Slvestr, Saunders comprehensve revew for the NCLEX-PN examnaton (Elsever Health Scences, 15). 32. L.-G. Lndberg, H. Ugnell, and P. Öberg, Montorng of respratory and heart rates usng a fbre-optc sensor, Med. Bol. Eng. Comput. 3(5), 533 537 (1992). 33. A. Johansson and P. Öberg, Estmaton of respratory volumes from the photoplethysmographc sgnal. part : expermental results, Med. Bol. Eng. Comput. 37(1), 42 47 (1999). 34. L. Nlsson, A. Johansson, and S. Kalman, Respratory varatons n the reflecton mode photoplethysmographc sgnal. Relatonshps to perpheral venous pressure, Med. Bol. Eng. Comput. 41(3), 249 254 (3). 35. J. L, J. Jn, X. Chen, W. Sun, and P. Guo, Comparson of respratory-nduced varatons n photoplethysmographc sgnals, Physol. Meas. 31(3), 415 425 (1). 36. P. S. Addson, J. N. Watson, M. L. Mestek, and R. S. Mecca, Developng an algorthm for pulse oxmetry derved respratory rate (rrox): a healthy volunteer study, J. Cln. Mont. Comput. 26(1), 45 51 (12). 1. Introducton Montorng of respraton s mportant n clncal care snce t provdes valuable nformaton of a person s health status. An abnormal respratory rate (RR) s a senstve early ndcator of crtcal llness that often accompanes, and may precede, changes n other vtal sgns such as heart rate, blood pressure, or reducton n perpheral oxygen saturaton (SpO 2 ) [1]. For example, events of apnea can lead to permanent bran damage and even death. Contnuous montorng of respraton has the potental to detect and prevent such events from occurrng. The common methods to montor respraton are contact-based and consequently requre one or more sensors attached to the body, e.g. electrodes or a belt [2]. Recently, Charlton et
al. [3] assessed the performance of 314 algorthms for the estmaton of RR from ECG and PPG waveforms under deal operaton condtons. They showed that most tme-doman technques perform better compared to frequency-doman technques. Ther superor performance may be explaned by the fact that the respratory sgnal s not requred to be quas-statonary, unlke frequency-doman technques. The PPG waveform contans three respratory features [4]. From the three PPG respratory features, the respratory nduced ntensty varatons,.e. baselne modulaton, provdes the hghest accuracy, and a combnaton, fuson, of all three features n general performed better compared to the features solely. It should however be noted that Charlton et al. used sngle-channel contact PPG sgnals, and furthermore, they benchmarked the algorthms under deal operaton condtons. Albet ther ablty to measure respraton, most of these methods are cumbersome and can therefore cause stress and dscomfort to the patent. Non-contact methods to montor respraton address these ssues. In ths paper, we ntroduce a non-contact-based respraton montorng system. Alternatve noncontact methods have been proposed n dfferent ranges of the electro-magnetc spectrum [5 7], e.g. radar-based or usng thermal cameras. These methods however requre expensve equpment whch lmt ther applcablty. Addtonally, non-contact methods have been documented usng low-cost cameras based on moton or remote PPG (rppg). Moton-based methods [8 1] detect the respratory nduced movements of the chest and/or abdomen. The challenge wth these methods s to dfferentate between respratory-nduced movements and other movements whch are not related to respraton. Furthermore, to clearly regster the mnute respratory-nduced chest movements, the range of vewponts of the camera s somewhat lmted. rppg-based methods extract respraton from respratory features present n the blood volume pulse sgnal [4, 11 14]. A number of approaches have been proposed to extract these features from the PPG waveform, ncludng wavelet decomposton [15], complex demodulaton [16] and auto-regresson [17]. The need for farly long tme wndows and assumptons on the regularty of the RR lmt the applcablty of these methods n real lfe condtons. Furthermore, all aforementoned contact and non-contact methods for RR measurement that use the PPG sgnal rely on sngle-channel PPG waveforms. Sngle-channel PPG waveforms do not allow to elmnate n-band dstortons, such as sensor nose and moton artfacts. Ths s especally problematc for non-contact based solutons snce these dstortons are typcally present n the rppg waveforms whch have a lower sgnal-to-nose rato (SNR) compared to contact PPG waveforms. In [18], t has been shown that robust pulse rate detecton s feasble when usng a mult-channel camera. Ths soluton explots the dfferent characterstcs of cardac-related blood volume varatons and dstortons, e.g. specular reflectance and moton. Inspred by ths result, our proposed method also uses a mult-channel camera. In ths paper, we present a novel moton-robust non-contact, camera-based method to extract the respratory sgnal near-contnuously n both vsble and dark (nfrared) lghtng condtons by explotng the respratory-nduced skn colour varatons present n the dfferent channels of the camera. Furthermore, we explot the spatal redundancy of the camera to obtan a good qualty rppg sgnal. From ths rppg waveform we extract the respratory-nduced baselne modulaton to obtan the respratory rate. Compared to earler methods, the length of tme wndows are sgnfcantly reduced and no assumptons on the perodcty of the respratory sgnal are made, whch makes t possble to detect rregular breathng patterns and even (central) apneustc events. Furthermore, the proposed method s robust to subject movements not related to respraton. 2. Materals and Methods In ths secton, we wll frst summarze the earler work on rppg and underlyng elementary physology and optcs, whch are the foundaton for our proposed method. Hereafter, we present the processng framework, the protocol and setup used for the creaton of our dataset, a descrpton of the benchmark algorthm and evaluaton metrcs, and fnally the mplementaton detals.
PPG RIFV RIIV RIAV Fg. 1: Respraton modulates the PPG sgnal n three ways; 1) RIFV s a synchronzaton of heart rate wth respratory rate, 2) RIIV s a change n the baselne sgnal due to ntrathoracc pressure varaton, and 3) RIAV s a change n pulse strength caused by a decrease n cardac output. 2.1. Background The beatng of the heart causes pressure varatons n the arteres as the heart pumps blood aganst the resstance of the vascular bed. Snce the arteres are elastc, ther dameter changes n sync wth the pressure varatons. These dameter changes occur even n the smaller vessels of the skn, where the blood volume varatons cause a changng absorpton of the lght. Photoplethysmography (PPG) uses ths prncple for the optcal measurement of blood volume varatons by capturng the reflected or transmtted lght from/through the llumnated skn, resultng n a PPG waveform. Respraton modulates ths PPG waveform n three ways [4], whch s vsualzed n Fg. 1: Respratory nduced frequency varaton (RIFV) - A perodc change n pulse rate that s caused by an autonomc nervous system response. The heart rate synchronzes wth the respratory cycle (RSA). Respratory nduced ntensty varaton (RIIV) - A change n the baselne sgnal that s caused by a varaton of perfuson due to ntra-thoracc pressure varatons. Respratory nduced ampltude varaton (RIAV) - A change n pulse strength that s caused by a decrease n cardac output due to reduced ventrcular fllng durng nspraton. Respraton, much lke the contracton of the heart, also causes blood-pressure varatons, as the varyng pressure n the chest-abdomnal area affects the pressure n the large blood vessels. Where the pulse causes volume varatons manly n the arteres, the respraton also affects the pressure, and consequently the volume n the vens. Arteres and vens have dfferent mechancal propertes. Under low pressure, vens are 1- tmes more complant than arteres [19]. Vessel complance (C) s defned as the ablty of a blood vessel to dstend and ncrease n volume ( V) wth ncreasng transmural pressure ( P): C = V P. (1) Transmural pressure s the dfference n pressure between two sdes of a vessel wall ( P = P nsde P outsde ). Wth small changes n pressure, the crculatng blood nsde the vens
experences large volume changes compared to the arteres because of the dfference n complance. The effects of pressure changes and venous return caused by breathng have been studed, but contradctng observatons have been found. Ths s probably due to the hgh complexty of the underlyng prncple whch s not fully understood yet. The volume of the thoracc cavty ncreases durng nspraton, and therefore ntra-thoracc pressure decreases, causng an ncrease n long volume forcng ar nto the lungs. For venous return to the heart, two large vens are present whch delver deoxygenated blood to the rght atrum of the heart. The nferor vena cava (IVC) returns blood from all body regons below the daphragm, whereas the superor vena cava (SVC) transports the venous blood from the upper part of the body to the heart. The ncrease n ntra-abdomnal and/or ntra-thoracc pressure, dependng on the type of breathng, durng nspraton causes a partal collapse of the venae cavae. Ths partal collapse ether leads to an ncrease or a decrease n venous return, dependng on the pressure gradent. Although t s relevant to understand the underlyng prncples, wthout knowng whether the venous return ncreases or decreases durng nspraton, useful analyss can stll be performed snce we are nterested n dfferences of the observed ampltude rather than the sgn of these changes. Current moton-tolerant rppg methods for pulse extracton requre a mult-spectral camera, e.g. RGB, whch captures blood volume varatons at dfferent wavelengths [12,, 21]. The pulsatle ampltude of the PPG waveform as functon of wavelength s smulated by Hulsbusch, who explaned that the relatve PPG-ampltude s determned by the contrast between the blood and the blood-free tssue [22]. The absolute PPG-ampltude as functon of wavelength was measured by Corral et al. usng a spectrometer [23] and a whte, halogen, llumnaton. Ths absolute PPG spectrum PPG(w), dsplayed n Fg. 2(a), s related to the relatve PPG-curve, RPPG(w), va the emsson spectrum of the halogen llumnaton, I(w), and the skn-reflecton spectrum, ρ s (w): PPG(w) = ρ s (w)i(w)rppg(w) (2) Normalzed Ampltude 1.9.8.7.6.5.4.3.2 (A) APPG (measured) RPPG (derved) Molar Extncton Coeffcent (cm-1/m) (B) 1 6 1 5 1 4 1 3 HbO 2 Hb.1 4 5 6 7 8 9 Wavelength (nm) 1 2 4 5 6 7 8 9 1 Wavelength (nm) Fg. 2: a) The measured absolute PPG spectrum of Corral [23] and the derved relatve PPG spectrum, scaled to 1 for ther peak locatons. b) The absorpton spectrum oxyhemoglobn (HbO 2 ) and hemoglobn (Hb) [24]. Snce venous blood has a dfferent rato of HbO 2 and Hb compared to arteral blood and these chromophores have dfferent absorpton spectra, also the venous and arteral blood have a dfferent absorpton spectrum. These curves are smulated/measured for arteral blood wth normal blood oxygenaton levels. However, because of ts lower oxygenaton level and hereby dfferent rato of oxygenated and deoxygenated hemoglobn, venous blood has a slghtly dfferent absorpton spectrum compared to arteral blood. From Fg. 2(b) t can be observed that because of the dfferent rato of Hb and HbO 2 and the dfference spectra of both chromophores, venous blood has a dfferent absorpton spectrum compared to arteral blood. In vsble lght, [4-7] nm, ths dfference s manly n red, 6 λ 7 nm. As can be observed from the same fgure, n near-nfrared (NIR),
λ > 7 nm, the absorpton spectra of Hb and HbO 2 also dffer, resultng n dfferent absorpton spectra for venous and arteral blood, wth the excepton of the sosbestc" pont around 85 nm. It has been shown n [21] that the man PPG-contrbuton n the red colour channel of a vdeo camera comes from the wavelength nterval between 5 and 6 nm. An mportant consequence of the above reasonng s that the lnear combnaton of the normalzed colour channels that provdes the pulse sgnal wth the best sgnal-to-nose rato (SNR), s approxmately the same lnear combnaton that would also provde the respratory sgnal wth the best SNR. In the followng secton, we shall elaborate ths method to obtan a robust respratory sgnal from an rppg-camera. 2.2. Processng Framework An overvew of our proposed processng framework s vsualzed n Fg. 3. In the next subsectons we wll provde a detaled descrpton of each processng step. 1. Trackng 2. Processng 3. Scalng 1 1 Pulse rate (PR) 2 Weghts 2 Weghts Multwavelength Input N Calculaton Selecton N Gan Respratory Sgnal Fg. 3: Overvew of the proposed framework for robust respraton detecton from remote PPG. 1) The manually ntalzed boundng-box ndcatng the face s tracked over tme and dvded nto equally-szed subregons. 2) The weghts for each (sub)regon are calculated. From ths collecton of weghts, the best are selected based on the SNR values of the pulse sgnals. 3) The extracted respratory sgnal s scaled based on the rato of respratory and pulse energes. 2.2.1. Trackng The frst stage of the framework s the trackng stage, where the movements of the selected Regon-of-Interest (ROI), ndcated wth the boundng box, are beng tracked. For ths task, the feature-based Kanade-Lucas-Tomas (KLT) tracker s employed [25] because of ts accuracy, smplcty and lmted assumptons made about the underlyng mage. Feature ponts (ndcated wth whte crosses) are calculated usng the mnmum egenvalue algorthm [26]. The geometrc transformaton of the feature ponts between two consecutve frames s calculated and appled to the boundng box. Ths boundng box s subsequently down-sampled nto equally-szed subregons. For each frame, the spatal average of both the pxels wthn the ROI and each subregon are calculated, whch enables to dscard dstorted, unrelable, subregons as wll be dscussed later. The temporally normalzed pxel dfferences between two adjacent frames for (sub)regon are defned as: C t t+1 = C t+1 C t = R t t+1 Ḡ t t+1 B t t+1 = R t+1 R t+1 (x+dx,y+dy) R t (x,y) (x+dx,y+dy)+r t (x,y) G t+1 (x+dx,y+dy) G t (x,y) G t+1 (x+dx,y+dy)+g t (x,y) B t+1 (x+dx,y+dy) B t (x,y) B t+1 (x+dx,y+dy)+b t (x,y), (3)
where d = (dx, dy) s the spatal dsplacement between t and t + 1. These moton-compensated, normalzed pxel dfferences are concatenated nto a collecton of traces, C n,, each correspondng to a (sub)regon, and subsequently ntegrated over the wndow-length l, whch s the nput for our method: C t t+l n, = R t t+l Ḡ t t+l B t t+l l 1 = l 1 = = l 1 = R t+ t++1 Ḡ t+ t++1 B t+ t++1. (4) In our system we selected the face as the ROI for both physologcal and practcal reasons. The practcal reason to use the face s that t s one of the few human body parts whch s typcally not covered by clothes and therefore skn s drectly exposed to the camera to measure blood volume varatons. One physologcal reason to use the face s that blood volume varatons are well-measurable from ths anatomc locaton. In ther study, Tur et al. [27] revealed a collecton of regons (fnger, palm, face, ear) for whch cutaneous perfuson s much hgher than all other locatons. Another physologcal reason s that the RIIVs are well-present n photoplethysmographc sgnals from the face because the dstance to the heart s small compared to the locatons at the extremtes whch are typcally also not covered, e.g. the hand. Nlsson [28] measured the respratory energes n the PPG sgnals from multple sensors attached to body parts ncludng hand and forehead, and found that the respratory energy on the forehead s approxmately a factor of sx larger compared to that on the fnger. For the trackng of the face we decded to manually ntalze the ROI and not to use a face detector, e.g. the commonly used Vola-Jones detector, snce these are typcally traned for full vew frontal uprght faces, whereas our dataset also contans faces whch do not meet ths crteron. 2.2.2. Processng After obtanng the moton-compensated, normalzed pxel dfferences, we am to fnd the optmal lnear combnaton to construct the cardac pulse sgnal, and hereafter, the respratory sgnal. The processng stage conssts of two operatons: 1) weghts calculaton, and 2) weghts selecton. 1) Weghts Calculaton The cardac pulse sgnal S can be wrtten as a lnear combnaton of the temporally normalzed colour channels C n : S = WC n, (5) where the weghts W can be determned by blnd source separaton technques (BSS) [12, 29]. However, a heurstc selecton crteron, e.g. based on the perodcty of the pulse-sgnal, s requred to select the component correspondng to pulse or respraton. Two current state-of-the-art rppg algorthms whch do not requre ths selecton crteron are CHROM [] and PBV [21]. We wll evaluate both methods to calculate the weghts. Essentally, the weghts are calculated for the fltered traces of normalzed pxel dfferences whch nclude the range of pulsatle frequences. The pulse rate of a healthy adult s n the range 4-24 BPM and breathng rates are typcally n the range 1-4 breaths/mn. More detals on the selecton of these flter parameters for adults and how these have been selected for neonates can be found n Secton 2.7. In the contnuaton of the method descrpton we wll use the flter parameters for adults. A flter wth a pass-band of 4-24 BPM s desgned and appled to the normalzed pxel dfferences, leadng to C fp. The weghts are calculated and appled to C fp, leadng to a frst sgnal,
S 1, whch contans the cardac pulse sgnal: S 1 = WC fp. Subsequently, these weghts are appled to dfferently fltered pxel dfferences, C fr, whch nclude only respratory frequences. The resultng sgnal, S 2, contans the respratory sgnal: S 2 = WC fr. We wll now brefly dscuss the two methods used to calculate the weghts. CHROM method The chromnance-based method (CHROM) uses colour dfference sgnals, X s and Y s, n whch the specular reflecton component s elmnated, assumng a standardzed skn-colour vector n RGB-space, [.77,.51,.38], enablng whte-balancng of the camera. Its weghts result as: wth W CHRO = 1 [2 α, 2α 4, α]. (6) 6α 2 α + α = σ( X s ) σ( Y s ), wth X s = [+.77,.51, ]C n and Y s = [+.77, +.51,.77]C n (7) and where the operator σ corresponds to the standard devaton. For applcatons n NIR, the assumpton of the standardzed skn-colour vector does not hold and has to be modfed, resultng n dfferent parameters for X s and Y s. The color of the lght s not very mportant, and the spectrum also does not have to be contnuous, e.g. fluorescent lamps are allowed. For more detals on the CHROM algorthm, we refer to []. PBV method Compared to CHROM, the PBV method does not make assumptons on the dstortons or skn-colour, but suppresses all varatons not algned wth the sgnature of the blood volume pulse,.e. the normalzed rato of pulse ampltudes n the dfferent colour channels, compled n P bv. Its weghts are determned as: W PBV = k P bv Q 1, wth Q = C n C T n, (8) where Q s the covarance matrx and k the gan. Based on earler work n vsble lght [21] and NIR [3], the values for P bv are selected as: P RGB bv = [.33,.78,.53] and P 675,8,84 bv = [.29,.74,.61]. The parameters of P bv are, among others, dependent on the camera senstvty and the llumnaton condtons. If the expermental condtons change compared to our expermental settngs, re-calbraton s requred, partcularly when narrow-spectrum lght-sources are appled. However, small varatons n setup, e.g. usng a camera wth dfferent senstvty specfcatons, wll not have a large mpact on the performance. For more detals on the determnaton of P bv we refer to [21]. 2) Weghts Selecton From the collecton of weghts from each (sub)regon, the best weghts need to be selected, whch are subsequently appled to the normalzed dfferences of the entre ROI, whch nclude only respratory frequences, C fr. Ths s acheved by selectng the weghts whch provde the pulse sgnal wth the hghest SNR. These weghts suppress dstortons best and are consequently best capable for the extracton of the respratory sgnal to suppress dstortons n ths frequency band. In order to calculate the SNR of each sub(regon), the correct pulse rate s requred. Snce not all sgnals wll have a clear energy peak n ts spectrum, a robust estmaton of pulse rate s requred, whch wll be employed for the calculaton of SNR values of all (sub)regons. From the collecton of pulse traces, an average pulse trace s constructed by calculatng the α-trmmed mean, where α s set to.7 based on expermental evaluaton. Prncpal component analyss (PCA) s performed on the perodc pulse traces to obtan the egenvectors, whch are ranked n
terms of varance. The egenvector (among the top 5 egenvectors) that has the best correlaton wth the mean pulse trace s selected to be the pulse sgnal after correctng the arbtrary sgn of the egenvector as: P t t+l selected = Pt t+l P t t+l egen, Pt t+l mean egen, Pt t+l mean Pt t+l egen, (9) where Pegen t t+l and Pt t+l mean ndcate the egenvector and mean pulse trace respectvely, <, > corresponds to the nner product (correlaton) between two vectors, and. denotes the absolute value operator. Pulse rate s subsequently determned by selectng the peak n the spectrum of the selected egenvector. The SNR s defned as: ( 24 f =4 SNR = 1 log (U t( f )S f ( f )) 2 ) 1 24 f =4 (1 U, (1) t( f )S f ( f )) 2 where f s the frequency n beats per mnute (BPM), S f = F (S) and (U t ( f ) s a bnary template wndow centered around the pulse rate peak and ts harmoncs wth a predefned margn. 2.2.3. Scalng In elmnate the nfluence of the momentary strength of the PPG-sgnal on the ampltude of the respratory sgnal, a gan factor, k, s computed by the rato between the energes n the respratory frequency band, e.g. 1-4 breaths/mn, and the energy of the pulse sgnal. It s far to assume that when the pulse ampltude doubles, also the respratory ampltude doubles. Hence, by usng the relatve ampltude of respratory energy versus pulse energy, one gets rd of the varatons n pulsatlty over tme: 4 f =1 S( f ) k = PR+margn f =PR margn S( f ), wth S f = F (S), (11) where pulse rate (PR) s determned by a peak-detector n the frequency doman and the margn value s determned based on the length of the Fourer wndow. After scalng, the partally overlappng tme-ntervals are glued together wth an overlap-add procedure smlar to [], by usng Hannng wndowng on ndvdual ntervals. 2.3. Expermental Setup For the recordng of vdeo sequences n vsble lght and nfrared, two separate expermental setups are used. In both setups, partcpants n the experments are asked to follow a partcular breathng pattern vsualzed on a screen n front of the partcpant. The vdeo sequences n vsble lght are recorded wth a global shutter RGB CCD camera (type USB UI-223SE-C of IDS) and stored n an uncompressed data format, at a frame rate of frames-per-second (fps), wth a resoluton of 768 576 pxels and wth 8 bts depth. Recordngs are made n a room wth stable lght condtons. Partcpants wear a fnger sensor (pulse-oxmeter), whch data s synchronzed wth the vdeo frames. To nclude both the face and chest-regon, the camera s placed at a dstance of 1 meter. An llustraton of the expermental setup s vsualzed n Fg. 4. The expermental setup used for recordngs n nfrared s smlar to the setup n an earler study [3]. Three monochrome cameras (type F46B of Alled Vson) wth 25 mm lenses capture at a frame rate of 15 fps, wth a resoluton of 64 48 pxels and wth 8 bts depth. Optcal flters of 675, 8 and 84nm are mounted to the cameras. The data s transferred to an acquston PC over FreWre, where t s stored uncompressed. An llumnaton unt consstng of ncandescent lght bulbs s placed n front of the partcpant.
1m Camera Laptop Illumnaton Fg. 4: Overvew of the expermental setup used for the creaton of the dataset. 2.4. Dataset The performance of our proposed method s evaluated on two dfferent datasets: A) guded breathng of healthy adults n a laboratory settng, and B) spontaneous breathng of neonates n an ntensve care envronment. The study was approved by the Internal Commttee Bomedcal Experments of Phlps Research, and the nformed consent has been obtaned for each adult subject. In addton, the medcal ethcal research commttee at Maxma Medcal Center (MMC) approved the neonatal study and nformed parental consents were obtaned pror to data acquston. Guded breathng scenaros enable to smulate challengng scenaros over the entre range of breathng rates, whch may not be present durng spontaneous breathng. In real lfe, durng spontaneous breathng, both RR and respratory effort are varyng contnuously. To evaluate the performance for these scenaros, a dataset consstng of vdeos recorded n a NICU s created. 2.4.1. Guded Breathng For the guded breathng scenaro s, recordngs are made of three (n vsble lght), and one (n nfrared), healthy, Caucasan adult males, whch are n sttng poston. The duraton of each recordng s 1 or 15 seconds, dependng on the scenaro. The partcpants are asked to follow the breathng patterns dsplayed on a screen n front of them and to keep ther head steady for all non-moton scenaros. Except for the shallow breathng scenaro, partcpants are asked to breath wth normal ar volumes. The benchmark dataset conssts of 32 recordngs wth a total duraton of 6 mnutes and contans 192 breaths n total. An overvew of the breathng patterns s provded n Fg. 5. (A,B,C) Constant respratory rate 1 seconds recordngs of 12 ( slow ), ( normal ) and 35 ( fast ) breaths/mn. (D) Lnearly ncreasng respratory rate 1 seconds recordng, wth a RR startng from 1 breaths/mn, whch lnearly ncreases to 4 breaths/mn.
(A) (B) (C) (D) (E) (F) Fg. 5: Overvew of breathng patterns for guded breathng scenaros n both vsble lght and nfrared. (E) Rapdly changng respratory rate 1 seconds recordng, a constant RR of 15, wth 3 events where the RR shortly ncreases to 35 breaths/mn for 1 seconds. (B) Central apneustc event 15 seconds recordng, a constant RR of, where the partcpant s asked to hold breath after 6 seconds for as long a possble, after whch the breathng pattern s followed agan. (B) Moton 15 seconds recordng at a constant RR of, wth head movements uncorrelated wth respraton after 3 seconds. Subjects were asked to move ther head quas-perodcally wth frequences non-equal to the constant breathng frequency to assure that the extracted respratory sgnal s not nduced by moton. The type of moton s translatonal, where we nstructed the subjects to move ther head wthn the sght of the camera, resultng n average ROI dsplacements of approxmately pxels. (F) Shallow breathng 1 seconds recordng wth shallow breathng at a constant rate of breaths/mn. 2.4.2. Spontaneous breathng Non-contact respraton montorng s partcularly mportant and nterestng for neonatal montorng, because of the senstve skn of newborns. Therefore, a dataset contanng vdeos (n dfferent scenes) from 2 neonates n supne poston s bult for evaluaton and demonstraton wth a total number of 588 breaths. The vdeos are recorded n the Neonatal Intensve Care Unt (NICU) of Máxma Medcal Center (MMC, Endhoven, The Netherlands) under vsble lght condtons, where the neonates are recorded from 4 dfferent camera vews: (1) zoomed top-vew of head and a part of the chest, (2) wde range top-vew, (3) zoomed sde-vew of head and a part of the chest, and (4) wde range sde-vew. The medcal ethcal research commttee at MMC approved the study and nformed parental consents were obtaned pror to data acquston. 1 2.5. Benchmark Algorthm To benchmark our proposed method, we compare the output wth the state-of-the-art usng the method of Karlen et al. [4] as our benchmark algorthm. Ths method s the best-performng PPG-based respraton detecton algorthm from the 314 respratory algorthms evaluated by Charlton et al. [3]. Karlen et al. use a smart fuson of all three respratory modulatons. Snce our method only uses the baselne modulaton of the PPG sgnal, we compare the performance of our method wth two versons of the benchmark algorthm: 1) the complete framework ncludng all 1The medcal ethcal research commttee at Máxma Medcal Center has revewed the research proposal and consdered that the rules lad down n the Medcal Research nvolvng Human Subjects Act (also known by ts Dutch abbrevaton WMO), do not apply to ths research proposal
three features (BM F ), and 2) wth only the RIIV as feature (BM I ). Because the algorthm can only be appled on a sngle waveform, we selected the wavelength wth the hghest SNR; the green colour channel n vsble lght and 84 nm n nfrared. 2.6. Evaluaton To assess the performance of the proposed method, the nstantaneous respratory rates are calculated. Here a peak detecton algorthm s appled on the nterpolated respratory sgnal and compared wth the ground truth: the breathng pattern for guded breathng scenaros, and the ECG-derved respratory sgnal for the NICU recordngs. We preferred the guded breathng pattern over a possble respraton belt, as the latter suffers too much from the subject moton to be a relable reference. The Pearson correlaton coeffcent r, slope of lnear ft B, mean absolute error (MAE), root-mean-square-error (RMSE) and standard devaton (σ) are calculated. Furthermore, correlaton plots are ncluded and Bland-Altman analyss s performed to test for magntude bas n respratory rate dfferences. Here, the 95% lmts of agreement were determned by [-1.96σ,+1.96σ]. 2.7. Parameter settngs To determne the flter parameters for bandpass flters of our method, the typcal ranges n breathng and pulse rate for neonates and adults have to be taken nto account. Both the respratory and pulse rate are approxmately a factor of three hgher for neonates compared to adults [31]. The normal restng breathng rate of adults s n the range [12-] breaths/mn, whereas for neonates normal rates are n the range [3-6] breaths/mn. Breathng rates lower than 1 breaths/mn. may occur for adults under extreme restng condtons, however, other addtonal rhythms,.e. Traube-Herng-Mayer (THM) rhythms and vasomoton, nterfere wth the respratory sgnal. THM rhythms, whch are caused by the sympathetc control of the tones of the vascular tree, have a fxed rate of about 6 mn -1 ; whereas vasomoton rhythms, whch are slow rhythmc changes n the dameter of the small blood vessels of the mcrocrculatory bed, have a frequency of 4-9 mn -1 [32]. The normal pulse rate of adults s n the range [6-1] BPM, whereas for neonates t s n the range [1-16] BPM. These values are for healthy subject under normal, restng condtons. To verfy the performance of our method for realstc scenaros where rates are outsde these ranges, we chose our flter parameters for C fp+fr as [4-24] and [1-24] BPM for adults and neonates, respectvely. For C fr, we set the flter parameters to [1-4] BPM for adults and [25-1] BPM for neonates. The number of subregons s set to 3 for all evaluated recordngs and the length of the tme-wndows s set to 8 seconds wth 8% overlap. Both parameters are set heurstcally. 2.8. Implementaton The proposed algorthm s mplemented n Matlab (The Mathworks, Inc.) and executed on a laptop wth a Intel Core 5 2.6 GHz processor and 8-GB RAM. A rectangular ROI ndcatng the face regon s ntalzed manually n the frst frame of the sequence. 3. Results 3.1. Guded breathng An overvew of the results for the dfferent guded breathng scenaros s vsualzed n Fg. 6 for both CHROM and PBV. The evaluaton results for each scenaro, both n vsble lght and nfrared, are summarzed n Table 1, together wth the results of the benchmarkng algorthm and the overall results. Correlaton plots and Bland-Altman analyss are dsplayed n Fg. 7.
(a) Constant RR: breaths/mn (b) Lnearly ncreasng RR: 1-4 breaths/mn (c) Rapdly changng RR, 15-35 breaths/mn. (d) Holdng breath, breaths/mn (e) Wth head moton, breaths/mn (f) Shallow breathng, breaths/mn Fg. 6: Results from the dfferent breathng scenaros n vsble lght condtons for both proposed methods. The spectrograms are calculated wth a wndow-sze of 8 seconds. 3 1 5 CHROM PBV 1 Mean + 1.96*SD Mean - 1.96*SD -1 CHROM PBV 4 Estmate - Reference Estmate - Reference Estmate (BPM) CHROM PBV 4 Estmate (BPM) 5 3 1 1 3 Reference (BPM) (a) 4 5 15 25 3 35 (Estmate + Reference) / 2 (b) 4 CHROM PBV Mean + 1.96*SD -1 Mean - 1.96*SD - 1 1 3 Reference (BPM) (c) 4 5 15 25 3 35 4 (Estmate + Reference) / 2 (d) Fg. 7: Correlaton and Bland-Altman plots for guded breathng: (a-b) n vsble lght, (c-d) n nfrared. The black lnes n the correlaton plots ndcate the lnear relatonshp y=x.
Vsble lght Infrared Breathng scenaro Method MAE RMSE σ MAE RMSE σ CHROM 1.63 3.27 3. 1.86 3.7 2.78 Constant 12 BPM PBV 1.93 3.61 3.42 1.38 2.89 2.81 BM I 2.11 3.95 3.62 2.25 3.93 3.77 BM F 2.49 4.17 3.98 2.93 4.36 4.1 CHROM 1.15 1.61 1.61 1.45 1.96 1.98 Constant BPM PBV 1.34 1.95 1.94.93 1.22 1.23 BM I 1.94 2.6 2.49 1.73 2.4 2.16 BM F 2.35 3.6 2.93 2.11 2.87 2.71 CHROM 1.33 2.29 2.27 1.88 3.4 3.32 Constant 35 BPM PBV 1.49 2.41 2.41 1.16 1.98 1.95 BM I 2.45 3.9 3.63 2.18 2.43 2.1 BM F 2.91 3.27 3.5 2.74 3.62 3.48 CHROM 1.19 1.54 1.43 2.33 3.85 3.85 Lnearly ncreasng 1-4 BPM PBV 1.62 1.93 1.58 1.73 1.6.97 BM I 2.79 3.82 3.53 2.95 4.3 3.86 BM F 3.13 4.28 4.1 3.48 4.8 4.51 CHROM 1.55 2.45 2.43 1.3 1.73 1.75 3 events 15/35 BPM PBV 1.68 2.58 2.52 1.6 1.38 1.39 BM I 2.84 4.17 3.8 2.59 3.73 3.46 BM F 3.23 4.61 4.35 3.29 4.83 4.58 CHROM 2.1 3.79 3.59 4.19 7.37 7.3 Moton BPM PBV 2.27 4.1 3.72 2.6 3.52 3.44 BM I 7.31 9.4 8.45 7.9 8.14 7.3 BM F 9.9 11.4 1.7 8.91 11.1 1.4 CHROM 3.22 5.82 5.71l 3.5 6.41 6.42 Shallow breathng BPM PBV 3.63 6.49 6.3 2.73 4.29 4. BM I 6.13 8.97 8.46 6.9 9.14 8.5 BM F 8.6 1.5 1.1 8.27 1.8 1.3 Vsble lght Infrared Breathng scenaro Method r B MAE RMSE σ r B MAE RMSE σ Overall CHROM.962.971 1.74 2.67 2.61.936.961 2.27 3.96 3.88 PBV.957.973 1.99 2.92 2.85.983.985 1.55 2.57 2.49 BM I.875.94 3.65 5.21 4.85.88.914 3.55 4.83 4.46 BM F.862.897 4.47 5.9 5.59.859.893 4.53 6.5 5.73 Table 1: Results from the guded breathng scenaros n both vsble and nfrared lghtng condtons. 3.2. Spontaneous breathng The evaluaton results of the neonatal dataset are dsplayed n Table 2. Fgure 8 provdes a vsual comparson between both proposed rppg methods and the reference sgnal derved from ECG. Correlaton plots and Bland-Altman analyss are dsplayed n Fg. 9. 4. Dscusson 4.1. Guded breathng Constant breathng rate: At low respratory rates, the breathng pattern, and consequently the respratory sgnal, s typcally not snusodal, whch s an mplct assumpton of our algorthm. Ths may lead to selectng of local peaks and consequently erroneous nstantaneous breathng rates. However, these low rates can stll be clearly observed n the
Front-vew Top-vew Close Wde Fg. 8: Both proposed rppg-based methods show large agreement wth the reference ECG-derved respratory sgnal. The snapshots on the rght llustrate the four dfferent vewpont and dstances ncluded n the NICU dataset. Camera dstance Method r B MAE RMSE σ Close-vew Wde-vew Overall CHROM.886.898 4.67 9.24 9.13 PBV.871.868 5.31 9.53 9.49 BM I.819.831 7.36 11.5 11.2 BM F.783.88 9.9 14.7 14.3 CHROM.869.89 4.76 9.49 9.29 PBV.855.861 5.46 9.72 9.77 BM I.814.827 7.43 11.7 11.3 BM F.776.813 9.17 14.8 14.4 CHROM.872.892 4.72 9.35 9.22 PBV.862.864 5.39 9.66 9.67 BM I.817.829 7.4 11.6 11.3 BM F.78.811 9.13 14.8 14.4 Table 2: Results from spontaneous breathng scenaros recorded n a NICU under vsble lght condtons. frequency spectra obtaned on longer tme-wndows, as can be observed from Fg. 6(b). For normal breathng rates, hgh breath-to-breath accuracy s acheved, as can be observed from Fg. 7. At hgh breathng rates a modest reducton n breath-to-breath accuracy s observed. Ths can be explaned by the decreased ampltude of RIIVs at ncreased breathng rates [33]. Responsveness: A large beneft of RIIV-based methods compared to RIFV-based methods s that short tme-wndows can be used to extract the respratory sgnal, allowng to track rapd changes n breathng rate near-contnuously wthout the requrement to make any assumptons on the perodcty of the respratory sgnal. Fgures 6(b) and 6(c) demonstrate ths responsveness of our method. Apnea: Durng central apnea, when breathng movements dsappear, the ntra-thoracc pressure varatons that drve the crculatory varaton synchronously wth respraton
Estmate (BPM) 1 8 6 4 CHROM PBV Estmate - Reference 6 4 - -4 CHROM PBV Mean + 1.96*SD Mean - 1.96*SD 4 6 8 1 Reference (BPM) -6 4 6 8 1 1 (Estmate + Reference) / 2 Fg. 9: Correlaton and Bland-Altman plots for spontaneous breathng. The black lnes n the correlaton plots ndcate the lnear relatonshp y=x. are gone. Although the rhythmc RIIV sgnal dsappears, there are fluctuatons that can nterfere wth regstraton of respraton. Irregular fluctuatons of low ampltude can be observed n the perpheral venous pressure and RIIV sgnals durng apnea as can be observed n Fg. 6(d) [34]. Durng obstructve apnea, when the arway s partly obstructed, an ncrease n the force of respratory movements takes place and RIIV s more promnent. Ths confrms the hypothess that, smlar to moton-based methods, respratory effort s detected wth RIIV-based methods, and not the actual arflow or related modulatons. Moton: Movement of the head causes ntensty varatons wthn the tracked ROI, whch are typcally stronger than the RIIVs. Most earler proposed methods ncludng the benchmark do not am to suppress these dstortons, but nstead detect these to exclude them from the measurement. As can be observed from Fg. 6(e), when applyng the weghts on the normalzed colour dfferences n the respratory frequency band, t s possble to elmnate non-respratory related ntensty varatons and accurately extract the respratory sgnal. Shallow breathng: Shallow breathng causes reduced ntra-thoracc/abdomnal pressure varatons compared to normal breathng, and consequently also reduced RIIVs. As a consequence, t can be observed from Fg. 6(f) that the breath-to-breath accuracy s decreased compared to Fg. 6(a). However, the constant breathng rate of mn -1 s stll clearly detectable n the spectrogram and our method stll clearly outperforms the benchmark. 4.2. Spontaneous breathng Comparson n performance between the two camera dstances shows that dstance has no large nfluence. Only a modest decrease n performance s observed for ncreased dstance,.9 BPM, whch s lkely caused by the reduced number of pxels per subregon, leadng to a lower SNR. Compared to the guded breathng scenaros, the performance s somewhat worse. Ths was expected for a number of reasons: 1) RIIVs are reduced n supne poston compared to sttng poston [35], 2) the varablty n both breathng rate and ampltude are much hgher, and 3) the ground-truth used for evaluaton s derved from ECG whch suffers from moton-artfacts tself, as can be observed from Fg. 8. Consequently, not all peaks can be accurately dentfed. It should also be mentoned that we evaluate our algorthm based on ndvdual breath-ntervals wthout post-processng, whereas many other algorthm, e.g. the RR ox algorthm of Addson et al. [36], use long tme-wndows and may addtonally average over
prevous estmates to arrve at a breathng rate. Ths may yeld a smaller error, but rapd changes n breathng rate cannot be tracked and the potentally dangerous events of apnea may not be detected. Overall, we consder the results promsng for a future transton to remote montorng of respraton. However, all results for guded breathng are obtaned on young, healthy subjects. Snce the neonatal study s lmted, extensve valdaton on subjects sufferng possble health ssues s needed to proof clncal valdty of our method. A potental mprovement can be dentfed n a hybrd system, where our rppg-based method s combned wth a moton-based method. Ths may elmnate the major lmtatons of the ndvdual approaches; the requrement of vsble skn for our method, and the requrement of vsble chest/abdomen for moton-based methods. 5. Concluson We have demonstrated that respraton can be detected wth a camera n both vsble and dark lghtng condtons by usng the close smlarty between pulse and respraton nduced colour varatons of the skn. The proposed method has been thoroughly evaluated usng 52 challengng vdeos contanng both seated adults performng guded breathng, and neonates n a supne poston breathng spontaneously. For guded breathng, all typcal respratory rates n the range of 1-4 breaths/mn can be detected, even when the changes n rate are rapd and transent. Furthermore, respraton can be detected durng head movements and also potentally dangerous breath-holdng events, e.g. durng central apnea, can be clearly dentfed. Overall, the mean absolute error for guded breathng scenaros s 1.74 BPM and 2.27 BPM n vsble lght and nfrared, respectvely, compared to 3.55 BPM and 3.65 BPM for the best-performng benchmark algorthm. For spontaneous breathng, the breathng rate can be detected wth an error of 4.72 BPM compared to 7.4 BPM for the benchmark. Ths result demonstrates an mportant step towards a non-contact alternatve for the commonly used contact sensor(s), whch may lead to trauma of the fragle skn of neonates. Our proposed method showed a large mprovement to earler PPG-based methods for respraton montorng. Fundng Ths research was performed wthn the framework of the IMPULS-II program for the Data Scence Flagshp project. Acknowledgement The authors would lke to thank Dr. I. Krenko, Dr. W. Verkrujsse and P. Bruns of Phlps Research for ther valuable contrbutons to ths paper. Furthermore, we would lke to thank all volunteers of Endhoven Unversty of Technology who partcpated n the experments.