T_AP00966 Active Pedestrian Protection System, Project Review 2010. 10. 29 Ho Gi Jung The School of Electronic and Electrical Engineering Yonsei University, South Korea
Project Overview Objectives To develop active pedestrian protection system (APPS), detecting pedestrian using sensor fusion-based, assessing risk of vehicle-pedestrian collision, and actuating countermeasures to avoid the collision. Duration 1 st Phase (Prototype development): Nov. 2006 Oct. 2009 (36 months) 2 nd Phase (Product development): Nov. 2009 Oct. 2011 (24 months) Fund Co-funded by participating companies and government, MKE (Ministry of Knowledge and Economy) For the 1 st phase, total budget is 4.7 million USD.
Team Organization MANDO Ho Gi Jung Project management System design Vision-based pedestrian classification Sensor fusion of NIR vision and range sensor Sensor fusion of FIR stereo and visible vision Risk assessment Actuation LG Innotek SL Yonsei Univ. VisLab TOF camera module TOF camera-based pedestrian recognition NIR camera module NIR headlamp Radar array-based pedestrian recognition System design Sensor fusion of NIR vision and range sensor
Scenario-Driven Method Almighty classifier is replaced with a combination of strict experts. False negative False positive Alberto Broggi, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, and Ho Gi Jung, A New Approach to Urban Pedestrian Detection for Automatic Braking, IEEE Transactions on Intelligent Transportation Systems, Vol. 10, Issue 4, Dec. 2009, pp. 594-605. Alberto Broggi, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, Ho Gi Jung, Localization and Analysis of Critical Areas in Urban Scenarios, 2008 IEEE Intelligent Vehicle Symposium, Eindhoven University of Technology, Eindhoven, The Netherlands, June 4-6, 2008, pp. 1074-1079. Alberto Broggi, Pietro Cerri, Luca Gatti, Paolo Grisleri, Ho Gi Jung, JunHee Lee, Scenario-Driven Search for Pedestrians Aimed at Triggering Non-Reversible Systems, 2009 IEEE Intelligent Vehicle Symposium, 2009 IEEE Intelligent Vehicle Symposium, June 3-5, 2009, Xi an, Shaanxi, China, pp. 285-291.
Critical Area-Centered Pedestrian Recognition The top ranked situation is when a pedestrian popped up from the behind of vehicle parked along a road-side. Critical area We assume that pedestrians in front of ego-vehicle without occlusion would be easily detected by the driver. Examples of critical area. The second row shows the critical area of situations of the first row.
Comparison of Four Sensing Methods Four sensing methods were investigated and compared: 1) Sensor fusion of NIR vision and range sensor (scanning laser radar) 2) TOF (Time Of Flight) camera-based 3) Range sensor array-based (mm-wave radar) 4) Sensor fusion of FIR stereo and visible vision Three Criteria 1) Critical area localization performance 2) Latency of popping up pedestrian detection 3) System cost and possibility of domestic series production Sensor fusion of NIR vision and range sensor was selected for the next phase.
Sensor Fusion of NIR Vision and Range Sensor Vision-based pedestrian classifier - Haar-feature-based Adaboost - Ad-hoc-features-based Adaboost Pietro Cerri, Luca Gatti, Luca Mazzei, Fabio Pigoni, Ho Gi Jung, Day and Night Pedestrian Detection Using Cascade AdaBoost System, the 13th IEEE International Conference on Intelligent Transportation Systems, Maderia Island, Portugal, 19-22 Sep. 2010, accepted on 2 Jul. 2010.
Sensor Fusion of NIR Vision and Range Sensor HDRC Camera Module with NIR Sensitivity NIR Lamp with horizontally wider FOV (a) General headlamp (b) Newly developed headlamp Bi-functional module NIR lamp
Sensor Fusion of NIR Vision and Range Sensor Detected critical area Result of vision-based pedestrian verification
TOF Camera-based Although prototype was made, this approach was rejected because of several reasons: 1) Light power in the near area does not satisfy US FDA eye-safe regulation. 2) Required power for the modulating light source is too high. 3) Too large size and cooling method.
Radar Sensor Array-based Although we developed radar signal-based object classification and tracking, this approach was rejected because it can not address popping up pedestrian scenarios. 1) Vehicle corner localization is instable because radar is too sensitive to the vehicle s pose and contains relatively large variation. 2) Filtering algorithm embedded into the radar prevents rapid detection of popping up pedestrian and it causes significant latency. Seongkeun Park, Jae Pil Hwang, Euntai Kim, Heejin Lee, and Ho Gi Jung, A neural network approach to target classification for active safety system using microwave radar, Expert Systems with Applications, Vol. 37, Issue 3, 15 March 2010, pp. 2340-2346.
Sensor Fusion of FIR Stereo and Visible Vision Although we recognized this approach was robust to day and night situations, it was rejected because of system price and domestic component sourcing. Input stereo images Segmentation result Stereo matching and object detection HOG-based feature extraction SVM-based classification
Additional Results about Pedestrian Classifier Gabor Filter Bank (GFB)-based Feature Sub-window Splitting In order to extract local characteristics of pedestrian images 36x18 image 18x9 nine overlapped sub-images 24 Gabor filters (6 sections in orientation, 4 sections in scale) 3 statistical values of filtered result: mean, variance, skewness Feature vector dimension: 9x24x3=648 Ho Gi Jung, Jaihie Kim, Constructing a pedestrian recognition system with a public open database, without the necessity of retraining: an experimental study, Pattern Analysis and Applications, Vol. 13, No.2, May 2010, pp. 223-233.
Additional Results about Pedestrian Classifier Radial Basis Function (RBF)-based SVM and Genetic Algoritm (GA)-based Optimization Candidate Image Sub-window Splitting Kernel parameter and regulation parameter are critical for classification performance. Filtering Extraction with Gabor Filter Bank Parameter Optimization is needed! GA is robust to non-linear and discontinuous optimization. Gene is composed of two parameters. Fitness function is defined by cross-validation. When Joachims performance estimator is used instead of cross-validation, all learning database can be used for the training. Consequently, it leads to improvement of recognition performance. Optimization of SVM Learning Parameters SVM Learning with Training Data n n n 1 maximize W( α) = α αα y y e 2 i i j i j i i j = 1 = 1 = 1 subject to C α 0, α y = 0 i i i i= 0 n SVM Execution with Cross-Validation Data Ns f( x ) = α y e i= 1 i i 2 si x 2 2σ P = P( non ped) + P( ped non) error GA-based Minimization of P error With respect to C and σ 2 xi xj 2 2σ
Additional Results about Pedestrian Classifier DCX pedestrian database - 5 datasets 3 for training (1 for learning and 2 for cross validation), 2 for final test. - One dataset: 4,800 pedestrian images (800 persons), 5,000 non-pedestrian images. Better performance and lower complexity 1 0.9 0.8 0.7 Detection Rate 0.6 0.5 0.4 0.3 0.2 0.1 S. Munder and D. M. Gavrila, An Experimental Study on Pedestrian Classification, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 28, No. 11, pp. 1863-1868, Nov. 2006. 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Positive Rate Comparison of ROCs. The dotted line is ROC of LRF quadratic SVM shown in Fig. 5(d) of [1] and the solid line is ROC of GFB RBF SVM.
Additional Results about Pedestrian Classifier Four useful facts were found: 1) GFB-RBF SVM is competent in pedestrian recognition. 2) Performance estimator can replace cross-validation. In this case, dataset used for cross-validation can be used for learning, and increases resultant performance. 3) If sampling method and pre-processing are common, a pedestrian recognition system constructed with a database can be used for actual application without re-training with the newly acquired database. In other words, publicly open database could be used for a general pedestrian recognition system. 4) A posteriori probability-based post-processing enhances recognition rate while losing a little in the false positive rate.
Risk Assessment Considering the possibility of collision avoidance using braking and steering, 3 risk levels were defined.
Risk Levels and Counter Measures Typical 5-step collision scenario and countermeasures were investigated.
Counter Measures To reduce danger of false actuation, pedestrian protection measures are activated according to a hierarchical strategy.
Lessons from PCDN Study Feasibility of pre-crash dipping nose (PCDN), developed for vehicle-vehicle collision, was investigated. PCDN showed severity increase in vehicle-pedestrian collision situation. Although PCDN was developed for vehicle-vehicle side crashes, there is a possibility that the range sensor for crash detection fail to distinguish a group of pedestrians from a sidefaced vehicle. vehicle-vehicle collision countermeasure also needs pedestrian recognition. (a) Without PCDN and AHS (b) With PCDN and AHS Ho Gi Jung, Byung Moon Kwak, Jeong Soo Shim, Pal Joo Yoon, Jaihie Kim, Pre-Crash Dipping Nose (PCDN) Needs Pedestrian Recognition, IEEE Transactions on Intelligent Transportation Systems, VOL. 9, NO. 4, Dec. 2008, pp. 678-687.
Experimental Results For cooperative development, two test vehicles in each site. SICK LMS 211, NIR sensitive camera. Active braking by MANDO s MGH-40 ESCplus via CAN. Parma University, Italy MANDO, Korea
Experimental Results 10 hours in complex urban scenarios 236km Various situations were included. 24 true positives 1 false positive (2 10-6 false positive/frame) 11 false negative (1 missing, others are alert missing or delayed detection) Fig. 10. Some suddenly appearing pedestrians correctly detected (a) in an underground parking, (b) in the rain, (c) behind a misaligned vehicle, (d) behind a wall, and (e) at night, and (f) a suddenly appearing pedestrian detected as a non-dangerous pedestrian (false negative).
Experimental Results In Italy In Korea
Thank You for Your Attention! E-mail: hgjung@yonsei.ac.kr Homepage: http://web.yonsei.ac.kr/hgjung