Smart Cars for Safe Driving Prof. Dr. Dariu M. Gavrila Environment Perception Group Research and Advanced Engineering XXXII Jornadas de Automática, Sevilla, 9-9-2011
We originally thought Machine Intelligence would look like 1956 "Forbidden Planet" Robby the Robot (Flickr) 2
Then more recently, some suggested it would be more like 2004 irobot 1983-2009 Terminator 3
when in fact, Machine Intelligence is already with us, and has a familiar embodiement 4
Driver Assistance in the current Mercedes Benz E-Class Nightview Plus Adaptive High Beam Lane Keeping Attention Blind Spot Speed Limit PRE-SAFE 5
Driver Assistance Technology is rapidly expanding the capabilities of modern vehicles. One breakthrough development over the past few years is the emergence of driver assistance systems. Use of sensor systems which continuously monitor vehicle surroundings and interior, provide information to the driver, and even perform vehicle control. Help drivers operate their vehicles in a safe, comfortable, and energyefficient manner. Enables market differentiation for vehicle manufacturers 6
What got us here: Sensors Radars Cameras Laser Scanners Better and cheaper. 7
What got us here: Computational Power CPU performance over time 10 6 *1,78/a MFlops in my vehciles 10 500 400 300 200 GFLOPS/MIPS Processing Power over Time 3DIP G80 IQ2 G70 Virtex 4 GPU (NVidia) G92 ASIC Virtex 5 FPGA (Xilinx) Prognosis 2030: optimistic (1.78/a): 100 PFlops pessimistic (1.41/a): 1 PFlops 100 NV40 Tyzx Standford engine Spartan3 Core2Duo CPU (Intel) Transputer/x86 P4 1990 2000 2010 time (Still) exponentially increasing. 8
Next Challenge: Active Pedestrian Safety Pedestrian are the most vulnerable traffic participants. Children are particularly at risk. Driver inattention and/or bad visibility are important accident causes. Worldwide fatalities of pedestrians, bicyclists, and motorcyclists (2006) Source: Bosch Accident Research 9
Why is it difficult? Large variation in pedestrian appearance (viewpoint, pose, clothes). Dynamic and cluttered backgrounds. Pedestrians can exhibit highly irregular motion. Real-time processing required. Stringent performance requirements (especially for emergency maneuvres). 10
Pedestrian System Architecture Obstacle Detection (Stereo, (Stereo, Flow, Flow, Radar) Radar) Object Object Classification Tracking Path Path Prediction & Risk Risk Assessment Driver Driver Warning / Vehicle Vehicle Control Control The benefit of object classification: improved detection reliability vs. obstacle detection only better path prediction: taking advantage of prior knowledge of object class motion and additional object class-specific cues allows object class-specific driver warning and vehicle control strategies D. M. Gavrila and S. Munder. Multi-Cue Pedestrian Detection and Tracking from a Moving Vehicle. IJCV 73(1), 2007. S. Munder, C. Schnörr and D.M. Gavrila. Pedestrian Detection and Tracking Using a Mixture of View-Based Shape-Texture Models. IEEE Trans. on Intelligent Transportation Systems, vol.9, nr.2, pp.333-343, 2008. C. Keller, T. Dang, A. Joos, C. Rabe, H. Fritz, and D.M. Gavrila. Active Pedestrian Safety by Automatic Braking and Evasive Steering, IEEE Trans. on Intelligent Transportation Systems, 2011 11
3D Position and Motion for Every Pixel (Scene Flow) stereo time t optical flow stereo optical flow Joint Optimization time t-1 I ( x, y, t 1) I ( x + d, y, t 1) l I ( x + u, y + v, t) l r = = = Motion l I ( x + u, y + v, t) I ( x + u + d + dd, y + v, t) I ( x + u + d + dd, y + v, t) r r A. Wedel, C.Rabe, T. Vaudrey, T. Brox, U.Franke, D.Cremers. Efficient Dense Scene Flow from Sparse or Dense Stereo Data. ECCV 2008. 12
Scene Flow 13
Pedestrian Classification Experimental Studies What features? E.g. Chamfer, Haar wavelets, HOG, and Local Receptive Field What pattern classifier? E.g. SVM, Neural Networks How to combine pattern classifiers? E.g. Cascading, Parallel (Sum/Max/Mixture) How to deal with occlusion? Haar wavelets + AdaBoost cascade [Viola & Jones, 2005] HOG features + linear SVM [Dalal & Triggs, 2005] Local receptive fields + NN [Wöhler & Anlauf, 1999] 14
Daimler Pedestrian Benchmark Data Sets 1. 2. 3. >130.000 samples (intensity, dense stereo, dense flow), 48x96 pixel Training: 14400 peds. / 15000 non-peds. Test: 9600 peds. / 10000 non-peds. All 18x36 pixel. Training: 15660 peds. / 6744 non-ped images Test: 21790 images with 259 ped. trajectories Available for download (Google) 1. Mono Pedestrian Classification S. Munder and D. M. Gavrila. An Experimental Study on Pedestrian Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.11, pp. 1863-1868, 2006. 2. Multi-Modal / Occluded Pedestrian Classification M. Enzweiler, A. Eigenstetter, B. Schiele and D. M. Gavrila. Multi-Cue Pedestrian Classification with Partial Occlusion Handling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010. 3. Mono/Stereo Pedestrian Detection M. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp.2179-2195, 2009. C. Keller, M. Enzweiler, and D. M. Gavrila. A New Benchmark for Stereo-based Pedestrian Detection. Proc. of the IEEE Intelligent Vehicles Symposium, Baden-Baden, Germany, 2011. 15
An intriguing question How many image examples are needed to learn pedestrian appearance? ROC performance improves with enlarged training set. No saturation effects (even) for N = 12.800 In fact, doubling training size matters more than selecting the best feature-classifier combination. Manually labeling humans in images is time-consuming and tedious! Can we do better? 16
Generating Virtual Pedestrians Shape variation Texture variation M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008. 17
Mixed Generative-Discriminative Classification Framework Enlarged training set significantly improved classification performance (30% less false positives at equal true positive rate) Meanwhile, current pedestrian classifier on-board vehicle uses more than 1.5 million samples ( real and virtual ) M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008. 18
Pedestrian Detection - Daytime (Videoclip) 19
Pedestrian Detection Nighttime (Videoclip) 20
Now with dense stereo 21
Pedestrian Recognition Performance (Historical Perspective) We need to get somewhere here Source: EU Final Review WATCH-OVER Correctly recognized pedestrians 100% 85% 65% 50% 40% EU WATCH-OVER (2008) 50 km/h EU SAVE-U (2005) 40 km/h EU PROTECTOR (2003) 30 km/h 0 10 100 600 1000 Number of falsely recognized pedestrian trajectories per hour N.B. # False alarms per hour << # Falsely recognized trajectories per hour 22
Pedestrian Path Prediction by Trajectory Matching longitudinal feature position (m) dimension longitudinal position (m) traj. prediction lateral position lateral position (m) aligned snippet distribution system trajectory main mode State-of-the-art path prediction: Kalman filtering based on position detected bounding box. Problem: first-order model does not capture non-linearities well during sudden motion changes. Our approach Use higher order model; match learned trajectory snippets (segment of fixed length). QRLCS (Hermes et al. IV 09) metric computes similarity after alignment (translation/rotation). Use of additional motion features. Path prediction by extrapolation of matched trajectory snippets (non-param. regression). Use of particle filter representation. C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize. 23
Path Prediction C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize. 24
Action Classification (Crossing or not) Predicting the correct pedestrian s action with accuracy 80% is reached: 570 ms before a possible standstill by the human (cyan). 180 ms before a possible standstill by the proposed system (black). only after the possible standstill by the IMM-KF (pink). Motion features help. 1 Frame 45ms C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize. 25
World Premiere (2009): Automatic Evasion on Pedestrians 26
Automated Test Driving (Videoclip) Source: Daimler Testing Department 27
Understand What Is What Localize and classify objects in the environment Vision Background Street Moving vehicle Pedestrian Sky Source: U. Franke 28
Driver Monitoring Head / Face / Gaze Tracking Mindlab Head / Face tracking using stereo vision and Active Appearance Models Driver intention estimation based on head motion, gaze, and vehicle trajectory Online EEG analysis of driver mental state (work load, fatigue) Use to objectively evaluate driver assistance systems (Attention, IHC) 29
Automation Systems: Gradually Getting There Autonomous 2nd Assisted 1st Feet off Assisted 2nd Hands off Autonomous 1st Eyes off Moderate takeover times Body out Ability to drive empty All on Traditional driving Today s ACC Short takeover times Not certifiable today Source: R. G. Herrtwich 30
Final Remarks Driver assistance is experiencing a breakthrough: a first major deployment of machine intelligence technology (sensing, reasoning, acting in physical environment). Computer vision and machine learning play a central role. Trend towards increased actuation of safety systems Driver information driver warning soft vehicle actuation / driver-initiated hard vehicle actuation automatic hard vehicle actuation Environment Perception is still the bottleneck. Need to recognize a wider set of traffic objects classes with better classification performance localize objects more accurately in 3D (perform segmentation and classification jointly). handle adverse visibility conditions Future systems will fuse data from lots of sensors and build a precise 3Drepresentation of the 360 car surrounding. The progress in environment perception, driver monitoring, communication as well as in precise 3D map data will bring us close to our vision of accident free driving. 31
The best is yet to come! Questions? 32