Smart Cars for Safe Driving

Size: px

Start display at page:

Download "Smart Cars for Safe Driving"

Cleopatra Shelton
5 years ago
Views:

1 Smart Cars for Safe Driving Prof. Dr. Dariu M. Gavrila Environment Perception Group Research and Advanced Engineering XXXII Jornadas de Automática, Sevilla,

2 We originally thought Machine Intelligence would look like 1956 "Forbidden Planet" Robby the Robot (Flickr) 2

3 Then more recently, some suggested it would be more like 2004 irobot Terminator 3

4 when in fact, Machine Intelligence is already with us, and has a familiar embodiement 4

5 Driver Assistance in the current Mercedes Benz E-Class Nightview Plus Adaptive High Beam Lane Keeping Attention Blind Spot Speed Limit PRE-SAFE 5

6 Driver Assistance Technology is rapidly expanding the capabilities of modern vehicles. One breakthrough development over the past few years is the emergence of driver assistance systems. Use of sensor systems which continuously monitor vehicle surroundings and interior, provide information to the driver, and even perform vehicle control. Help drivers operate their vehicles in a safe, comfortable, and energyefficient manner. Enables market differentiation for vehicle manufacturers 6

7 What got us here: Sensors Radars Cameras Laser Scanners Better and cheaper. 7

What got us here: Computational Power CPU performance over time 10 6

Power over Time 3DIP G80 IQ2 G70 Virtex 4 GPU (NVidia) G92 ASIC Virtex 5

8 What got us here: Computational Power CPU performance over time 10 6 *1,78/a MFlops in my vehciles GFLOPS/MIPS Processing Power over Time 3DIP G80 IQ2 G70 Virtex 4 GPU (NVidia) G92 ASIC Virtex 5 FPGA (Xilinx) Prognosis 2030: optimistic (1.78/a): 100 PFlops pessimistic (1.41/a): 1 PFlops 100 NV40 Tyzx Standford engine Spartan3 Core2Duo CPU (Intel) Transputer/x86 P time (Still) exponentially increasing. 8

Next Challenge: Active Pedestrian Safety Pedestrian

9 Next Challenge: Active Pedestrian Safety Pedestrian are the most vulnerable traffic participants. Children are particularly at risk. Driver inattention and/or bad visibility are important accident causes. Worldwide fatalities of pedestrians, bicyclists, and motorcyclists (2006) Source: Bosch Accident Research 9

10 Why is it difficult? Large variation in pedestrian appearance (viewpoint, pose, clothes). Dynamic and cluttered backgrounds. Pedestrians can exhibit highly irregular motion. Real-time processing required. Stringent performance requirements (especially for emergency maneuvres). 10

11 Pedestrian System Architecture Obstacle Detection (Stereo, (Stereo, Flow, Flow, Radar) Radar) Object Object Classification Tracking Path Path Prediction & Risk Risk Assessment Driver Driver Warning / Vehicle Vehicle Control Control The benefit of object classification: improved detection reliability vs. obstacle detection only better path prediction: taking advantage of prior knowledge of object class motion and additional object class-specific cues allows object class-specific driver warning and vehicle control strategies D. M. Gavrila and S. Munder. Multi-Cue Pedestrian Detection and Tracking from a Moving Vehicle. IJCV 73(1), S. Munder, C. Schnörr and D.M. Gavrila. Pedestrian Detection and Tracking Using a Mixture of View-Based Shape-Texture Models. IEEE Trans. on Intelligent Transportation Systems, vol.9, nr.2, pp , C. Keller, T. Dang, A. Joos, C. Rabe, H. Fritz, and D.M. Gavrila. Active Pedestrian Safety by Automatic Braking and Evasive Steering, IEEE Trans. on Intelligent Transportation Systems,

u, y + v, t) I ( x + u + d + dd, y + v, t) I ( x + u + d + dd, y + v, t) r r A. Wedel, C.Rabe, T.

12 3D Position and Motion for Every Pixel (Scene Flow) stereo time t optical flow stereo optical flow Joint Optimization time t-1 I ( x, y, t 1) I ( x + d, y, t 1) l I ( x + u, y + v, t) l r = = = Motion l I ( x + u, y + v, t) I ( x + u + d + dd, y + v, t) I ( x + u + d + dd, y + v, t) r r A. Wedel, C.Rabe, T. Vaudrey, T. Brox, U.Franke, D.Cremers. Efficient Dense Scene Flow from Sparse or Dense Stereo Data. ECCV

13 Scene Flow 13

Pedestrian Classification Experimental Studies What features? E.g.

classifier? E.g. SVM, Neural Networks How to combine pattern classifiers?

Haar wavelets + AdaBoost cascade [Viola & Jones, 2005] HOG features +

14 Pedestrian Classification Experimental Studies What features? E.g. Chamfer, Haar wavelets, HOG, and Local Receptive Field What pattern classifier? E.g. SVM, Neural Networks How to combine pattern classifiers? E.g. Cascading, Parallel (Sum/Max/Mixture) How to deal with occlusion? Haar wavelets + AdaBoost cascade [Viola & Jones, 2005] HOG features + linear SVM [Dalal & Triggs, 2005] Local receptive fields + NN [Wöhler & Anlauf, 1999] 14

Daimler Pedestrian Benchmark Data Sets 1. 2. 3. >130.000 samples (intensity, dense stereo, dense flow), 48x96 pixel Training: 14400 peds. / 15000 non-peds. Test: 9600 peds. / 10000 non-peds.

An Experimental Study on Pedestrian Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.11, pp. 1863-1868, 2006. 2. Multi-Modal / Occluded Pedestrian Classification M.

3. Mono/Stereo Pedestrian Detection M. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp.

15 Daimler Pedestrian Benchmark Data Sets > samples (intensity, dense stereo, dense flow), 48x96 pixel Training: peds. / non-peds. Test: 9600 peds. / non-peds. All 18x36 pixel. Training: peds. / 6744 non-ped images Test: images with 259 ped. trajectories Available for download (Google) 1. Mono Pedestrian Classification S. Munder and D. M. Gavrila. An Experimental Study on Pedestrian Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.11, pp , Multi-Modal / Occluded Pedestrian Classification M. Enzweiler, A. Eigenstetter, B. Schiele and D. M. Gavrila. Multi-Cue Pedestrian Classification with Partial Occlusion Handling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Mono/Stereo Pedestrian Detection M. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp , C. Keller, M. Enzweiler, and D. M. Gavrila. A New Benchmark for Stereo-based Pedestrian Detection. Proc. of the IEEE Intelligent Vehicles Symposium, Baden-Baden, Germany,

An intriguing question How many image examples are needed to learn pedestrian appearance? ROC performance improves with enlarged training set. No saturation effects (even) for N = 12.

16 An intriguing question How many image examples are needed to learn pedestrian appearance? ROC performance improves with enlarged training set. No saturation effects (even) for N = In fact, doubling training size matters more than selecting the best feature-classifier combination. Manually labeling humans in images is time-consuming and tedious! Can we do better? 16

17 Generating Virtual Pedestrians Shape variation Texture variation M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR

pedestrian classifier on-board vehicle uses more than 1.5 million samples ( real and virtual ) M.

18 Mixed Generative-Discriminative Classification Framework Enlarged training set significantly improved classification performance (30% less false positives at equal true positive rate) Meanwhile, current pedestrian classifier on-board vehicle uses more than 1.5 million samples ( real and virtual ) M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR

19 Pedestrian Detection - Daytime (Videoclip) 19

20 Pedestrian Detection Nighttime (Videoclip) 20

21 Now with dense stereo 21

22 Pedestrian Recognition Performance (Historical Perspective) We need to get somewhere here Source: EU Final Review WATCH-OVER Correctly recognized pedestrians 100% 85% 65% 50% 40% EU WATCH-OVER (2008) 50 km/h EU SAVE-U (2005) 40 km/h EU PROTECTOR (2003) 30 km/h Number of falsely recognized pedestrian trajectories per hour N.B. # False alarms per hour << # Falsely recognized trajectories per hour 22

23 Pedestrian Path Prediction by Trajectory Matching longitudinal feature position (m) dimension longitudinal position (m) traj. prediction lateral position lateral position (m) aligned snippet distribution system trajectory main mode State-of-the-art path prediction: Kalman filtering based on position detected bounding box. Problem: first-order model does not capture non-linearities well during sudden motion changes. Our approach Use higher order model; match learned trajectory snippets (segment of fixed length). QRLCS (Hermes et al. IV 09) metric computes similarity after alignment (translation/rotation). Use of additional motion features. Path prediction by extrapolation of matched trajectory snippets (non-param. regression). Use of particle filter representation. C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize. 23

24 Path Prediction C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize. 24

25 Action Classification (Crossing or not) Predicting the correct pedestrian s action with accuracy 80% is reached: 570 ms before a possible standstill by the human (cyan). 180 ms before a possible standstill by the proposed system (black). only after the possible standstill by the IMM-KF (pink). Motion features help. 1 Frame 45ms C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize. 25

26 World Premiere (2009): Automatic Evasion on Pedestrians 26

27 Automated Test Driving (Videoclip) Source: Daimler Testing Department 27

28 Understand What Is What Localize and classify objects in the environment Vision Background Street Moving vehicle Pedestrian Sky Source: U. Franke 28

Driver Monitoring Head / Face / Gaze Tracking Mindlab

Appearance Models Driver intention estimation based on

analysis of driver mental state (work load, fatigue) Use

29 Driver Monitoring Head / Face / Gaze Tracking Mindlab Head / Face tracking using stereo vision and Active Appearance Models Driver intention estimation based on head motion, gaze, and vehicle trajectory Online EEG analysis of driver mental state (work load, fatigue) Use to objectively evaluate driver assistance systems (Attention, IHC) 29

30 Automation Systems: Gradually Getting There Autonomous 2nd Assisted 1st Feet off Assisted 2nd Hands off Autonomous 1st Eyes off Moderate takeover times Body out Ability to drive empty All on Traditional driving Today s ACC Short takeover times Not certifiable today Source: R. G. Herrtwich 30

31 Final Remarks Driver assistance is experiencing a breakthrough: a first major deployment of machine intelligence technology (sensing, reasoning, acting in physical environment). Computer vision and machine learning play a central role. Trend towards increased actuation of safety systems Driver information driver warning soft vehicle actuation / driver-initiated hard vehicle actuation automatic hard vehicle actuation Environment Perception is still the bottleneck. Need to recognize a wider set of traffic objects classes with better classification performance localize objects more accurately in 3D (perform segmentation and classification jointly). handle adverse visibility conditions Future systems will fuse data from lots of sensors and build a precise 3Drepresentation of the 360 car surrounding. The progress in environment perception, driver monitoring, communication as well as in precise 3D map data will bring us close to our vision of accident free driving. 31

32 The best is yet to come! Questions? 32

Active Pedestrian Safety: from Research to Reality

Active Pedestrian Safety: from Research to Reality Dariu M. Gavrila Environment Perception Research and Development Oxford University, 04-10-2013 We originally thought Machine Intelligence would look like