http://en.wikipedia.org/wiki/darpa_grand_challenge - The Urban Challenge requires designers to build vehicles able to obey all traffic laws while they detect and avoid other robots on the course. This is a particular challenge for vehicle software, as vehicles must make intelligent decisions in real time based on the actions of other vehicles. Basic Rules Vehicle must be stock or have a documented safety record. Vehicle must obey the California state driving laws. Vehicle must be entirely autonomous, using only GPS and the information it detects with its sensors. DARPA will provide the route network 24 hours before the race starts. Vehicles will complete the route by driving between specified checkpoints. DARPA will provide a file detailing the order the checkpoints must be driven to 5 minutes before the race start. Vehicles may stop and stare for at most 10 seconds. Vehicles must operate in rain and fog, with GPS blocked. Vehicles must avoid collision with vehicles and other objects such as carts, bicycles, traffic barrels, and objects in the environment such as utility poles. Vehicles must be able to operate in parking areas and perform U-turns as required by the situation. http://www.darpa.mil/grandchallenge/docs/urb_challenge_announce. pdf - Grand Challenge 2005 proved that autonomous ground vehicles can travel significant distances and reach their destination, just as you or I would drive from one city to the next, said DARPA Director Dr. Tony Tether. After the success of this event, we believe the robotics community is ready to tackle vehicle operation inside city limits. - The November 2007 Urban Challenge will feature autonomous ground vehicles executing simulated military supply missions safely and effectively while in a mock city environment, merging into moving traffic, navigating traffic circles, negotiating busy intersections and avoiding obstacles. http://en.wikipedia.org/wiki/stanley_%28vehicle%29 - The sensors used by Stanley include five LIDAR laser-ranging units, a pair of 24 GHz RADAR units, a stereo camera, and a single-lens camera. Position sensing was provided by a GPS receiver, a GPS compass, an inertial guidance system, and wheel odometry information provided by the Touareg's internal CAN bus. Computing was provided by six low-power 1.6 GHz Intel Pentium M based computers in the boot [trunk], running various incarnations of the Linux operating system, which actuate a drive-by-wire system developed by the ERL.

- Stanley was characterized by a machine learning based approach to obstacle detection. Data from the LIDARs was fused with images from the vision system to perform more distant look-ahead. If a path of drivable terrain could not be detected for at least 40 meters in front of the vehicle, speed was decreased and the LIDARs used to locate a safe passage. - Also, Stanley drove by recording how a human drove the car through the desert, then assigning an accuracy value to each bit of data generated by its slew of sensors. After this modification was made, it began to speed at 45 mph down roads which were crisscrossed by shadows of trees. Before it started assigning accuracy values to its data, it would have shied away from the road - it would have been perceived as being crisscrossed with ditches, not shadows. http://www.darpa.mil/grandchallenge/overview.asp - The DARPA Urban Challenge is an autonomous vehicle research and development program with the goal of developing technology that will keep warfighters off the battlefield and out of harm s way. The Urban Challenge features autonomous ground vehicles maneuvering in a mock city environment, executing simulated military supply missions while merging into moving traffic, navigating traffic circles, negotiating busy intersections, and avoiding obstacles. http://www.darpa.mil/grandchallenge/docs/urban_challenge_rules_1 21106.pdf - The Urban Challenge course tests the vehicle s ability to operate safely and effectively with other vehicles in and around an urban environment. The course will be nominally 60 miles in total distance, with a time objective of 6 hours. The road surface will range in quality from new pavement to potholes and broken pavement. Sections of dirt roads with low berms may also be encountered. The vehicle may negotiate sharp curbs, downed branches, traffic barrels, drains, hydrants, rocks, brush, construction equipment, concrete safety rails, power line poles, and other stationary items likely to be found in an urban environment. Vehicles will obey traffic laws as they negotiate traffic circles, intersections, and merge with moving traffic. Traffic on the route may be provided by manned vehicles, tele-operated vehicles, and other autonomous vehicles. Static vehicles may also be parked or stopped along the route. Roads may be blocked by DARPA during the course of the event. Trees and buildings along the route may interfere with GPS. Along some road segments there may be significant distances between waypoints, requiring vehicles to use their sensors to stay in the travel lane. - To complete the Urban Challenge, a vehicle must negotiate all hazards, re-plan for alternate routes, and avoid static and dynamic obstacles while completing a complex, multi-part mission at speeds of up to 30 mph, resulting in an average speed of at least 10 mph. 1.7. Objectives

The objective of this program is safe and correct autonomous driving capability in traffic at 20 mph. To do this, vehicles will demonstrate the following capabilities: Complete a mission defined by an ordered series of checkpoints in a complex route network. The vehicle will have 5 minutes to process a mission description before attempting the course. Interpret static lane markings (e.g., white and yellow lines) provided with the route network definition file and behave in accordance with applicable traffic laws and conventions. DARPA s intent is for the RNDF lane boundary descriptors to match the physical lane markings on the ground. DARPA cannot ensure that this will be the case in all areas, and as such the RNDF shall take precedence over the physical ground markings in conflicting areas. (Need to recognize lane markers for the purpose of staying in the lane, but not interpreting their meaning solid vs. dashed, white vs. yellow.) Exhibit context-dependent speed control to ensure safe operation, including adherence to speed limits. (Speed limits are provided in route files, not obtained from signage.) Exhibit safe-following behavior when approaching other vehicles from behind in a traffic lane. This includes maintaining a safe-following distance. Exhibit safe check-and-go behavior when pulling around a stopped vehicle, pulling out of parking spot, moving through intersections, and in situations where collision is possible. Stay on the road and in a legal and appropriate travel lane while en route, including around sharp turns, through intersections, and while passing. The route network definition file will specify the GPS coordinates of the stop signs. The RNDF specifies the location of stop lines on the ground. On paved areas, each stop line will be represented by a painted stop line on the pavement. Physical stop signs, however, may or may not be present at the stop line locations. (Remember that GPS may be blocked in some areas, though, so the vehicle shouldn t depend entirely on the GPS coordinates to stop in the right place good application for the vision system, recognizing the stop lines.) Navigate safely in areas where GPS signals are partially or entirely blocked. Follow paved and unpaved roads and stay in lane with very sparse or low accuracy GPS waypoints. (Another good vision application recognizing the lane boundaries, whether they be curbs, painted lines, or raised edges/ditches for dirt roads.) Change lanes safely when legal and appropriate, such as when passing a vehicle or entering an opposing traffic lane to pass a stopped vehicle. Vehicles must not pass other vehicles queued at an intersection. (Vehicle recognition is key possible vision application.) Merge safely with traffic moving in one or more lanes after stopping at an intersection. (More vehicle regonition need good static and dynamic recognition with position and velocity estimation.)

Pull across one lane of moving traffic to merge with moving traffic in the opposing lane. Stop safely within 1 meter of the stop line at a stop sign intersection and proceed without excessive delay (less than 10 seconds) according to intersection precedence rules. (Need to recognize the stop line and whether or not there are other cars waiting around.) Exhibit proper queue behavior at an intersection, including stopping at a safe distance from other vehicles and stop-and-go procession to the stop line without excessive delay. Navigate toward a destination in a large, open area where minimal or no GPS points are provided, as in loading dock areas or parking lots. These areas may contain fixed obstacles such as parked vehicles and moving obstacles including other vehicles. (General obstacle recognition stopped and moving in the road and in open spaces like parking lots.) Safely pull into and back out of a specified parking space in a parking lot. (Parking space boundary recognition.) Safely execute one or more three-point turning maneuvers to effect a U-turn. (Need to recognize curbs/gutters/other road boundaries.) Dynamically re-plan and execute the route to a destination if the primary route is blocked or impassable. The following behaviors or capabilities are outside the scope of this program: Recognition of external traffic signals such as traffic lights and stop signs, through the use of sensors. Recognition of pedestrians and pedestrian avoidance. Behaviors necessary for highway driving such as high speed passing or high speed merge at an onramp. Speed limits for the Urban Challenge will be 30 mph or less. Driving in difficult off-road terrain is outside the scope of the program. Off-road navigation in an unpaved area, travel along roads with potholes, and travel along a dirt road are within scope. (Potholes, ditches, and other negative obstacle recognition and avoidance.) - DARPA will provide the Route Network Definition File (RNDF) that includes all accessible road segments and provides information such as waypoints, stop sign locations, lane widths, checkpoint locations, and parking spot locations. The route network has no implied start or end points. Road blockages will not be indicated in the RNDF.

- DARPA will also provide a Mission Data File (MDF) containing the checkpoints that must be reached and minimum and maximum speed limits for the road segments. MDFs will be provided by DARPA for all Urban Challenge test events. - Vehicle Separation: Vehicle maintains a minimum standoff of 1-meter on the sides and rear from all obstacles and vehicles in all areas. In safety areas, vehicle maintains a minimum forward vehicle separation equal to 2-meters. In travel areas, vehicle maintains a minimum forward vehicle separation of one vehicle length for every 10 miles-per-hour of speed, one vehicle length minimum. (Need accurate environment modeling including location, size, and motion of obstacles/vehicles.) - Merge: Vehicle always merges into moving traffic when there is a delay of 10 seconds or more before the arrival of the next traffic-vehicle. Vehicle may pull into a gap of less than 10 seconds when conditions permit. (Need to be able to estimate vehicle positions and speeds to figure out gap time lengths.) - Emergency braking: Vehicle comes to a complete and safe stop to avoid collision when a moving obstacle suddenly moves into the travel lane. - Defensive driving: Vehicle performs defensive driving maneuvers to avoid impending head-on collision and maintains the required forward vehicle separation distance. - The event will be conducted during daylight hours. (But, technical FAQ specifies that the weather conditions may be naturally or artificially poor Precipitation, wind, dust, smoke, or fog is possible from natural or artificial sources. ) - The Route Network Definition File (RNDF) specifies accessible road segments and provides information such as waypoints, stop sign locations, lane widths, checkpoint locations, and parking spot locations. The route network has no implied start or end point. In addition to road segments, the RNDF specifies free-travel zones with a defined perimeter, for which no interior waypoints are provided. Zones are used to represent parking lots and open areas with moving or stationary obstacles. Road blockages created by DARPA will not be indicated in the RNDF. - The Mission Data File (MDF) specifies the sequence of checkpoints to be visited by the vehicle for a given mission. The MDF specifies minimum and maximum speed limits for all segments in the RNDF; speed limits may vary with each mission over the same route network. http://www.darpa.mil/grandchallenge05/techpapers/stanford.pdf - Stanford s 2005 Grand Challenge Technical Paper - LIDAR and RADAR are used exclusively for the range finding and obstacle detection. LIDAR is far more accurate, but range-limited. It is used to figure out what is the best, flattest driving surface within its view. Then, the camera data is used with this range information to determine what the best driving surface looks like in the image and develop an adaptive model for the road surface. This model is then used with the camera s longer range, higher density information to predict where the road is further ahead than the LIDAR can see.

- Much of the drivability modeling and maximum velocity limits were created using machine learning techniques by letting Stanley learn from human driving examples. http://www.darpa.mil/grandchallenge/docs/technical_evaluation_crit eria_061206.pdf A.3. Checkpoints - Vehicle front bumper passes over each checkpoint in the MDF in the correct lane or spot and in the correct sequence. - Checkpoints will generally be in the middle of a travel lane or parking spot. The checkpoint location will be marked on the ground or pavement as a circle of approximately 15 cm in diameter for viewing by the officials. This mark may or may not be visible to the vehicle. To get credit for visiting a waypoint, the vehicle front bumper must pass over the checkpoint as marked. Checkpoints will not be placed next to walls or barriers such that they are difficult to access. Vehicles must be located in the appropriate orientation to visit a checkpoint. This is illustrated in Figure A.3. (When checkpoints are visible to the vehicle, locating them with a camera is likely to be a much more accurate method of passing over them than by relying on GPS coordinates alone. This is especially true considering that GPS outages are likely. I m not sure how the marks could be visible to the officials and not the vehicle, but there would have to be a contingency plan if somehow the camera doesn t pick them up even though the navigation system says that the vehicle is in the right place.) A.4. Stay in lane - Vehicle remains entirely in travel lane at all times except when performing a legal traffic maneuver such as a left turn or maneuvering to avoid an obstacle. - All wheels of vehicle should remain in travel lane, at least one meter from the center line. Although the travel path may not be marked at an intersection, the appropriate vehicle path is understood to be a smooth arc that connects the lane centers (Figure A.4). Vehicles may leave the travel lane under exceptional circumstances, such as passing a static obstacle or vehicle or during maneuvers such as left turns or U-turns. (Need lane recognition for both the road segment you are currently driving on and perpendicular/angled road segments at intersections so that an appropriate path can be planned for turns.) A.7. Collisions - Vehicle acts to avoid collisions and near-collisions at all times, as judged by DARPA. - Effective navigation while avoiding collisions is the primary guiding principle for the Urban Challenge. This principle encompasses a range of behaviors involving both careful navigation and lane keeping, but also defensive driving. Vehicles must continuously monitor the path ahead for vehicles and obstacles and avoid collisions, and should not assume that other vehicle behavior is entirely predictable. (By one means or another, obstacles must be detected and avoided even if they are moving in an unexpected fashion.)

A.8. Stop line - Vehicle stops so front bumper is within 1 meter of the center of the stop line at intersection. - This is depicted in Figure A.8. Precision driving is a prerequisite for safety in an urban environment, and vehicles must be capable of precise vehicle control. - In all cases, the vehicle should err on the side of caution- a vehicle that stops short of the line and inches forward is preferred to a vehicle that overshoots. Vehicles must never stop in a location that protrudes into or partially blocks an intersection. (GPS coordinates for the stop locations will be provided, but the required accuracy in finding the stop line pretty much dictates that a vision system be used to find the stop line. This would be a perfect computer vision application.) B.2. Intersection precedence - Vehicle respects precedence order at intersections and does not proceed out of turn. B.3. Minimum following distance - When following a moving traffic-vehicle, autonomous vehicle maintains the required forward vehicle separation distance. B.4. Queueing - Vehicle exhibits correct stop-and-go queueing behavior in a line of stopped vehicles, always maintaining a minimum spacing equal to the forward vehicle separation distance and a maximum spacing of two vehicle lengths. (All of these behaviors require accurate vehicle identification and tracking to correctly determine which ones are part of moving traffic and/or queues and which ones may be stopped and in need of passing.) C.3. Parking lot - Vehicle exhibits correct parking lot behavior and demonstrates the ability to pull forward into and reverse out of a specified parking spot. (Have to get the vehicle into the spot accurate enough to cover two checkpoints this will be accomplished if it is lined up with the lines of the parking spot. No guarantees are given that there will be lines, but they are implied by figure C.3b.) C.6. GPS outage - Vehicle does not exhibit excess delay or leave travel lane due to intermittent loss of navigation signals such as GPS. - Operation with degraded GPS due to foliage or buildings is a requirement. An inertial navigation unit or other technique should be used to enable vehicle to continue to travel safely within lane, possibly at a reduced speed until GPS is re-acquired. A vehicle should pull to the side to re-acquire GPS, however, rather than drift into oncoming traffic. (Lane and road boundary recognition is very important to accomplish this.)

D.6. Zones - Vehicle navigates zones safely and effectively in the presence of moving traffic and other obstacles. - This is consistent with C.2 and C.3, with the addition of moving traffic. Moving traffic in zones creates a particular challenge because of the absence of established traffic lanes. Vehicles must have the ability to detect moving traffic and obstacles from both behind and in front to avoid collisions. (Will need a rear-facing camera and/or LIDAR) Questions - Can distance/size/volume/speed of motion information be derived from a single camera image, or would it require stereo vision? The products on www.mobileye.com advertise being able to derive range information from a single camera, but no details of how that is done are offered. I suspect it involves making some strong assumptions and using some sophisticated models. - Would stereo vision with two cameras even be enough or would more than that be required to get reasonably-accurate distance, height, surface slope, and obstacle estimates? - How do you go about differentiating between general obstacles, vehicles/moving obstacles, and the road surface? - How do you go about recognizing lane markers, curbs, other road boundaries, parking space lines, and the stop lines in paved sections? Notes and Thoughts - Stop line recognition would probably be more effective if combined with lane recognition. That is because a stop line is essentially a line painted perpendicular to the lane of travel which crosses only the lanes in the direction of travel. If the lane is curving, a line perpendicular to it could be recognized much earlier on by knowing the path that the lane is following. - Need to look for any additional details about the shape, size, and color of stop lines. I haven t seen anything very specific yet. If nothing is really specified, it would probably be safe to assume that they will conform to the normal white lines in standard used. - Vehicle recognition one commonality between many different types of vehicles is the presence of wheels with tires. If we could successfully identify different styles of wheels from any viewing angle, then you can assume that a vehicle is attached to them and you know the whole thing could start moving at any moment. Furthermore, if you can figure out how the wheels are pointed, you have a pretty good idea of what directions the vehicle can move at that moment (they can t go perpendicular to the orientation of the wheels).

- Problems: tires are black and don t show up very distinctly in photos, vehicles may have the wheels covered to some extent (like the Honda Insight), the tires are hard to see and may be obscured by the body or mud flaps when looking straight at the front or back of a car. - Possible solution for darkness issue: brightness, contrast, and/or colormap adjustments. - Another common feature for any vehicle (or other object for that matter) is that it will cast a shadow on the road. This will depend on the time of day and lighting conditions, but may help find obstacles. Something important for early morning and late afternoon will be differentiating between shadows, obstacles, and potholes/ditches.