Panacea: An Active Senso Contolle fo the ALVINN Autonomous Diving System Rahul Sukthanka, Dean Pomeleau and Chales Thope Robotics Institute Canegie Mellon Univesity 5 Fobes Ave., Pittsbugh, PA 15213-3891 Abstact Panacea is a modula system which incopoates a steeable senso into an existing neual netwok diving system, ALVINN. A xed camea cannot see the oad when it makes shap bends. Fo a vision system that builds a map of the oad, it is staightfowad to point the camea down the oad; but ALVINN diectly outputs a steeing command without geneating an intemediate oad epesentation. Insight fom the taining scheme used in ALVINN, howeve, povides an intepetation of the steeing command in tems of the oad geomety and appopiate camea pointing stategies. Tests on the Canegie Mellon Navlab II with a steeable camea have shown that the system signicantly impoves ALVINN's pefomance, paticulaly in situations equiing shap tuns and quick esponses. The Panacea active camea contol system illustates a tend in the ALVINN poject away fom teating neual netwoks as simple black box function appoximatos. Instead, the neual netwok's behavio is modeled symbolically and easoned about to impove system pefomance. Two othe examples of model based easoning about netwok pefomance in the ALVINN system ae biey descibed. 1. Intoduction ALVINN (Autonomous Land Vehicle in a Neual Netwok) is a neual netwok based system which has been successful in diving obot vehicles in a vaiety of situations [1, 2]. Howeve, since ALVINN maintains no state infomation about the wold, but pocesses each senso fame individually, it can become confused on shap cuves when the eld of view no longe displays the impotant featues in the scene. A steeable senso allows the peception system to select the desied eld of view to maximize the infomation content of a senso fame [4]. Fo a vision system that builds a map of the oad, it is staightfowad to point the camea in the desied diection, but ALVINN diectly outputs a steeing command, without geneating an intemediate oad epesentation. Panacea intepets this steeing command as a point on the oad and pans the camea in the desied diection. Howeve since ALVINN is tained with a xed senso oientation, the position of the senso duing taining is implicitly encoded in the weights and moving the camea esults in the outputs of the netwok being invalid fo the given conguation. Panacea solves this poblem by post-pocessing the steeing esponse of the neual netwok as a function of the cuent senso conguation. A signicant advantage of this appoach is that existing netwoks can un unde this new system without any modication o etaining. Panacea was implemented on the Canegie Mellon Navlab II and has demonstated impoved pefomance of ALVINN netwoks, paticulaly on oads with shap cuves. 2. ALVINN Achitectue and Taining The ALVINN system's basic achitectue is a thee layeed aticial neual netwok shown in Figue 1. A educed esolution camea image is fed into a 3x32 aay of input units, which ae fully connected to a hidden laye of 4 units. The hidden units ae fully connected to a vecto of 3 output units, and the steeing esponse is given as a Gaussian activation level centeed on the coect steeing cuvatue. ALVINN's neual net is tained \on the y", and the human dive's steeing esponses ae used as the teaching signal. ALVINN is able to lean fom this limited data by aticially expanding its taining set. Each oiginal image is shifted and otated in softwae to ceate 14 additional images in which the vehicle appeas to be situated dieently in elation to the oad (See Figue 2). The taining signal fo each of these new images is calculated by assuming a pue-pusuit [5] model of diving and tansfoming the oiginal steeing esponse accod-
Shap Left Staight Ahead Shap Right 3 Output Units 4 Hidden Units l l 3x32 Senso Input Retina Figue 1: ALVINN diving netwok achitectue. Oiginal Image Figue 3: Illustation of the \pue pusuit" model of steeing. ingly. One of the advantages of using a weak model like pue-pusuit is that it is independent of the diving situation. Figue 3 illustates this model. With the vehicle at position A, the pue pusuit model assumes the goal is to bing the vehicle to the oad cente at the taget point T, a pedetemined distance ahead of the vehicle. Afte tansfoming the image with a hoizontal shift s and otation to make it appea that the vehicle is at point B, the appopiate steeing diection accoding to the pue pusuit model should also bing the vehicle to the taget point T. Mathematically, the fomula to compute the adius of the steeing ac that will take the vehicle fom point BtopointTis = l2 + d 2 2d (1) whee is the steeing adius, l is the lookahead distance and d is the distance fom point T the vehicle would end up at if diven staight ahead fom point B fo distance l. The displacement d can be detemined using the following fomula: d = cos (d p + s + l tan ) (2) Shifted and Rotated Images Figue 2: The single oiginal video image is shifted and otated to ceate multiple taining exemplas in which the vehicle appeas to be at dieent locations elative to the oad. whee d p is the distance fom point T the vehicle would end up if it dove staight ahead fom point A fo the lookahead distance l, s is the hoizontal distance fom point A to B, and is the vehicle otation fom point A to B. The quantity d p can be calculated using the following equation: d p = p, q 2 p, l2 (3) whee p is the adius of the ac the peson was steeing along when the image was taken.
3. Panacea Panacea uses the pue-pusuit diving model to adjust an existing ALVINN netwok's steeing output in esponse to vaiations in senso oientation. Since the model is also used intenally by ALVINN duing taining, the same assumptions ae made in the two modules. When used with a xed senso, both systems poduce identical esponses. ALVINN outputs a steeing esponse which can be symbolically intepeted as a tuning adius, o a desied ac. In the pue-pusuit model, evey such ac maps to a single taget point TP, at the specied look-ahead distance fom the senso. Thus fo a given vehicle pose, the position of the TP should emain invaiant unde changes in senso oientation. In othe wods, the puepusuit model implies that thee is a \coect" TP fo the cuent vehicle pose, which is independent of the senso pan. ALVINN's esponse is in senso coodinates since it implicitly assumes that the camea is pointing diectly ahead. Howeve, since the senso is not in its oiginal oientation, the tuning adius given by ALVINN no longe stees the vehicle towads the taget point. Theefoe we have to compensate fo the change in senso oientation, and geneate the ac which coectly stees the obot towads the TP coesponding to the vehicle's actual position. Panacea thus convets ALVINN's outputs into a taget point epesentation, and geneates the ac (in the cuent vehicle fame) which dives the obot towads the TP. Figue 4 illustates this tansfomation. The equations fo this tansfom ae deived below: p d =, sgn 2, l 2 (4) l = (l, a) cos, d sin + a (5) d = (l, a) sin + d cos (6) = d2 + l 2 2d (7) whee is the steeing adius epoted by ALVINN and is the compensated adius calculated by Panacea, while d and d ae the osets. l is the analog of l, ALVINN's lookahead distance, in the vehicle efeence fame. The steeing adius epoted by Panacea is used to contol the vehicle. To gain a bette undestanding of the equations, a suface plot of the compensation against the input paametes was made. Fo claity, tuning adii wee conveted to cuvatues, and the compensation expessed as the dieence between the input and output cuvatues. Figue 5 displays compensation as a function of input cuvatue and camea pan angle fo two dieent lookahead distances. The gaph on the left coesponds to a l ALVINN efeence fame l -a a Vehicle efeence fame θ l-a d Senso d Taget Point Figue 4: Senso pan compensation using Panacea. typical Navlab II conguation (l = 1 metes, a =3:3 metes). The compensation seems to be independent of the input cuvatue, and vaies popotionally with the camea pan angle ove the values encounteed in pactice. Howeve it is inteesting to note that this is not tue in geneal. The gaph on the ight shows the same suface with an exteme value fo l = 25 metes. Note that the compensation is no longe independent of the input cuvatue. Although the implementation on the Navlab II could have been appoximated using a plana model of the suface, the computational savings would be insignicant since the oiginal equations ae aleady quite simple. Theefoe Panacea computes the pecise compensation using equations 4 to 7. 4. Senso Pointing Panacea also addesses the issue of intelligent senso contol. ALVINN's output, which may be intepeted as a TP on the cente of the oad ahead of the vehicle, can be used to pan the camea in ode to keep the oad in view. The following equation elates the position of the TP to the pan angle: = tan,1 d l, a (8) whee l and d ae dened in Equations 5 and 6 espectively. This allows us to contol the senso diectly fom the output of ou neual netwok, in a manne which is completely consistent with the pue-pusuit model. The
.1.2 1/ -1/ -.2 2.55 1/ -1/ -.55 2 -.2 -.1 theta -.1 -.2 -.1 theta 1/.1-2 1/.1-2.2.2 Figue 5: Cuvatue compensation with lookahead of 1m and 25m espectively. actual implementation is somewhat complicated by contol issues such as oscillations caused by the dynamics of the system. In pactice this was solved by intoducing a damping tem which smoothed the senso's esponse. Thee ae a numbe of advantages associated with contolling the senso based on the netwok's output: By diecting the senso towads the TP, the impotant featues of the scene as peceived by ALVINN ae centeed in the eld of view. Images of this type ae close to those seen duing taining, and theefoe accuacy of the netwok is inceased. Since the senso esponds moe quickly than the obot vehicle, the netwok is able to \look befoe it leaps". Panacea is implemented so that the compensation fo senso displacement and the contol of the senso ae decoupled. Thus ALVINN can dive the vehicle even when the senso is being used to look at othe featues in its envionment, such as signs, povided that the oad emains at least patially in the eld of view. 5. Results and Discussion This system was implemented on the Canegie Mellon Navlab II, using a video camea on a pan/tilt mount (with constant tilt used thoughout the expeiments). Tests wee conducted on a single-lane bicycle path, and on a two-lane steet. The netwok was tained with the video camea pointing diectly ahead. In the st expeiment, the camea was oset at a constant angle and the vehicle switched to autonomous contol. Panacea compensated coectly fo the change in oientation and dove successfully. Subsequent tests wee conducted with the senso unde Panacea's contol and the system dove as eliably as the unmodied ALVINN system. A compaison between the two systems was then made at a shap fok in the oad (See Figue 6). With a xed camea, ALVINN was unable to negotiate this stetch of the oad. The main eason fo ALVINN's diculty in this situation is that oad featues on a shaply cuved oad fall outside a xed camea's eld of view. In addition, the obot vehicle eacts slowly to steeing commands wheeas a steeable senso can pan fast enough to keep the oad in sight at all times. A senso which pans unde Panacea's contol esults in impoved pefomance since the view seen by the senso tends to coespond moe closely to the images in the taining set. Since the senso points towads the TP, the impotant featues in the scene ae always within the eld of view and the netwok is less likely to make steeing eos. In paticula, when the obot sees a fok in the oad, the new system is less likely to dithe ove the decision since whicheve oad segment st appeas most appopiate is immediately centeed into the eld of view, and the chance of the netwok choosing the othe fok is thus substantially educed. Highe level planning systems could exploit this by pointing the senso in the appopiate diection at an intesection, causing ALVINN to choose one fok ove anothe. This extension
Figue 6: Panacea successfully negotiates a shap fok in Schenley pak. Note the panning of the pan/tilt platfom (bottom cente) at the intesection to keep the ight fok in view. has not yet been implemented. Panacea embodies the following benecial attibutes: Sound theoetical basis: Since Panacea uses the pue-pusuit model, which is implicit in ALVINN, no additional assumptions ae intoduced. Futhemoe, when the senso conguation is static, the outputs of both systems ae identical, so Panacea is tanspaent in that case. Modulaity: Panacea is a post-pocessing module fo existing ALVINN systems. No additional time is equied to tain ALVINN diving netwoks. This also means that netwoks tained on a xed senso can be used without modication in the new system. Eciency: The equations given above ae vey ef- cient, and the ovehead of using Panacea on the ALVINN system is negligible. 6. Futue Wok Panacea has shown that active peception and neual netwoks can be successfully integated into a modula system fo autonomous diving. Although the implemented system aleady demonstates some advantages of this mege, thee ae many inteesting topics which meit futhe exploation. In paticula, the notion of decoupling the senso motion fom the diving netwok can be exploited futhe. One application whee it may be desiable to point the senso at the TP without necessaily diving towads it is duing obstacle avoidance. Hee it is impotant that the video camea used fo oad following continue to focus its attention on the oad, even duing the tempoay evasive maneuveing so that the diving algoithms can continue uninteupted afte the obstacle has been successfully avoided. Convesely, an example whee it may be desiable to point the senso away fom the cente of the oad, while continuing to dive towads it, is in oad sign detection. This is also an example of how multiple systems could successfully shae the same active senso, since the ALVINN system, when augmented by Panacea, does not need the senso to point at the cente of the oad as long as the elevant featues emain visible in the senso's eld of view. 7. Conclusion Panacea's success stems fom the ability to intepetate ALVINN's output as moe than just a steeing command. The pocedue used to tain the neual netwok in the ALVINN system insues that its output also epesents the position of the oad ahead of the vehicle. Reasoning about the behavio of the netwok to extact additional useful infomation has poven to be a poweful technique fo impoving ALVINN's pefomance. Two othe examples of easoning about netwok behavio in the ALVINN system ae the Input Reconstuction Reliability Estimation (IRRE) technique [3], and ou ecent wok on latency compensation. In IRRE, the eliability of the netwok's esponse is estimated fom the accuacy with which the netwok can econstuct the input image fom its intenal epesentation. This technique augments the standad steeing diection output fom the ALVINN netwok with a measue of the netwok's condence in the appopiateness of that steeing diection. This condence measue has been employed to select the most appopiate netwok among seveal tained fo dieent situations. In the latency compensation technique developed fo ALVINN, the delays inheent in captuing the video image and pocessing it with the neual netwok ae mod-
eled and then used to ene the steeing diection poduced by the neual netwok. Reasoning about the delay in executing the netwok's command allows ALVINN to eliminate contol oscillations which can occu when diving at highway speeds. By teating ALVNN's neual netwok as a system which can be modeled and easoned about, instead of meely a black box which poduces steeing commands, it has been possible to make extensions and impovements to the ALVINN system which would have been dicult o impossible to achieve using stictly connectionist methods. Acknowledgements This eseach was patly sponsoed by DARPA, unde contacts \Peception fo Outdoo Navigation" (contact numbe DACA76-89-C-14, monitoed by the US Amy Topogaphic Engineeing Cente) and \Unmanned Gound Vehicle System" (contact numbe DAAE7-9- C-R59, monitoed by TACOM). Refeences [1] Pomeleau, D.A. (1991): Ecient Taining of Ati- cial Neual Netwoks fo Autonomous Navigation. Neual Computation 3:1 pp. 88-97. [2] Pomeleau, D.A. (1992): Neual Netwok Peception fo Mobile Robot Guidance. Canegie Mellon technical epot CMU-CS-92-115. [3] Pomeleau, D.A. (1993): Input Reconstuction Reliability Estimation. Advances in Neual Infomation Pocessing Systems 5 Giles, C.L., Hanson, S.J., and Cowan, J.D. (eds.) [4] Tuk, M., Mogenthale, D., Gemban, K., and Maa, M. (1988): VITS A Vision System fo Autonomous Land Vehicle Navigation. IEEE Tansactions on Patten Analysis and Machine Intelligence, Vol 1, Numbe 3 [5] Wallace, R., Stentz, A., Thope, C., Moavec, H., Whittake, W., and Kanade, T. (1985): Fist Results in Robot Road-Following. Poc. IJCAI-85