Online Learning of a Full Body Push Recovery Controller for Omnidirectional Walking

Size: px

Start display at page:

Download "Online Learning of a Full Body Push Recovery Controller for Omnidirectional Walking"

Amy Parker
5 years ago
Views:

1 th IEEE-RAS International Conference on Humanoi Robots Ble, Slovenia, October 26-28, 2011 Online Learning of a Full Boy Push Recovery Controller for Omniirectional Walking Seung-Joon Yi, Byoung-Tak Zhang, Dennis Hong an Daniel D. Lee Abstract Bipeal humanoi robots are inherently unstable to external perturbations, especially when they are walking on uneven terrain in the presence of unforeseen collisions. In this paper, we present a push recovery controller for position-controlle humanoi robots which is tightly integrate with an omniirectional walk controller. The high level push recovery controller learns to integrate three biomechanically motivate push recovery strategies with a zero moment point base omniirectional walk controller. Reinforcement learning is use to map the robot walking state, consisting of foot configuration an onboar sensory information, to the best combination of the three biomechanical responses neee to reject external perturbations. Experimental results show how this online metho can stabilize an inexpensive, commerciallyavailable DARwin-OP small humanoi robot. Keywors: Bipeal Omniirectional Walking, Full Boy Push Recovery, Reinforcement Learning, Online Learning I. I NTRODUCTION Fig. 1. Overview of the omniirectional walk controller integrate with full boy push recovery controller. Due to their small footprint an high center of mass, bipeal humanoi robots are not statically stable. Open loop ynamic walking methos typically assume flat, well-efine surfaces for walking that ensure the pre-efine joint trajectories exhibit ynamic stability. However, these techniques can fail ue to unevenness of the surface, collisions with obstacles, an unknown moeling errors of the robot or its actuators. Active stabilization is therefore crucial for operation of humanoi robots in realistic environments, an has been a topic of great interest in humanoi research. There have been two main approaches propose to hanle active stabilization of humanoi robots. The first approach uses sensitive force sensors an an accurate full boy moel of the robot to calculate the proper joint torques require to reject any external isturbance forces [1], [2], [3], [4], [5]. The other is a biomechanically motive approach which focuses on fast, reactive push recovery behaviors using a simplifie moel of the robot [6], [7], [8], [9], [10], [11], [12]. One common rawback of these approaches is that most physical implementations require specialize harware such as triaxial force an torque sensors, torque-controlle actuators, in aition to rapi computational processing an feeback control, making their implementation on inexpensive, commercially-available, position-controlle humanoi robots such as the Nao1 or DARwIn-OP2 infeasible. In previous work, we escribe a machine learning approach to learn a hierarchical full boy push recovery controller for such harware constraine humanoi robots [13]. In this paper, we aress several shortcomings of the prior work. The first issue is that our previous work, along with most other humanoi push recovery implementations, assumes that the robot is either staning still or walking in place, an oes not aress more general situations when the robot is walking with nonzero velocity. The secon issue is that a simulator was use to learn the controller, like other approaches utilizing machine learning [4], [5], which may not correctly moel the robot an environmental properties. Thus, the aim of this work is twofol. The first is to esign an integrate walk controller with push recovery for positioncontrolle humanoi robots without force sensors that can stabilize the robot against external isturbances uring omniirectional walking. The secon is to use the physical robot an experimental apparatus instea of a simulate environment to learn the push recovery controller parameters in an online fashion. We first esign a step-base omniirectional walk controller that integrates with the biomechanically motivate push recovery controllers. Our metho generates foot an GRASP Laboratory, University of Pennsylvania, Philaelphia, PA {yiseung,lee}@seas.upenn.eu BI Laboratory, Seoul National University, Seoul, Korea. btzhang@ceas.snu.ac.kr RoMeLa Laboratory, Virginia Tech, Blacksburg, VA hong@vt.eu /11/$ IEEE 1

torso trajectories in real time base upon ynamic stability criteria, which can be switche to reject external perturbations.

The robot is then use in conjunction with a motorize moving platform to apply a series of controlle perturbations to the robot.

Experimental results show that our metho can be successfully applie to the DARwin-OP robot platform. The remainer of the paper procees as follows.

Section III explains the etails of the stepbase omniirectional walk controller which is fully integrate with the push recovery controller.

Finally, we conclue with a iscussion of outstaning issues an potential future irections arising from this work. II.

perturbations: which we enote as ankle, hip an step push recovery strategies [7].

2 torso trajectories in real time base upon ynamic stability criteria, which can be switche to reject external perturbations. The current foot configuration an walking phase are taken into consieration to etermine the control inputs for push recovery controllers. The robot is then use in conjunction with a motorize moving platform to apply a series of controlle perturbations to the robot. The parameters for the hierarchical controller are then learne in an online fashion using reinforcement learning. Experimental results show that our metho can be successfully applie to the DARwin-OP robot platform. The remainer of the paper procees as follows. Section II reviews three biomechanically motivate push recovery controllers an its implementation on position controlle humanoi robots. Section III explains the etails of the stepbase omniirectional walk controller which is fully integrate with the push recovery controller. Section IV explains how the DARwIn-OP robot is use to learn the appropriate controller from experience in an online fashion. Section V shows the experimental results using the learne controller. Finally, we conclue with a iscussion of outstaning issues an potential future irections arising from this work. II. B IOMECHANICALLY MOTIVATED PUSH RECOVERY CONTROL FOR POSITION CONTROLLED ROBOTS Biomechanical stuies show that humans isplay three istinctive motion patterns in response to suen external perturbations: which we enote as ankle, hip an step push recovery strategies [7]. Although there has been some theoretical analysis of these strategies using simplifie moels, physical implementation of such analytical controllers on a generic, position controlle humanoi robot is not straightforwar, an there has been little research on how to optimally mix these three strategies as humans o. Instea of fully relying on analytic moels, we suggeste a machine learning approach to etermine the appropriate push recovery controller which minimizes a preetermine cost function [14]. To generate the proper combination of push recovery strategies, a hierarchical high level controller uses current proprioceptive an inertial sensory information to etermine the optimal combination of the three recovery strategies. In this section, we review the implementation of the three biomechanically motivate push recovery controllers on a position controlle robot, as shown in Figure 2. (c) Fig. 2. Three biomechanically motivate push recovery strategies an corresponing controller for a position-controlle robot. Ankle strategy that applies control torque on ankle joints. Hip strategy which uses angular acceleration of torso an limbs to apply counteractive groun reaction force. (c) Step strategy which moves the base of support to a new position. as θankle = xankle (1) where θankle is the joint angle bias an xankle is the original ankle controller input. In aition to the ankle joints, we also moulate arm position to apply aitional effective torque at the ankles in a similar way, unless overrien by the hip controller. A. Ankle push recovery The ankle controller applies control torque on the ankle joints to keep the center of mass within the base of support. For position controlle actuators, controlling the ankle torque can be one by either controlling the auxiliary zero moment point (ZMP) which accelerates the torso to apply effective torque at ankle [13], [15] or moulating the target angle of the ankle servo. In this work we use the latter metho, which is foun to be effective for the particular robot we use in our experiments. Then the controller can be simply implemente B. Hip push recovery The hip controller uses angular acceleration of the torso an limbs to generate a backwar groun reaction force to pull the center of mass back towars the base of support. For maximum effect, a bang-bang profile of the following form can be use τhip (t) = τmax u(t) 2τmax u(t TR1 ) + τmax u(t TR2 ) 2 (2)

where u(t) is the unit step function, τ max is the maximum torque that the joint can apply, T R1 is the time the torso stops accelerating an T R2 is the time torso comes to a stop.

3 where u(t) is the unit step function, τ max is the maximum torque that the joint can apply, T R1 is the time the torso stops accelerating an T R2 is the time torso comes to a stop. After T R2, the torso angle shoul return to the initial position. This two-stage control scheme can be approximately implemente with position controlle actuators as { xhip 0 t < T R2 θ torso = T x R3 t (3) hip T R3 T R2 T R2 t < T R3 where θ torso is the torso pose bias, T R3 the time torso angle completes returning to initial position, an x hip is the input of hip controller. Arm rotation can also be use uring hip push recovery to apply aitional groun reaction force on the robot. C. Step push recovery The step controller effectively moves the base of support by taking a step. This is implemente by overriing the step controller to insert a new step with relative target foot position x capture. In orer to not lift the current support foot, the state of the robot nees to be monitore to etermine the current foot configuration as well as the perturbation irection. Only uring the appropriate walking phase is the walk control overrien with the new target foot position. III. A STEP-BASED OMNIDIRECTIONAL WALK CONTROLLER WITH PUSH RECOVERY CONTROL In this section, we escribe the etails of our omniirectional walk controller which is integrate with the full boy push recovery controller. As the active push recovery requires walk phase moulation an real-time moification of laning position, we cannot irectly use a typical perioic time-base walk controller. Instea, we use a step-base walk controller that generates a series of steps with iniviual ajustable parameters such as uration, support foot selection an laning position. The overall walk control is separate into a step controller an trajectory controller, where the step controller etermines the parameters for each step an the trajectory controller generates foot an torso position trajectories base upon that information. In this section we cover each controller in etail, an show how they are integrate with the high level push recovery controller escribe in the previous section. A. Step controller The step controller etermines the parameters of each step, which is efine as ST EP i = {SF,WT,t ST EP,L i,t i,r i,l 1+1,T i+1,r i+1 } (4) where SF enotes the support foot, WT enotes the type of the step, t ST EP is the uration of the step, an L i,t i,r i, an L i+1,t i+1,r i+1 are the initial an final 2D poses of left foot, torso an right foot in (x,y,θ) coorinate. L i,t i,r i are etermine by the final feet an torso poses from the last step, an L i+11,r i+1 is calculate using the comman walk velocity an current foot configuration to enable omniirectional walking. Foot reachability an self-collision constraints are Fig. 3. The step-base walk controller. An example of walking behavior which is compose of two steps, ST EP i an ST EP i+1. Corresponging ZMP an torso trajectory p an x. Timing parameters of φ 1 = 0.2 an φ 2 = 0.8 are use. also taken into account when calculating target foot poses. Target torso pose T i+1 is set to the mile point of L i+1 an R i+1 by efault, so that the transition occurs at the most stable posture where the center of mass (CoM) lies on the mipoint of the two support positions. Once we have the initial an final poses of the feet an torso, we can efine the reference ZMP trajectory p i (φ) as the following piecewise-linear function for the left support case p i (φ) = T i (1 φ φ φ 1 ) + L i φ 1 0 φ < φ 1 L i φ 1 φ < φ 2 T i+1 (1 1 φ ) + L i 1 φ φ 2 φ < 1 or for the right support case p i (φ) = T i (1 φ φ φ 1 ) + R i φ 1 0 φ < φ 1 R i φ 1 φ < φ 2 T i+1 (1 1 φ ) + R i 1 φ φ 2 φ < 1 where φ is the walk phase an φ 1,φ 2 are the timing parameters etermining the transition between single support an ouble support phase. Finally, the step controller allows for inter-step overrie from the push recovery controller. The hip an step push recovery controller can temporarily stop the walking to stabilize the robot after each push recovery behavior is initiate, an the step push recovery controller can initiate a special capture step to reject large perturbations. The walk type parameter W T is use to escribe these special types of steps. B. Trajectory controller The trajectory controller generates foot an torso trajectories for the current step efine in (4). First we efine the single support walk phase φ single as φ single = 0 0 φ < φ 1 φ φ 1 φ 2 φ 1 φ 1 φ < φ 2 1 φ 2 φ < 1 (5) (6) (7) 3

4 then we use following parameterize trajectory function to generate the foot trajectories f x (φ) = φ α + βφ(1 φ) (8) L i (φ single ) = L i+1 f x (φ single ) + L i (1 f x (φ single )) (9) R i (φ single ) = R i+1 f x (φ single ) + R i (1 f x (φ single )) (10) The torso trajectory x i (t) is calculate accoring to the following ZMP criterion for a linear inverte penulum moel p = x t ZMP ẍ (11) The piecewise linear ZMP trajectory we use in (5), (6) yiels for x i (φ) with zero ZMP error uring the step perio x i (φ) = p i (φ) + a p i eφ/φ ZMP + a n i e φ/φ ZMP φ ZMP m i sinh φ φ 1 φ ZMP 0 φ < φ 1 p i (φ) + a p i eφ/φ ZMP + a n i e φ/φ ZMP φ 1 φ < φ 2 p i (φ) + a p i eφ/φ ZMP + a n i e φ/φ ZMP φ ZMP n i sinh φ φ 2 φ ZMP φ 2 φ < 1 (12) where φ ZMP = t ZMP /t ST EP an m i, n i are ZMP slopes which are efine as follows for the left support case an for the right support case m i = (L i T i )/φ 1 (13) n i = (L i T i+1 )/(1 φ 2 ) (14) m i = (R i T i )/φ 1 (15) n i = (R i T i+1 )/(1 φ 2 ) (16) Then the parameters a p i an a n i can be uniquely etermine by the bounary conition x i (0) = T i an x i (1) = T i+1. This analytic solution has zero ZMP error uring the step perio but iscontinuous jerk at the transitions. However, as this transition occurs in the mile of the most stable ouble support stance, we foun this oes not hamper stability. In aition to calculating foot trajectories base upon preetermine target foot poses, the intra-step overrie from the push recovery controller can also moify the foot laning position while stepping. When the step push recovery is triggere uring the initial phase of walking, the target laning position is overrien an a new foot trajectory is generate uner foot reachability constraints an foot maximum velocity constraints. C. Push recovery controller The push recovery controller consists of three low level biomechanically motivate push recovery strategies an a high level controller that generates the control inputs for them base on sensory feeback an current state information. There is no close-form analytic solution for combining the iniviual low level controllers, so we use a machinelearning approach that trains the overall controller to optimize a cost function from experience. We formalize the high level controller as a reinforcement learning problem with the following state S = { θ IMU,θ gyro,θ f oot,φ,fd,hcs } (17) where θ IMU an θ gyro are the torso pose an gyroscope ata from inertial sensors, θ f oot is the support foot pose calculate using forwar kinematics an onboar sensors, φ is the walk phase, FD is the foot isplacement, an HCS is the hip controller state. The learne action is efine as the combine inputs for the three low level controllers A = { } x ankle,x hip,x capture (18) an the rewar is efine as R = θ gyro 2 + g z 0 θ IMU 2 (19) where g is the gravitational constant an z 0 is the COM height. IV. ONLINE LEARNING OF THE PUSH RECOVERY CONTROLLER We have implemente the integrate walk controller with push recovery on a commercially available small humanoi robot an traine the controller in an online fashion. In this section, we present the etails of the learning setup. A. Harware setup We use the commercially available DARwIn-OP humanoi robot evelope by Robotis Co., Lt. an the RoMeLa laboratory. It is 45 cm tall, weighs 2.8kg, an has 20 egrees of freeom. It has a USB camera for visual feeback, an 3- axis accelerometer an 3-axis gyroscope for inertial sensing. Position-controlle Dynamixel servos are use for actuators, which are controlle by a custom microcontroller connecte by an Intel Atom-base embee PC at a control frequency of 100Hz. To generate repeatable external perturbations, a motorize moving platform was constructe using Dynamixel servomotors. The motorize platform is attache to the same bus connecting the actuators of the robot, an controlle by the same I/O process. To generate maximum peak acceleration, the platform is slowly accelerate in one irection an then suenly accelerate in the opposite irection. We have foun the platform can generate accelerations greater than 0.5g while carrying the robot, proviing large enough perturbations to make the robot fall. B. Learning setup In previous work [13], a policy graient reinforcement learning algorithm was use to learn the push recovery controller in a simulate environment. However, the same approach cannot be irectly applie to the physical robot for online learning. The main issue with machine learning approaches on physical robots is that the amount of training ata is severely restricte. Unlike in simulation, small humanoi robots cannot continuously operate for long perios of time ue to heat builup at the actuators, an the process of gathering training ata is slower an may require 4

(c) Fig. 4. Learning setup incluing the servo controlle platform connecte to the DARwIn-OP robot. Commane position of the platform to inuce instantaneous isturbance.

Learne parameter set after training that triggers hip strategy when large amount of isturbance is applie. The same amount of isturbance is applie to the robot. more human intervention.

To overcome these problems, we simplify the reinforcement learning formulation of the high level controller as follows.

Learning the push recovery controller (20) We traine the push recovery controller using the servo controlle platform.

Each trial takes roughly 10 secons on average, an 20 trials are one per episoe to improve the controller.

5 (c) Fig. 4. Learning setup incluing the servo controlle platform connecte to the DARwIn-OP robot. Commane position of the platform to inuce instantaneous isturbance. (c) Actual acceleration of the platform. Fig. 5. Learning process of the push recovery controller. Initial parameter set which oes not trigger hip strategy. Learne parameter set after training that triggers hip strategy when large amount of isturbance is applie. The same amount of isturbance is applie to the robot. more human intervention. Also, the sensory noise present in physical experiments are not well-moele in simulations. To overcome these problems, we simplify the reinforcement learning formulation of the high level controller as follows. First we use the heuristic isturbance variable θ as a single continuous state variable which is efine as θ = s(θimu + kgyro θgyro ) V. E XPERIMENTAL RESULTS A. Learning the push recovery controller (20) We traine the push recovery controller using the servo controlle platform. The robot is set to walk in ifferent irections, an a platform movement is triggere at ifferent phases of the walking. Then the robot is repositione by the human operator to start a new trial. Each trial takes roughly 10 secons on average, an 20 trials are one per episoe to improve the controller. Figure 5 shows a comparison of the robot behavior before an after 20 episoes of training with the same amount of perturbation. We can see that the robot learns to apply the appropriate push recovery behavior to reject large perturbations uring walking. where s is a moving average smoothing function an kgyro is a heuristic mixing parameter. The rest of the state variables are iscretize to form the iscrete state variables ˆ. ˆ HCS S = φˆ, FD, (21) Simple parametric functions with preset parameters kankle, khip, kstep are then use to escribe the possible controller actions: θ < θankle 0 DB kankle (θ θankle (S )) θ θankle xankle = DB (S )) θ θ kankle (θ + θankle ankle (22) θ < θhip 0 khip θ θhip xhip = (23) khip θ θhip T H θ < θstep (S ) 0 kstep θ θstep xcapture = (24) T H kstep θ θstep (S ) B. Testing the push recovery controller We let the robot with learne controller freely walk over a flat surface an applie a number of pushes with various irections an magnitue. Figure 6 shows some examples of robot response against external isturbances3. We see that the learne controller can successfully trigger appropriate push recovery behaviors to keep the omniirectionally walking robot from falling own. VI. C ONCLUSIONS Finally, to reuce the effects of incorrect actions uring normal walking, a negative training set is generate by letting the robot walk without any external perturbations on the platform to prevent the controller from learning overly sensitive corrections. We have emonstrate an integrate controller that enables full boy push recovery uring omniirectional walking 3 5

for humanoi robots without specialize sensors an actuators.

The proper parameters for the hierarchical controller are learne online using a commercially-available small humanoi robot along with a servo-controlle moving platform.

Possible future work inclues incorporating more sophisticate learning algorithms to utilize the limite training ata, an implementing these algorithms on full-size humanoi robots.

This work was also partially supporte by the IT R&D program of MKE/KEIT (KI002138, MARS), the inustrial strategic technology evelopment program (10035348) fune by the Korean government (MKE), an the

23, no. 4, pp. 817 822, 2007. [3] S.-H. Hyon, R. Osu, an Y. Otaka, Integration of multi-level postural balancing on humanoi robots, in ICRA, 2009, pp. 1549 1556. [4] T. Matsubara, J. Morimoto, J.

Yamaguchi, S.-H. Hyon, an T. Ogasawara, Reinforcement learning for balancer embee humanoi locomotion, in Humanois, ec. 2010, pp. 308 313. [6] B. Jalgha, D. C. Asmar, an I.

Goswami, A biomechanically motivate twophase strategy for bipe upright balance control, in ICRA, 2005, pp. 2008 2013. [8] J. Pratt, J. Carff, an S.

Kuffner, A feeback controller for bipe humanois that can counteract large perturbations uring gait, in ICRA, 2005, pp. 1989 1995. [10] D. N. Nenchev an A.

6 for humanoi robots without specialize sensors an actuators. Three low level biomechanically motivate push recovery strategies are implemente on a position controlle humanoi robot, an integrate with a non-perioic, stepbase omniirectional walk controller. The proper parameters for the hierarchical controller are learne online using a commercially-available small humanoi robot along with a servo-controlle moving platform. Experimental results show that the traine controller can successfully initiate a full boy push recovery behavior uner external perturbations while walking in an arbitrary irection. Possible future work inclues incorporating more sophisticate learning algorithms to utilize the limite training ata, an implementing these algorithms on full-size humanoi robots. ACKNOWLEDGMENTS We acknowlege the support of the NSF PIRE program uner contract OISE This work was also partially supporte by the IT R&D program of MKE/KEIT (KI002138, MARS), the inustrial strategic technology evelopment program ( ) fune by the Korean government (MKE), an the BK21-IT program. R EFERENCES [1] S.-H. Hyon an G. Cheng, Disturbance rejection for bipe humanois, ICRA, pp , apr [2] J. Park, J. Haan, an F. C. Park, Convex optimization algorithms for active balancing of humanoi robots, IEEE Transactions on Robotics, vol. 23, no. 4, pp , [3] S.-H. Hyon, R. Osu, an Y. Otaka, Integration of multi-level postural balancing on humanoi robots, in ICRA, 2009, pp [4] T. Matsubara, J. Morimoto, J. Nakanishi, S.-H. Hyon, J. G. Hale, an G. Cheng, Learning to acquire whole-boy humanoi center of mass movements to achieve ynamic tasks, Avance Robotics, vol. 22, no. 10, pp , [5] A. Yamaguchi, S.-H. Hyon, an T. Ogasawara, Reinforcement learning for balancer embee humanoi locomotion, in Humanois, ec. 2010, pp [6] B. Jalgha, D. C. Asmar, an I. Elhajj, A hybri ankle/hip preemptive falling scheme for humanoi robots, in ICRA, [7] M. Aballah an A. Goswami, A biomechanically motivate twophase strategy for bipe upright balance control, in ICRA, 2005, pp [8] J. Pratt, J. Carff, an S. Drakunov, Capture point: A step towar humanoi push recovery, in Humanois, 2006, pp [9] T. Komura, H. Leung, S. Kuoh, an J. Kuffner, A feeback controller for bipe humanois that can counteract large perturbations uring gait, in ICRA, 2005, pp [10] D. N. Nenchev an A. Nishio, Ankle an hip strategies for balance recovery of a bipe subjecte to an impact, Robotica, vol. 26, no. 5, pp , [11] B. Stephens, Humanoi push recovery, in Humanois, [12] B.-K. Cho an J.-H. Oh, Practical experiment of balancing for a hopping humanoi bipe against various isturbances, in IROS, 2010, pp [13] S.-J. Yi, B.-T. Zhang, an D. D. Lee, Online learning of uneven terrain for humanoi bipeal walking. in AAAI 10, [14] S.-J. Yi, B.-T. Zhang, D. Hong, an D. D. Lee, Learning full boy push recovery control for small humanoi robots. in ICRA, [15] S. Kajita, M. Morisawa, K. Haraa, K. Kaneko, F. Kanehiro, K. Fujiwara, an H. Hirukawa, Bipe walking pattern generator allowing auxiliary zmp control, in IROS, oct. 2006, pp (c) () (e) (f) Fig. 6. Responses of the learne push recovery controller against perturbation uring omniirectional walking. step strategy uring walking forwar. step strategy uring siestepping. (c) hip strategy uring walking forwar. () hip strategy uring turning. (e) hip an step strategies uring walking forwar. (f) hip an step strategies uring siestepping. 6

Practical Bipedal Walking Control on Uneven Terrain Using Surface Learning and Push Recovery

2011 IEEE/RSJ International Conference on Intelligent Robots and Systems September 25-30, 2011. San Francisco, CA, USA Practical Bipedal Walking Control on Uneven Terrain Using Surface Learning and Push