NRI: INT: Individualized Co-Robotics

Transcription

1 Figure 1: Design studies for the full lower body co-robot emulator and portable version (far right). NRI: INT: Individualized Co-Robotics The NRI-2.0 program significantly extends this theme [collaborative robots (co-robots)] to focus on issues of scalability [and variety of behavior]:...; how robots can be designed [optimized online] to facilitate achievement of a variety of tasks in a variety of environments [for a variety of users], with minimal modification to the hardware and software;... [NSF17518] Future physical co-robots will need to be easily customizable. This proposal focuses on rapidly individualizing physical co-robots and other physical interfaces using optimization. Our preliminary results with an ankle exoskeleton are exciting, with about a 20-25% reduction in metabolic cost after optimization customizes assistance and augmentation for an individual user performing a specific task. We have seen similar preliminary results in a range of conditions including unilateral and bilateral ankle assistance during walking at various speeds with various loads, walking uphill and downhill, and running. This proposal focuses on 1) Improving the customization process, including exploring other optimization parameters and structure, optimization algorithms, and parameter settings for the current algorithm (CMA-ES). Currently customization takes about one hour to optimize an interaction policy for a single behavior for most individuals. Our long term goal is to build a library of customized interaction policies for an individual performing a variety of tasks, and tune the library online as the user does desired tasks. 2) Developing a model of the co-optimization process, in which a robot optimizes an interaction policy while a human is simultaneously adapting and optimizing their behavior for a different optimization criterion. When does this process converge to desirable optima? What undesired transient optimization behaviors need to be detected and reduced or eliminated? 3) Understanding the effects of gross muscle properties (a Hill-type model) and molecular behavior (a Huxley-type model) on the co-optimization process. We hope to be able to predict the final equilibrium of any human-robot co-optimization (or co-learning) process. 4) We will evaluate our results on lower body physical co-robot testbeds we propose to build (Figure 1), the first in a laboratory using a treadmill, and the second outdoors on irregular terrain. The outdoors testbed will use a similar exoskeleton frame, but replace the benchtop actuators and lab power supply with smaller motors and batteries. These systems will take advantage of our improved optimization processes based on our improved understanding of co-optimization and muscle modeling, and produce new data to further improve our understanding, in a virtuous cycle. A Mystery: Based on our prior co-robot augmentation experiments [16, 18, 38], optimal augmentation is usually achieved with exoskeleton torques that are about half the values of human joint torques observed during unassisted locomotion. Why don t users fully relax and let the co-robot do all the work, as patients did with early rehabilitation robots? Hypotheses as to why this is the case include nonlinear muscle properties, discomfort caused by forces being applied to soft tissue instead of bones, a desire on the part of the subject to avoid muscles completely relaxing and going slack during the imposed movement, a desire to remain in control, and lack of trust of the co-robot by the user. In order to increase acceptance of physical co-robots, we need to understand this issue more clearly. In addition, when designing physical co-robots it is important to consider how muscle-tendon mechanics might change due to interactions with the device and to ensure that any 1

2 Figure 2: (A) Parameterization of ankle torque augmentation. Each control law determined applied torque as a function of time, normalized to stride period, as a cubic spline defined by peak time, rise time, fall time and peak torque. (B) Examples of possible torque patterns in this space. (C) Co-robot emulator system used to apply torque to the human ankle in experiments. Off-board motor and control hardware actuated a tethered exoskeleton worn on one ankle while participants walked on a treadmill. compromised function of the biological system such as detuning the mass-spring dynamics of legged locomotion are sufficiently compensated for. The proposed work will help us understand these issues, and identify what is desirable in human-robot physical interfaces. Why is the proposed research appropriate for the NRI 2.0 program? From the call for proposals: To scale up effectively, robots will need to be easily customizable... Features of both the hardware and software should facilitate robots achieving a wide variety of tasks, in a wide variety of situations, for a wide diversity of people. We are investigating easily customizable robots for achieving a variety of tasks in a variety of situations, as well as facilitating physical collaboration (including peer-to-peer; collaborative manipulation; and augmentation of human capabilities). Users and co-robots must learn to work together, and thus we need to understand what happens when two optimizers or learning systems are interacting yet working towards different goals. We note the NSF s support of co-robot assistance and exoskeletons. The DoD is interested in co-robots that augment users (Soft exosuits and the SOCOM Talos program, for example.) The DoD is interested in dynamic modeling of the human-robot partnerships to allow continuous improvement of joint performance in real-world applications, as well as investigations regarding the effectiveness of various models of human-robot interaction. We note the DOE s interest in improving worker ergonomics as well as reducing physical demands and stress using exoskeletons. We believe our approach can augment workers while protecting them from internal injuries as well as learn the unique movements of a particular user. We note that interdisciplinary research and research in collaboration with government labs is especially encouraged. We are currently working with the U.S. Army Natick Soldier Research, Development and Engineering Center (NSRDEC) (Collins) on exoskeletons and are negotiating a contract with the US Special Operations Command (SOCOM) (Atkeson) on building an exoskeleton through a company (Apptronik). Our work with laboratory and outdoor co-robot testbeds will allow us to evaluate our approach on complete physical co-robotic systems in real-world settings, integrating relevant technologies. Our longer-term vision is to use our co-robot testbeds to explore physical designs and develop software for future co-robot systems. Versatile testbed systems like the ones employed here could be used to identify optimal device characteristics during a design or prescription process, and then customized mobile devices, adaptive or static, could be fabricated. We expect that the resulting high-performance exoskeletons, prostheses and other devices will be used to improve mobility for people with a wide range of unique physiological needs, from individuals with amputation or disability due to stroke to athletes and soldiers. Our objectives can only be attained by combining expertise in biomechanics, exoskeletons, and optimization, rather than with just a collection of smaller projects provided with similar resources. The overall impact of the proposed joint work on co-robot science and engineering will indeed be greater than the sum of potential individual investigator contributions. 2

3 Motivation Physical co-robots in general have great promise, but few have yet enhanced performance. A critical obstacle may be the reliance on intuition and hand tuning when determining device function. We have developed a method for automatically identifying optimal assistance and augmentation patterns for individual humans (Figure 2). In preliminary tests an evolution-inspired optimization algorithm (CMA-ES), tolerant of measurement noise and human adaptation, determined augmentation torque patterns for each subject that minimized a rapidly-updated estimate of metabolic rate. After optimizing augmentation for an exoskeleton worn on one ankle, participants (N=11) experienced a 23.6±8.0% decrease in metabolic cost compared to a zero-torque condition. This exceptional improvement in energy economy arose from customized augmentation patterns, which varied widely across participants, and from facilitation of human motor adaptation. Optimizing human performance can dramatically improve the effectiveness of assistive and augmentation devices for users with diverse physiological needs. Methods for automatically discovering, customizing and continuously adapting assistance and augmentation would overcome these challenges, allowing physical co-robots to achieve their potential. We call one such approach, in which device control is systematically varied during use so as to maximize human performance, human-in-the-loop optimization. However, closing the loop on human performance is also challenging. Objective functions based on measurements of human performance typically require lengthy evaluation periods and contain substantial noise; the best available estimate of metabolic energy cost, for example, requires about one minute of respiratory data per evaluation [68, 46]. The human part of the system also has time-varying dynamics, because humans may adapt slowly to new device behaviors and change their reactions with new exposures [69]. The spaces explored are often high-dimensional, because control laws that are general enough to approximate globally-optimal assistance and augmentation strategies are likely to require multiple parameters per assisted joint [31]. Initial efforts in this domain have demonstrated the ability to optimize a single gait or device parameter using line search [24] or gradient descent [46], but these methods are inefficient, due to sensitivity to noise and drift, and scale poorly, particularly in the presence of parametric interactions. Many optimization methods that work well in simulation [72] are subject to these problems; building a quadratic approximation takes time, and the human is changing during that time. Related Work Discovering effective strategies for assisting human gait is challenging. For more than a century, inventors and scientists have developed exoskeletons and prostheses intended to improve human locomotor performance, particularly in terms of energy economy [21]. Few approaches have been successful [33, 53, 59, 20, 50], however, with only modest enhancements compared to the potential benefits expected based on simulations [80, 31, 77]. An overreliance on intuition and specialized hardware may be partially responsible for these shortcomings. Assistance and augmentation strategies have typically been derived from mathematical models [19], biomechanics observations [81], and humanoid robots [40], but each of these sources of inspiration simplifies important aspects of the human-robot system [36]. Experiments have primarily been conducted using specialized prototypes that embed a single intuited functionality, with each prototype requiring years of development, limiting exploration to only a small set of potential assistance and augmentation strategies. Compounding the challenge, physiological and neurological differences between individuals can cause divergent responses to the same device [94, 38, 62], and responses can vary strongly during the course of adaptation [30, 69]. Preliminary Work In preliminary work on online optimization of physical interfaces participants wore a torque-controlled, tethered exoskeleton on one ankle (Figure 2). The exoskeleton applied torque as a function of time when the foot was on the ground, defined by four control parameters that set the magnitude of peak torque and the timing of torque onset, peak and removal, constituting a control law. During the optimization phase of the experiment, an optimization algorithm, Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [32], was 3

4 Figure 3: (D) Optimized ankle exoskeleton torque pattern for each participant. Patterns varied widely and spanned a large portion of the allowable space. Lines are measured torque, normalized to stride time and averaged across strides. (E) Torques applied in the static and zero-torque conditions. The static pattern, based on [38], is similar to the optimized patterns, but resulted in higher metabolic rate. Torques were negligible in the zero-torque mode. Lines are measured torque, normalized to stride time and averaged across strides and participants. used to identify the control law that minimized the metabolic energy cost of walking for each participant. The metabolic rate corresponding to each control law was estimated by fitting a first-order model to two minutes of breath-by-breath metabolic rate data, using an inverse dynamics approach similar to [68, 24]. Optimized augmentation substantially improved energy economy for all participants, confirming the effectiveness of the algorithm. Parameters that minimized energy expenditure were identified after four generations (64 min of walking) for all but two participants; Subjects 6 and 10 appeared to become trapped in local minima, requiring a reset of the algorithm and additional walking (128 and 208 min total, respectively). In separate validation trials, optimized augmentation reduced the metabolic cost of walking to 2.20±0.43 W/kg (mean±standard deviation), down from 2.87±0.39 W/kg for walking with the exoskeleton in a fully passive zero-torque mode, an average reduction of 23.6±8.0%. The range of energy cost reductions was %. By the same measure, the largest average reduction provided by hand-tuned exoskeletons worn on both legs has been 14.5% [59]. Walking in street shoes is about 9.7% less costly than the zero-torque mode, suggesting about a 14% net improvement with optimal augmentation compared to normal walking, also exceptionally large. Optimal augmentation patterns varied widely across subjects (Figure 3), demonstrating the importance of customization. For example, optimal torque onset timing ranged from 17% to 37% of the stride period (Figure 3D), or 26% to 57% of the stance phase, about half the testable range in this and prior studies [53]. Optimized augmentation did not replace ankle torque as much as possible, nor did it provide the maximum possible positive mechanical work, inconsistent with some predictions [59, 31, 77]. Optimized torque patterns (Figure 3D) do share some qualitative features with each other, such as a peak torque that occurs at about 50% stride, suggesting qualities that may be beneficial for most people and useful initial parameters for future optimizations. However, even subtle differences in torque can have large and unexpected effects on energy use arising from complex interactions with the musculoskeletal and nervous systems [23, 38, 39]. Human-in-the-loop optimization accommodates this complexity. Comparisons to a static control law confirm the advantages of optimization and customization and suggest that facilitating human motor adaptation may be critical. During validation trials, we included a static control law designed to maximize positive ankle work (Figure 3E) that we previously found to reduce metabolic rate by 5.8% compared to the zero-torque mode in tests with the same emulator system [38]. In the current study, optimized control resulted in 5.2±7.2% lower metabolic rate than with the static control law. This is a substantial 4

5 improvement in performance, equivalent to previous findings for the total reduction due to augmentation [20]. Interestingly, the static control law yielded a much larger benefit compared to the zero-torque condition than previously observed, a 19.1±8.8% reduction in metabolic cost. This difference suggests an interaction between co-robot behavior and human motor adaptation. Participants had similar duration of exposure to the co-robot in both studies, but in the prior study were trained with a narrow range of eight static controllers, whereas they experienced 32 diverse controllers here. The wide-ranging, sometimes uncomfortable, control laws participants were exposed to during the optimization process may have forced them to explore new motor control strategies, which has been shown to be a necessary part of skill acquisition in some interventions [69]. While one may think of the co-robot as primarily adapting to the human, co-adaptation between human and robot seems to be essential to improved performance. We have also conducted single subject experiments on other conditions, with exciting results as well. We demonstrated the generality of our approach by applying it to several additional devices and locomotion conditions. One participant from the first study (N=1) wore an ankle exoskeleton [87] on each leg and experienced a 30% reduction in metabolic rate with optimized assistance compared to the bilateral zero-torque condition, demonstrating the effectiveness of the approach across different co-robot types. The improvement was larger than the 20% they experienced by assisting only one leg, suggesting that augmentation at additional joints will lead to greater improvements in performance. In this test we also measured a 17% reduction in metabolic rate compared to walking with no exoskeleton, confirming the expected benefits in absolute performance. In tests on another participant (N=1), we found the algorithm to be effective at reducing the metabolic cost of walking at a typical speed (1.25m/s; 28% vs. zero-torque; 24% vs. no exoskeleton), walking at a faster speed (1.75m/s; 34% vs. zero-torque; 24% vs. no exoskeleton) and walking uphill (10% uphill grade; 22% vs. zero-torque; 18% vs. no exoskeleton). Interestingly, our approach is failing to improve slow walking (N=1), driving the applied torque to zero (0.75m/s). We suspect this is because an open loop torque profile is not an appropriate augmentation policy for slow behaviors. Rather, a closed-loop feedback law would make more sense. These results demonstrate the effectiveness of the algorithm across different walking conditions, including cases where the best action is none at all. We also applied the approach to running with bilateral ankle exoskeletons (N=1), and found a 30% improvement in energy economy compared to the zero-torque mode and a 13% savings compared to running in normal shoes. This demonstrates the effectiveness of the algorithm across different gaits. Proposed Research: Exploring Other Optimization Algorithms We will explore a range of online data efficient optimization algorithms in addition to further exploration and tuning of CMA-ES. We will explore issues that span many possible optimization algorithms, such as convergence criteria. Algorithms we will test in simulation, and if successful, in actual experiments with human subjects include grid search, sequential one-dimensional hill-climbing (Powell s method), ordinal support vector machine, Nelder Mead/Simplex/Amoeba, linear modeling and LQR optimal control design, and trajectory optimization based on EMG and kinematic variables, In early pilot tests of related methods, model-explicit optimization techniques seemed to be ineffective due to sensitivity to noise and adaptation dynamics. CMA-ES worked well in pilot tests. The method is wellsuited to human-in-the-loop optimization because it handles noisy measurements, expensive objective function evaluations, nonlinear objective functions with unknown structures, and complex, subject-dependent human learning and adaptation processes well. CMA-ES is stochastic, which makes it less sensitive to noise than derivative-based methods such as gradient descent and hill-climbing methods such as Nelder-Mead. CMA- ES includes mechanisms to grow or shrink the standard deviation of the randomly-selected parameter values depending on the evolution of the estimate of the mean over time. These features make CMA-ES more robust against thresholds, discontinuities and local minima, as long as the initial values of the mean, covariance matrix, and standard deviation are well-chosen. CMA-ES is less sensitive to noise and drift because it uses only the rank order of the objective function values (trial scores), rather than the actual objective function values or their partial derivatives. 5

6 Proposed Research: Exploring Other Policy Parameterizations We will explore a much wider range of control policies in the proposed work as compared to the open loop torque trajectories used in the preliminary work. In that work the co-robot applied torque as a function of time when the foot was on the ground, defined by four control parameters that set the magnitude of peak torque and the timing of torque onset, peak and removal, relative to ground contact and normalized to the current stride period (Figure 2). The curve was composed of two cubic splines. Additional low-torque ramp-in and ramp-out patterns were included for the rest of the stance phase to improve torque tracking. Stride period was estimated online by low-pass filtering measured stride periods during walking. The ankle torque curve had a hill-like shape that can be divided into four sections: a shallow, low-torque setup ramp; a rising s-like cubic spline linking the onset point to the peak; a falling arc-like cubic spline linking the peak to the removal point; and a shallow, low-torque settling ramp. We used limits determined in pilot tests to set constraints on the four control parameters that defined the pattern of ankle torque. Given the approximate nature of these hand-selected parameter constraints, it is possible that a larger solution space could be achieved with further refinement. For example, we re-parameterized the optimization problem to improve the initial guess of the distribution defined by the covariance matrix, based on the results of pilot testing. Pilot tests suggested that the peak time and fall time, in units of percent stride, had smaller comfortable ranges than rise time, in units of percent stride, and peak torque, in units of Newton meters. After reparameterization, we found that this allowed subjects to complete the first generation with relative comfort, while covering a sufficient portion of the parameter space so as to span the eventual optimal parameter values. The solution space can be parameterized and constrained in other ways, which could be used to reflect desirable device characteristics such as energetic passivity. As mentioned previously, we expect closed-loop augmentation policies to be useful for slow movements, postures such as standing balance, and working against gravity. We may find that combinations of open-loop and closed loop policies may be more effective than pure open-loop policies for normal and fast movements. Proposed Research: Exploring Other Optimization Criteria Metabolic rate (indirect respirometry [68]) was used as a measure of overall performance in preliminary studies. Successful optimization of metabolic rate suggests that alternate objective functions with similar properties could be optimized, for example criteria related to speed or balance. We will explore the use of center of mass kinetics (individual limbs) to capture inter-limb coordination effects, joint kinetics and kinematics (inverse dynamics) to estimate musculotendon force and work, muscle activity (electromyography (EMG)) to capture neuromuscular effects, and muscle fascicle mechanics (measured with ultrasound and predicted with our Open- Sim model) to obtain fascicle work that could all be used to define more appropriate optimization criteria. User satisfaction [14] in terms of absolute rating. and comparisons could also be used. Optimizing co-robot assistance and augmentation based on measurements of the user is challenging. First, human measurements such as metabolic rate, muscle activity, and joint torques are noisy, owing both to complicated human physiological and mechanochemical dynamics and to shortcomings in measurement hardware. A second challenge is that evaluation of candidate conditions is very expensive in terms of time and human effort. Measurement of metabolic rate requires on the order of minutes of respiratory data from a human interacting with the device, due to delays in the expression of energy used by muscles in expired gasses [68]. Often, multivariate optimization methods require a large number of function evaluations per step, and this number increases with the dimensionality of the control parameter space. A third challenge is that the nature of the relationship between co-robot control parameters and human physiological processes such as metabolic rate is not known in advance and may include complex nonlinearities and local minima. A fourth challenge is that humans exhibit complex, individualized learning and adaptation processes when using a co-robot. This is problematic for gradient-based and quadratic approximation methods, because calculating the gradient or quadratic requires substantial time, during which the human is changing. The calculated gradient or quadratic will often be inaccurate or out of date. Human adaptation also is difficult for methods that attempt to develop models of the space 6

7 based on all available data, because data collected early in the adaptation process are likely to conflict with data from later in the process. Proposed Research: Scaling Up To More Joints A challenge is to increase the dimensionality of the optimization parameter space. In this preliminary work, we optimized four control parameters and therefore chose a population size of eight control laws per generation (using a formula recommended by the author of CMA-ES). This population size is intended to be robust and therefore applicable to a wide range of parameter spaces [32]. We aimed for four generations of optimization per participant, based on pilot tests suggesting that convergence was typically achieved in four generations or fewer. Three subjects experienced more or fewer generations of optimization. The optimized control law was defined by the final calculated (untested) mean parameter values. Colleagues who use CMA-ES tend to use small population sizes (16) even in high dimensional optimization. CMA-ES aggregates information across generations to compensate for the small sample size. We will test whether this approach will work in our context. Theoretical results on CMA-ES scaling with dimensionality suggest a logarithmic scaling law, so we are hopeful. Proposed Research: Understanding Co-Optimization The optimization problem we are solving is complex because on each trial the human is also learning what the co-robot is doing, adapting to it, and in general optimizing their behavior for an optimization criterion that is different from the co-robot s criterion. In our preliminary work we made some simplifying assumptions to tackle this problem. We kept the assistance fixed during any trial, rather than continually varying the assistance on each foot step (0.5s) or time step (2ms). We assumed the human would rapidly learn and adapt to the novel assistance, and treated the human as a stationary system on each trial. On each trial the co-robot applied a new policy, and the human learned what that policy was, and created their own policy in response. From the point of view of the co-robot the system being optimized is non-stationary across trials. We refer to this as co-optimization. When does this process converge to desirable optima? What undesired transient optimization behaviors need to be detected and reduced or eliminated? How does this limit or alter what the co-robot can do? How do we handle the more complex situation where the co-robot changes its policy more rapidly, such as on every time step? There are many ways to approach these questions. Perhaps the most general is to formulate a model of the interaction as a incomplete information repeated two player game where each player is trying to learn the dynamics and optimization criteria of the other player [9, 29]. In this case the players are not fully competing or cooperating. The optimization criteria may be similar or unrelated. We know that even simple two player games with learning can generate arbitrarily complex transients and even chaotic dynamics [66], so part of the proposed work will be trying to understand when our learning algorithms will converge, and have satisfactory transients. A related formulation is Multiagent Reinforcement Learning (MARL) [12] in which multiple interacting agents simultaneously attempt to learn to improve their policies. There are a number of simplifying assumptions that can make analysis of the game or reinforcement learning context easier. The most aggressive is to assume the human is a fixed dynamic system, and does not adapt to the co-robot s policy. A less aggressive assumption is to assume the human s reaction to the co-robot s change in policy will be linear in policy parameterizations for the co-robot and the human policies. The human may be globally complex with learning and optimization, but locally the resulting adaptation is a linear function of the co-robot s change in behavior. This approach can be used with more complex model structures for the human s adaptation, such as a quadratic function, or some form of neural network. Another reasonable assumption is that the human is trying to keep the system behavior invariant, matching some desired behavior, similar to model reference adaptive control. Another simplifying assumption is that the human s optimization criterion is the same as the co-robot s, or is known. We propose to explore these different assumptions in the game and multiagent reinforcement learning contexts, searching for a model that can fit and then predict the human 7

8 adaptation we see in our experiments. There is related work in the areas of adaptive control, in which human pilots are driving or flying adaptive vehicles and driver assistance, in which automobiles are tuning engine and vehicle behavior while drivers are responding to those changes. Proposed Research: The Muscle-Level Basis Of Co-Robotics We would like to understand human adaptation to physical co-robotic interfaces. For example, lower-limb exoskeletons often produce odd adaptations in humans. This section describes our planned work on musclelevel mechanics and energetics, estimated in data-driven simulations of exoskeleton-assisted walking, which can potentially explain why. The next section explains our planned work in muscle mechanochemistry, in which we try to understand how current models of molecular mechanical behavior can improve our modeling of how humans adapt to physical co-robotic interfaces. We plan to use Hill-type phenomenological ( curve-fit ) muscle models to explain why users adapt to and optimize physical co-robots the way they do. To describe how, we will describe some of our preliminary work in this area [39]. Using data from preliminary experiments, we performed electromyography-driven forward dynamic simulations of a musculoskeletal model to explore how changes in exoskeleton augmentation affected plantarflexor muscle-tendon mechanics, particularly for the soleus. We used a model of muscle energy consumption to estimate individual muscle and whole-body metabolic rate. As average exoskeleton torque was increased, while no net exoskeleton work was provided, soleus muscle fibers performed more positive mechanical work and experienced greater lengthening and shortening throughout the stance phase of gait. There was a 90% correlation between simulated estimates of average changes in whole body metabolic energy consumption and experimental measurements, providing confidence in our model estimates. Our simulation results suggest that the main benefits of the series tendon are to reduce positive work done by the muscle fibers by storing and returning energy elastically and to reduce the total excursion of the muscle fibers throughout stance. So far we have used a generic lower-body musculoskeletal model adapted from a previously published model (OpenSim musculoskeletal modeling software (v3.1) [22, 5]). The model includes the pelvis and both legs, with segments and degrees of freedom as defined in [5]. Of the original 35 lower-limb muscles in the model, we only include the muscles for which we had electromyographic data: lateral gastrocnemius, medial gastrocnemius, soleus, tibialis anterior, vastus medialis, rectus femoris, and biceps femoris long head. Due to the fact that the electromyography-driven approach prescribes joint kinematics of the model, omitting muscles did not invalidate our simulations. Muscle parameters were based on measurements of 21 cadavers [82]. The raw electromyographic data was first high-pass filtered (20 Hz), full-wave rectified, and low-pass filtered (6 Hz). It was then normalized to maximum muscle activity measured during normal walking, scaled, and delayed. We used the results of the electromyography-driven simulations to estimate the energy consumed by each muscle using a modified version of Umberger s muscle metabolics model [79, 78, 77]. From this type of preliminary study, we have come to believe that usefully interacting with biological muscles and tendons, via an external device, is much more difficult than expected. Tendon stiffness and other muscle-tendon properties seem to be tuned such that the biological ankle operates efficiently. Subtle disturbances to the system can result in undesirable changes in coordination patterns and whole body metabolic energy consumption. For example, providing increasing amounts of average exoskeleton torque, while maintaining zero net exoskeleton work, detuned soleus muscle-tendon interactions without compensating for reduced performance. Disrupted muscle-tendon interactions have similarly been observed in human hopping with ankle exoskeletons [64]. Assistive and augmentation devices should be designed and controlled to compensate for any compromised performance or functioning of biological mechanisms. Analyses similar to those discussed above can be used to help understand how different co-robot behaviors affect muscle-level mechanics, and provide insights into why certain device behaviors are more effective than others. For instance, torque support with a device can be an effective augmentation strategy [18], but subtleties of how the external torques are applied and how the device 8

9 Figure 4: Left: Comparison of simulated (predicted) inverse-dynamics-derived ( measured ) ankle joint mechanics. Top row: Simulated muscle-generated ankle joint moments compared to inverse-dynamics-derived ankle joint moments. Bottom row: Simulated muscle-generated ankle joint powers compared to inverse-dynamicsderived ankle joint powers. Simulated muscle-generated joint moments and powers were calculated by summing the individual contributions of the exoskeleton-side lateral gastrocnemius, medial gastrocnemius, soleus, and tibialis anterior. Each line is the subject mean (N = 8) for a given condition. Conditions with increasing average exoskeleton torque are shown in green. Conditions with increasing net exoskeleton work rate are shown in purple. Darker colors indicate higher values. Normal walking, without an exoskeleton, is shown by the gray dashed line. All values were normalized to body mass. For reference, exoskeleton torque trajectories for each of the different conditions can be found in Figure 4 in [38]. Right: A cartoon of the two possible crossbridge cycles. Green arrows show rapid transitions. Arrows with other colors show proposed controlled transitions. The gold arrow shows where calcium ions (Ca++) control the attachment of myosin heads (M, blue block) to the actin (A, yellow block) thin filaments. The red arrow shows where low crossbridge (XB) strain permits transitions to the UNLOCKED state. The brown arrow indicates where crossbridge strain that exceeds a mechanical limit causes detachment and slipping. The faint blue cycle shows the traditional dominant pathway when the muscle is shortening. The faint green cycle shows the proposed dominant pathway when the muscle is lengthening, a cycle of detachment and attachment that leads to a viscous-like resistance to changes in muscle length, without using ATP. In the loaded isometric case the crossbridges remain in the locked state. The purple line is the lever arm of the myosin head, and the red line is a hypothesized elastic element. interacts with the biological system greatly impact coordination patterns and overall effectiveness. The modeling approaches used in this study can be applied to a wide array of human motions. The results suggest that, given a coordination pattern, via measured muscle activity and joint kinematics, it is possible to generate reasonable estimates of energy use and other physiological parameters. In the future it may be possible to invert the process. Based on what we know about the mechanics and energetics of individual muscles, we can try to generate a set of desirable coordination patterns. It may even be possible to prescribe co-robot behaviors that elicit these desirable changes in coordination. Although the results produced from this approach seem reasonable, there are a number of limitations. If the parameters used in the model were inaccurate, this could have led to invalid estimates of muscle mechanics and energetics. The parameters we used are, however, comparable to previously published work [4, 5] which are based on cadaver studies [82]. Furthermore, to validate the approach, we compared muscle-generated ankle joint moments and powers to inverse-dynamics-derived ankle joint moments and powers (Figure 4). We optimized parameters to reduce the root-mean-square error between the two and an in-depth sensitivity analysis shows that the qualitative trends are robust to model parameters. We are also limited by the number of muscles we can measure. Results from this study still produced reasonable estimates of metabolic energy consumption. Including more muscles in future experiments would make these analyses more complete. 9

10 Muscle-generated ankle joint mechanics did not perfectly match inverse-dynamics-derived ankle joint mechanics. Most trends were consistent across the two methods. Results from inverse dynamics, however, suggested that total exoskeleton-side positive ankle joint work decreased as average exoskeleton torque increased, while results from the forward simulations suggested that total exoskeleton-side positive ankle joint work remained relatively unchanged. This inconsistency could have implications for our understanding of why contralateral-limb knee mechanics and vastus metabolic energy consumption were affected by torque applied at the exoskeleton-side ankle joint. These results illustrate the importance of knowing the limitations and assumptions inherent in a model and taking these into consideration when analyzing and interpreting outputs from a model. To account for these limitations, we minimized inconsistencies between inverse-dynamics-derived and muscle-generate joint mechanics by optimizing those model parameters in which we had the least confidence. We believe this type of work is the first data-driven investigation of changes in muscle mechanics during walking with an exoskeleton using reasonably accurate muscle and tendon mechanical models. We were able to explain experimentally-observed changes in coordination patterns and metabolic energy consumption. Models without muscles and tendons would not have been able to capture these effects. We expect the results from this type of study applied to more complex co-robots and the entire human body to lead to greater insight into the functioning of muscle-tendon units and to guide the design and control of co-robots that interact effectively with these biological mechanisms. We will also use our co-robot testbeds for perturbation studies to evaluate our understanding of how the musculoskeletal system works. Proposed Research: The Molecular Basis Of Co-Robotics In a Hill-type phenomenological muscle model, curve-fits are used to justify the claim that reducing the total excursion of the muscle fibers throughout stance results in less energy usage. We believe that modern Huxleytype models that take into account the molecular mechanochemistry can now predict this effect more accurately (Figure 4) [48, 56, 57, 61, 17, 55, 70, 49, 13]. Furthermore, it is becoming clear that muscle lengthening while generating force is very different than muscle shortening while generating force. We believe that Huxley-type models can explain this effect as well. The research described in this section attempts to bridge muscle molecular mechanisms and human-co-robot physical interaction, and stems from our attempts to more accurately model energy use, muscle force generation, muscle stiffness, and muscle resistance to motion. The key issue is that muscle uses much less energy to generate the same force when it holds a position or resists lengthening as compared to generating the same force while shortening. This is not true of electric motors. Energy use is proportional to the absolute value of motor current, and thus torque. Hydraulic energy use is proportional to the absolute value of joint velocity, in addition to a large constant term due to internal leakage. How does muscle not burn a lot of energy under load when the muscle (not including the tendon) is staying at the same length (isometric) or lengthening under load (doing negative work)? The crossbridges are bathed in stored energy (Adenosine Triphosphate, ATP) all the time, so it is not that the energy source is removed. If ATP hydrolysis is used for crossbridge detachment, why doesn t any form of crossbridge cycling (which has to happen in muscle lengthening under load) burn energy [60]? It has been recently confirmed that skeletal muscle myosin shares with other myosin isoforms a catch-bond property. The myosin head is much less likely to detach from the actin filament if the crossbridge is stretched. In addition, it has been hypothesized that there are rapid crossbridge detachment and reattachment processes that do not require the use of stored energy in the form of Adenosine Triphosphate (ATP). These processes allow a crossbridge to translate or jump along the actin filament to continue to resist a muscle being lengthened. The catch-bond and jump properties allow muscle to macroscopically act as a brake with little use of energy when lengthening, as well as like a spring for small perturbations. These properties also imply that large scale use of ATP only occurs when the muscle is shortening, and only when the muscle is doing positive work as a unidirectional motor. We will augment the Hill-type models discussed in the previous section with the effects predicted by modern 10

11 Huxley-type models [48, 56, 57, 61, 17, 55, 70, 49, 13]. We will use whole limb data from our experiments to curve-fit the augmented models. We believe that the improved augmented models will do a better job explaining what a human user is adapting towards during our experiments. We believe the view in biomechanics and neuroscience of muscle as a spring with fixed length-tension and force-velocity curves selected by activation is questionable. While tissues other than crossbridges may have fixed (but nonlinear) spring-like properties and generate the same force when returned to the same position and velocity, the macroscopic behavior of a set of crossbridges can be different depending on the history of the muscle activation and load, in addition to the current muscle length and velocity. Understanding the mechanochemistry of muscle allows us to go beyond current phenomenological (curve-fit) Hill-type muscle models to better explain metabolic costs, forcevelocity relationships, short range stiffness, force enhancement, catch phenomena, and other muscle properties and nonlinearities, and make more accurate muscle models to simulate and predict behavior, as well as provide therapy, rehabilitation, and physical assistance and augmentation. Evaluation We will pursue an innovative approach to evaluation, building on our prior work. Instead of prematurely committing to a particular co-robot design, we propose building two physical co-robot emulators for lowerbody co-robots (Figure 1). The first emulator uses very powerful and fast benchtop actuators to physically simulate a wide range of co-robot designs, and allows us to explore and evaluate many proposed co-robots and co-robot control schemes, before investing the effort and resources to actually build each one. The second co-robot emulator uses a similar lower-body structure, but uses smaller motors and power source (batteries) in a backpack, allowing realistic tests on outdoor irregular terrain. The laboratory evaluation system is being designed and constructed. We request funding for Humotech, a commercial spinoff from the Collins lab, to design and build the portable co-robot emulator. These testbeds will allow us to explore a range of co-robot physical interaction customization strategies in a range of behaviors including level and inclined and unloaded and loaded walking and moderate speed running over irregular terrain, as well as scrambling over boulders, climbing and descending steep slopes and rock faces, and walking, running, and jumping across stepping stones and pole tops. We will build on evaluation metrics we have used, such as metabolic cost, amount of muscle electrical activity (EMG), magnitude of co-robot forces on the user, magnitude of user internal forces and torques, features (such as asymmetry) of user kinematic patterns, and user self reports. Mechanical Design: The co-robot testbeds will be designed based on principles and techniques we have developed and experimentally validated over the past five years. Ankle end-effectors will be refined versions of our current successful devices. Knee end-effectors will be refined versions of current prototypes, redesigned for lower mass to allow running. Hip end-effectors are planned to include a revolute flexion-extension joint, which allows direct sensing of joint angle and reduces reaction forces applied to the users back. The hip joint will also include a passive flexure for ad-abduction, which will allow the subject to move their foot mediolaterally for balance but not add substantially to worn mass. The torso frame will contact the pelvis and shoulders, providing a large moment arm that results in low contact forces for a given applied torque. Contacting the back and pelvis in this way also results in normal contact forces, as opposed to shear contact forces, which results in a stiffer and more comfortable interface. Widely spacing the contact points further reduces slop at the interface, since it reduces the angular displacement for a given linear displacement at the contact points. Our specific design goals are as follows. Design goals; speed and bandwidth: High exoskeleton speed is essential to avoid interference with natural motions of the limbs, especially during leg swing. The maximum speed of our current ankle exoskeleton is 16rad/s. Closed-loop torque bandwidth of the current laboratory testbed with benchtop actuation (38Hz) is about three times greater than that of human muscle and exceeds values of all other exoskeletons capable of generating similar magnitudes of torque (due to remotizing the actuation). A design goal for the outdoor testbed is greater than 2 revolutions/second maximum velocity at each joint, in order to support rapid error responses 11

12 and moderate speed running [63]. Design goals: maximum torques: We will seek to maximize co-robot joint torques given a weight budget of about 1 kg/joint. Based on our prior augmentation experiments[16, 18, 38] we estimate that optimal augmentation will be achieved with exoskeleton torques that are about half the values observed during unassisted locomotion (50-100Nm depending on the joint), so this is our minimum design goal. Peak torque in excess of these values will be useful. Our current treadmill-based ankle exoskeleton is more powerful than this, with a maximum torque of 120Nm. Design goals: weight: Our current ankle exoskeleton weighs 0.8kg. Weight limits for the full indoor system are as follows: The entire worn portion of the system is expected to weigh 6 kg. Each foot-ankle-shank section is expected to weigh 0.75 kg, based on current, established hardware. Each knee-thigh section is expected to weigh 1.0 kg, based on a current prototype. Each hip-torso section is expected to weigh 1.25 kg (2.5 kg for both hips and the torso), extrapolating from ankle and knee hardware. Low mass is the result of off-board power and control together with careful mechanical design. Design goals: range of motion: The range of motion of each exoskeleton emulator joint will exceed values observed during walking, running and sprint running. We will include additional range in hip flexion, knee flexion and ankle plantarflexion, which will allow for changes in kinematics following adaptation to use of the exoskeleton. Hip extension, knee extension and ankle dorsiflexion will be limited to the ranges observed during walking and running, which correspond to natural limits, to avoid hyperextension. All limits will correspond to hard stops on the device, after which further torque development that could injure the user is not possible. Design goals: sensing: All joints will be instrumented with high-resolution encoders to measure joint angle. All Bowden cable termination points will be instrumented with strain gages to measure torque. We have developed reliable strain gage instrumentation approaches that allow torque to be measured at 500 Hz with less than 1% measurement error. Research Plan Our experimental work provides whole-limb and whole-body data that informs our modeling work and our exploration of alternative optimization approaches, as well as helps us understand how the human motor control system works. Better models and model-driven optimization improve our experimental results. Our research plan focuses on this cycle. This rearch involves one postdoc and two students. We expect the postdoc to take a lead role in coordinating the project and performing the evaluation. We expect one student to focus on optimization and co-optimization issues (involving applied mathematics and optimal control) and one student to focus on muscle modeling (involving mechanochemisty, physiology, and biomechanics). Year 1: Continue experiments with existing laboratory testbed (bilateral ankles). Improve optimization approach in simulation using current Hill-type muscle model. Develop a library-based interaction policy library approach, with online policy tuning using current Hill-type muscle model. Build full lower-body laboratory testbed and initiate experiments. Build outdoor bilateral ankle testbed and initiate experiments. Develop co-optimization theory, and extensively test proposed algorithms in simulation using current Hill-type muscle model. Improve current Hill-type muscle model. Develop molecular and sarcomere-level muscle modeling. Begin evaluation. Year 2: Year 2. Continue full lower-body laboratory testbed experiments with improved optimization approach and with policy tuning. Improve optimization approach in simulation using improved Hill-type muscle model. Improve library-based interaction policy library approach and online policy tuning using improved Hilltype muscle model. Continue outdoor bilateral ankle testbed experiments with improved optimization approach and with policy tuning. Build full lower-body outdoor testbed and initiate experiments. Continue to develop co-optimization theory, and extensively test proposed algorithms in simulation using improved Hill-type muscle model. Extend molecular and sarcomere-level muscle modeling to whole muscle and lower-body modeling (Huxley-type model). Continue evaluation. 12