EVOLVING GOALKEEPER BEHAVIOURS FOR SIMULATED SOCCER COMPETITION

Transcription

1 Proceeings of the 3r IASTED International Conference on Artificial Intelligence an Applications (AIA 2003 September 8-0, 2003, Benalmáena, Spain EVOLVING GOALKEEPER BEHAVIOURS FOR SIMULATED SOCCER COMPETITION Christopher Lazarus an Huosheng Hu Department of Computer Science, University of Essex, Wivenhoe Park, Colchester CO 3SQ, Unite Kingom Abstract This paper escribes our approach on evolving a robot controller to generate a set of goalkeeper behaviours in the Simulation League of the RoboCup competition. The goalkeeper agent is a software construct that performs in a simulate environment provie by the Soccer Server. The soccer server is a noisy environment provie as a testing groun for various artificial intelligence techniques use by ifferent research teams aroun the worl. Our experiments were set out to etermine the conitions that nee to be met for an evolve goalkeeper agent to perform aequately in a efensive situation. A framework was evelope to test the agent s performance in the simulator. The results show that our approach was able to prouce a robust robot controller. Keywors: Simulate Soccer, Evolutionary Robotics, Genetic Programming.. Introuction Robot controller evelopment is an area that traitionally lies uner the omain of human programming. This area is an important part of robotic research an therefore has been the most active area in robotics. Traitional robotic research has been to break own the require behaviours of a robot into smaller manageable parts an architectures are built to make them work. On the other han, this is often accomplishe as a result of a self-organizing process in the Evolutionary Robotics (ER approach [3]. It is well known that natural selection combine with genetics has the power of fining out ingenious solutions to problems that are similar to the one facing a robot controller esigner. Researchers han coe solutions to robotic control an navigation problem. Successful solutions ten to be complex, rigi an highly customise. During recent years, researchers began to use techniques erive from Evolutionary Computation (EC for eveloping robot controllers. The major benefits in using this technique are automatic programming an aaptability of solutions evelope. Programming a robot to perform various tasks an ultimately relate to a single goal is not trivial. Our efforts focus on evolving behaviours for a goalkeeper agent by aopting the Genetic Programming (GP technique an the focus of the research was on the esign of a set of specifications for the esire behaviours. These behaviours emerge ue to evolutionary pressure set out by the esigner []. The work presente here is base on our previous work on evolving navigation an obstacle avoiance behaviours for the wall-following problem. In that work, the evolution of a robot s behaviours involves the optimisation of a single objective function []. However, our current work involves the optimisation of multiple objectives [6]. There have been many attempts by researchers to evelop an improve ways on robot evolutions. In many ways they are able to show that their solutions can be similar if not better than han-coe solution, i.e. the traitional robot navigation solution. These works inclue evolution of both simulate an real robots. The RoboCup omain Error! Reference source not foun. in particular is a rich omain for the exploration of these techniques. It gives researchers a platform where they can compare the performance of their techniques in a like environment [7]. Given that more than one robot is involve in a soccer match, RoboCup also gives researchers a platform on which to explore multi-robot cooperation algorithms. Accoringly, several teams have use GP in evolving their soccer team [8] an []. Lunberg i a comparison of these works [9]. This paper is organise as follows. We escribe our experiments in more etail in section 2 incluing the methoology that we chose an shows in a conceptual manner the architecture of the system that we have evelope. In section 3, the implementation of our architecture is escribe. The experimental ata are presente an analyse in section. Finally, we show what our experiments have uncovere an what further work coul be one. In the Simulation League, the soccer server was esigne to be a noisy environment. Therefore any solution pose by researchers nees to be able to hanle the uncertainty of sensory ata

2 2. Methoology The focus of our experiments is to fin out the effects of using multiple objective fitness functions in our GP implementation. To accomplish this, we have evise a training groun situation to test the evolve behaviours of our goalkeeper. The architecture consists of a soccer server, a coach trainer agent, a han-coe attacker agent an an evolving goalkeeper agent. The soccer server Error! Reference source not foun. acts as a server to all of our agent clients. It accepts action commans an sens back sensory information. In other wors, the soccer server runs the simulation. The coach agent performs the synchronisation between all of the agents. It oes this by relaying messages to let the agents know when a trial begins an ens. It also resets the positions of the agents an the ball. The han-coe attacker agent is a simple agent that kicks the ball towars the centre of the goal. The goalkeeper agent is the agent that has the capability to evolve ue to the GP system that generates its control program. GP System problem omain is not essential. The focus of the problem shifts from the careful esign of the behaviours to the esigning of specifications that encourages emergence of the esire behaviours []. In this case, specifications can be equate as the fitness criterion of each iniviual within the GP population. No One iniviual Initialise population evaluation Genetic operators Maximum generation? Yes Goalkeeper Agent Best Controller Coach Agent One Iniviual Actions Sensors Figure System Configuration Goalkeeper Agent Soccer Server Han-coe Attacker Agent Our control experiment uses a ranom goalkeeper agent that has exactly the same capability as our GP goalkeeper. The only ifference is that the ranom goalkeeper agent sens ranomly generate commans to the soccer server instea of being generate by GP. Nevertheless, its fitness is still evaluate uring the experiment. We run both ranom an evolving goalkeeper experiments to collect the ata. We compare both of the goalkeeper s performance to measure the effectiveness of the GP implementation. In esigning our functions an terminals, we note that GP may give surprising results even with a simple set of functions an terminals although it is generally accepte that GP is a robust technique [0]. Therefore, if the performance of the GP algorithm is not quite what we expect then more complex functions an terminals may be ae to the gene pool. Nevertheless, one of the attractions of using GP is that etaile knowlege of the Figure 2 GP System Therefore, we see the nee for correctly representing the problem omain in the fitness functions. In our previous work we have use a single fitness function for evaluating a robot s navigation solution []. In our current work, we use multiple fitness functions. A goalkeeper has a single overriing objective, which is to stop any goal from being score into its own net. Nevertheless, to achieve this, there is a set of other objectives that nee to be met. The first objective concerns navigation, another is localisation (position must be kept near the centre of the goal, another is blocking the ball trajectory, the ability to kick the ball out of the goal area an lastly to catch the ball when possible. Therefore the aim of our optimisation is to transform the vector objectives to a scalar version an then procee with the assignment of fitness values to each iniviual. 3. Implementation The GP system we use in our experiments is lil-gp []. We also use the client networking library libsclient [2]. We evelope a coach agent to provie synchronisation an monitoring uring the experiment run. The coach agent resets the experiment setting before the start of each trial case. This involves placing the goalkeeper, ball an attacker in their respective starting positions. Once in place, the ball is kicke by the attacker in a straight line irectly towars the goal. There are 0 ifferent ball placements for each evaluation (see Figure 3. In short each iniviual in the GP population representing the goalkeeper is teste on its performance in efening the goal from 0 ifferent ballstarting positions. The attacking area in front of the goal is ivie equally into 0 sectors. At the start of each

3 experiment, the 0 starting positions for the attacker is ranomly generate within each of the 0 attacking sectors. Each attacker starting position is then teste sequentially in a clockwise orer for each iniviual in the population. This variation in the attacker s starting position shoul enable a robust controller to be evolve. The evolve behaviours are not biase towars a particular attacking irection only. We evaluate each iniviual by parsing the expression trees an sening a comman every 0ms to the soccer server. Normally this woul be set to 00ms but we have reuce it to 0ms to spee up the simulation 2. Only one comman is allowe per 0ms. Figure 3 A segment of the soccer fiel showing a goalkeeper at location 0, 0 with the attacking area ivie equally into 0 sectors. Figure is not accoring to scale. Evaluation time The length of timesteps allowe for an attacker to attack an score a goal has a irect impact on the length of time each evaluation takes. For the purpose of the experiments each evaluation takes 0 timesteps. Each timestep correspons to a set perio of 0ms. We allow the attacker 0 timesteps per trial to give it enough time to try to score a goal. measure s9 s8 s7 s6 s s s3 s2 s s0 Goal keeper At t acki ng Sect ors = s0... s9 The fitness evaluation we aopte is calle the Weighte Sum Approach 2 Only one comman is allowe per 00ms. The rules of the competition stipulate that no flooing of the soccer server is allowe. If this rule is violate, the soccer server may run only one action comman arbitrarily in a single time step an ignore the rest. f = n w f i = i i where 0.0 f. 0. f enotes the overall fitness of an iniviual, n enotes the number of fitness measures, wi enotes the weights an f i enotes the objectives to be optimise [3]. There are fitness cases that influence the evolution of esire behaviours. cases f = Ball saving f = Localisation 2 f = Ball locate 3 f = Movements f = Positioning Max Generation 0 Population Type Generational Size 00 Initialisation Half an Half Selection Type Tournament with size 7 Genetic Operators Crossover rate 0.9 Mutation Rate 0. Function Set iflte, a, sub, mult, iv, sin, cos, catch, ash, kick, turn, put, ADF0 Terminal Set play_ir, play_pos, ist_to_ball, ir_to_ball, ir_to_ogoal, ist_to_ogoal, ist_to_opp, ist_to_opp, ran, get0, get, ARG0, ARG Table GP Parameter Settings The rationales for the choice of the fitness measures above are that we want the evolve goalkeeper to be mobile but moving towars the ball an intercepting it. The movements an positioning fitness cases reflect this esire. f an f might seem to overlap in functionality but without fitness f, the movement of the ball instea of the goalkeeper might be the cause for the proximity of the goalkeeper to the ball. Therefore, by encouraging selection of iniviuals that are mobile ensures that there are cases where the proximity of a goalkeeper to a ball is cause by the goalkeeper s own movements. We gave fitness f a higher weight because of similar reasons. Nevertheless, fitness f an f are ineffective if they on t result in a ball saving event. So we give a higher weight for behaviours that resulte in a ball saving event

4 case is given by the equation g f = t where g is the goal score by an attacker an tmax is the maximum trial case per evaluation f is neee to measure the goalkeeper s 2 ability to estimate its own location within the playing fiel. If both initial an ening position of the goalkeeper max within each trial is estimate then localisation is eeme as successful an fitness f is allocate. 2 f measures the goalkeeper s ability to locate 3 the ball. This is one by checking whether the goalkeeper is able to estimate the istance between itself an the ball at the start an en of a trial. The goalkeeper is only able to o this if the ball is in its fiel of vision. Therefore this fitness can also be equate with the ability of the goalkeeper in turning towars the ball. f is as follows where f if p s is equal to p e then else no movement is etecte so f 2 = 0 movement etecte so f 2 = ps is the position of goalkeeper at the start of an evaluation, an pe is the position of goalkeeper at the en of an evaluation f is given below. This fitness measure uses a logarithmic scale function of the istance between the goalkeeper an the ball at the start of an evaluation an also at the en of an evaluation. This has the effect of ensuring that the relationship between the istance between the goalkeeper an the ball is logarithmic. This ensures that the fitness ecreases following a logarithmic curve as the istance between the ball an the goalkeeper becomes larger. f where if ( else e s set f 3 to 0 set f 3 to is less than 0 then e s s is the istance between goalkeeper an ball at the start of an evaluation greater than. e is the istance between goalkeeper an ball at the en of an evaluation greater than Termination-criteria Termination of the GP run is set to the number of maximum generation preetermine at the beginning of the GP run.. Experimental Results an Analysis Table 2 presents all experiments that have been carrie out in this research an the results are collecte an summarise in Figure. Experi Behaviour Memo ADF Weights ment generate by ry Ranom N/A N/A Equal 2 Ranom N/A N/A Lexicographic 3 GP Yes No Equal GP Yes No Lexicographic GP No Yes Equal 6 GP No Yes Lexicographic 7 GP Yes Yes Equal 8 GP Yes Yes Lexicographic 9 GP Yes Yes SQRT (Lex. Table 2 Experiment runs The first two experiments, where the goalkeeper s behaviour was generate ranomly, are use as a benchmark for subsequent experiments. The starting position for the goalkeeper is set at location 0, 0 on the playing fiel. Three sets of weights were use uring the experiments. Weights Equal Lexicographic Square root (Lexicographic Table 3 Weights use uring the experiments FITNES S COMPARISON OF WEIGHTS 2 3 WEIGHTS Equal Lexicographic Normalise square root of Lexicographic Figure The fitness istribution when lexicographic weights are normalise

5 Run Mean Stanarise Generations Exp. Exp. 2 Exp. 3 Exp. Exp. Exp. 6 Exp. 7 Exp. 8 Exp. 9 Figure Comparison of mean population fitness Best of Run Iniviual Stanarise Generations Exp. Exp. 2 Exp. 3 Exp. Exp. Exp. 6 Exp. 7 Exp. 8 Exp. 9 Figure 6 Comparison of iniviual fitness As shown in Table 3, one set of weights is esigne to give equal importance to all of the fitness measures. The secon set of weights is esigne to bias the importance of the fitness measures in a lexicographical manner in terms of fitness. The thir set of weights is erive from normalise square roots of the lexicographic weights. The thir set of weights has a smoother istribution compare to the secon set. This is shown in Figure. This allows for the fitness measures to be more uniformly istribute across the fitness range but woul still be skewe accoring to the fitness measures ranking. All of the experimental runs base on lexicographic weights have lower overall fitness when compare with runs that are base on equal weights. Lexicographic weights biase the fitness measurement too much towars fitness which is ball saving. This results in GP not being able to capitalise on the contributions mae by the other fitness measures. This also shows that even though ball saving is

6 the main objective, ball saving is not enough on its own to allow GP to explore potential solutions. By specifying that all of the fitness measures have equal weights, we see that the fitness is istinctively higher compare to using lexicographic weights. We can also see this assertion by comparing both of the ranom runs. The ranom run that uses equal weights has higher fitness compare to the ranom run that uses lexicographic weights Experiments 2, 3, an were run to observe whether the GP performance woul be enhance by the aition of either memory or ADF. The results show that the GP with ADF has higher fitness than the GP with memory when equal weights are use. In contrasts, the GP with memory has higher fitness than the GP with ADF when lexicographic weights are use. Looking at both the equal weights an the lexicographic weights groups, we can see that in both groups, the highest performing GP is the one that implements both memory an ADF in their architecture. The aition of either ADF or memory in isolation oes improve the performance of the GP but its performance increases further if both of them are implemente together. When we compare performances of best of run iniviuals in contrasts to populations, we start to see that the istinction cause by ifferent weights set becomes less obvious. Experiments that use the lexicographic weights, has best of run iniviual nearing the performances from the experiments that uses equal weights. An aitional experiment (experiment 9 that uses the normalise lexicographic weights was run to observe the effects of smoothing the istribution of the fitness measures. The population fitness for this experiment shows a marke improvement. However its population fitness is still below the population fitness for experiments that uses equal weights.. Conclusions an Future Work The selection of weights in the Weighte Sum Approach as we have shown in this paper is important to the performance of the GP implementation. In this approach most weight selection problem are solve heuristically. Consiering the simulation results, each of the fitness measure is equally important. Attempts to bias the fitness measures by ajusting the weights shows lower overall population fitness. Nevertheless, the fittest iniviual within the population where the weights are not equally istribute still manages to be fitness-competitive to the fittest iniviual using equal weights. Other multiobjective optimisation methos are escribe by [3] an [6]. The aition of memory an ADF as part of an agent s memory an an iniviual s program tree respectively for the uration of the run increases the GP s performance. This correspons with the increase in ifficulty of the task that the goalkeeper agent is given. The overall results show that the best iniviual reaches a plateau at approximately two thirs of the fitness range. This shows that although GP manages to evolve a goo set of behaviours, they are still short of optimum fitness. The next stage of this work will be to investigate on ways to improve the performance of the evolve goalkeeper further. Acknowlegements: We wish to thank Prof. Ewar Tsang an Dr. John For for their useful inputs an avices uring this research. References [] The RoboCup Feeration, "RoboCup Official Site", [2] I. Noa, "Soccer Server System", [3] S. Nolfi, "Evolutionary Robotics: Exploiting the full power of self-organization," Connection Science, vol. 0, pp , 998. [] D. Anre an A. Teller, "Evolving Team Darwin Unite," in RoboCup-98: Robot Soccer Worl Cup II, M. Asaa, E.: Springer-Verlag, 998. [] C. Lazarus an H. Hu, "Using Genetic Programming to Evolve Robot Behaviours," Proceeings of the 3 r British Workshop on Towars Intelligent Mobile Robots (TIMR '0, Manchester, 200. [6] E. Zitzler, K. Deb, an L. Thiele, "Comparison of multi-objective evolutionary algorithms: Empirical results (Revise Version Technical Report 70," Computer Engineering an Communication Networks Lab (TIK, Swiss Feeral Institute of Technology (ETH Zurich, Gloriastrasse 3, CH-8092 Zurich 999. [7] G. Aorni, S. Cagnoni, an M. Moronini, "Genetic Programming of a Goal-Keeper Control Strategy for the RoboCup Mile Size Competition," presente at Genetic Programming: Secon European Workshop, EuroGP'99, Goteberg, Sween, 999. [8] S. Luke, C. Hohn, J. Farris, G. Jackson, an J. Henler, "Co-Evolving Soccer Softbot Team Coorination with Genetic Programming," presente at Proceeings of The First International Workshop on RoboCup, IJCAI-97, Nagoya, Japan, 997. [9] J. Lunberg, "Survey over Genetic Programming Approaches to RoboCup", pers/999/int7/lunberg.pf, 2003 [0] W. Banzhaf, P. Norin, R. E. Keller, an F. D. Francone, "Genetic Programming An Introuction," Morgan Kaufmann Publishers, Inc., 998, pp. 2. [] B. Punch an D. Zongker, "lil-gp Genetic Programming System", [2] I. Noa, "libsclient", RoboCup/Libs/libsclient.0.tar.gz, 2003 [3] C. M. Fonseca an P. J. Fleming, "Multi-objective Optimisation," in Evolutionary Computations 2, T. Bäck, D. Fogel, B, an Z. Michalewicz, Es. UK: IOP Publishing Lt.,