Master s Project in Computer Science April Development of a High Level Language Based on Rules for the RoboCup Soccer Simulator

Master s Project in Computer Science April 2006 Development of a High Level Language Based on Rules for the RoboCup Soccer Simulator José Ignacio Núñez Varela jnunez@cs.pitt.edu Department of Computer Science University of Pittsburgh

ABSTRACT The RoboCup Simulation League was created around the mid-90 s, it was the first competition focusing on the new proposed problem for A.I. and Robotics, Soccer. The main tool of the League is the use of the RoboCup soccerserver, that simulates the actions of players in a virtual environment. The server provides an excellent way of working on aspects of soccer that does not require the knowledge of mechanics and/or electronics. Still, to work with the server requires the knowledge of specific things such as how the server works, which variables are available, how can we synchronize with the server, etc. Things that some people might want skip and go directly to the problem of creating teams of players, deal with the strategies and the way players behave in the field. This report describes the development and implementation of a high level language, based on rules, for the RoboCup soccerserver. The main objective is to provide directives, in the form of conditions and commands, to create players, and ultimately a team, in an easy and fast way without having to know or understand all the specific details found in the simulation server. With just two or three rules users can create a player and start thinking in ways to improve its performance and behavior in the game. The conditions and commands available allow to create a large number of rules for almost every possible situation in the game. One of the ultimate goals of this project is to be used as an educational platform where students can explore the development of strategies and behaviors by creating team of players and playing against each other. 2 2

1. INTRODUCTION Soccer is probably the best-known sport in the world, millions of people around the world surely have played soccer or at least know how to play it. To play soccer all you need is a ball and two teams of players. The rules are relatively simple, in order to win you need to score more goals than the other team, using only your feet for kicking the ball toward the opponent goal. Of course, the official rules are much more complicated than just kicking the ball, but still the objective is easy to understand even for kids. Analyzing in more detail this sport we can notice several interesting characteristics that make it suitable for being the focus of research in areas such as Artificial Intelligence, Robotics and Sociology. Since it is played by teams, cooperation is needed among the players to fulfill the overall objective of winning the game. Each player think and act trying that his actions will accomplish a major good for the team and not just for himself. Therefore it must be capable of analyze every situation as fast as possible in order to perform some action in the best possible way. Furthermore, the player must take into account the environment around him and the opponents, in order to take better decisions. For these reasons, soccer is an interesting, complex and challenging problem that is currently the focus of many researchers around the world. 1.1. Robot World Cup Initiative The Robot World Cup Initiative has its origins in Japan, where Kitano, Asada and Kuniyoshi proposed a research program based on soccer called Robot J-League. But since the interest of this research problem grew in other places, the Robot World Cup Initiative was created (known as RoboCup for short) [4]. Later the RoboCup Federation was formed to organize and promote RoboCup events. The main objective of the RoboCup Initiative was to present a new common problem for the A.I. and Robotics community. Having a standard and common problem is important so that the effort of everyone involved can be focused in the solution of such problem. And the progress of such solution can be easily measured and compared. Soccer proved to be a good candidate for this common problem given the characteristics implicit in this sport and its popularity in the world. Therefore, once the problem has been accepted by a large group of researchers, the RoboCup Federation set some goals and timetables for such research. The ultimately goal is the following: By mid-21st century, a team of fully autonomous humanoid robot soccer players shall win the soccer game, comply with the official rules of the FIFA, against the winner of the most recent World Cup. [21] This goal is very ambitious, complex and interesting. It is hard to believe at this moment that there could been such robots, but fifty years has proved to be enough time to accomplish some extraordinary tasks, such as the landing of the man on the moon, or the development of computers million of times faster than the first machines. Hence, there is a good probability that the goal could be fulfilled in such time. Even if the ultimate goal is not accomplished, it is sure that there would be a big technological advancement in the Robotics and A.I. fields. 3 3

Another goal set by the Federation is from the educational perspective. The objective is to stimulate the interest for this problem to the general public by promoting and organizing competitions around the world. We could say that this goal has been already achieved since each year there has been more interest from the general audience in the RoboCup event. And more importantly, more and more young students has shown interest in these events, even small kids. This fact is important since it is probable that some of those young people could be the ones responsible to build the robotic team to accomplish the ultimate goal. The RoboCup is composed of several soccer leagues and other competitions. Among the non-soccer leagues we can find the RoboCup Rescue League, which has the objective of develop techniques and implement heterogeneous agents for cooperation in order to save human lives inside buildings. The RoboCup Junior League is part of the educational effort of the RoboCup, since it is aimed to the participation of young students. Of course, the main focus is within the RoboCup Soccer League, which includes several categories. The Simulation League, the Small-size League, the Middle-size League, the Sony Legged League and the latest addition, the Humanoid League. Currently, the Humanoid League is just a competition where the participants show the advances they have achieved, such as performing a penalty. But this league is the starting point for reaching the ultimate objective. Since this report is based on the Simulation League, we will not talk more about the other leagues, but all the information can be found on [22]. 1.2. Simulation Soccer League The RoboCup Simulation Soccer League was the first formal competition intended to solve the problem of playing soccer. It started in the mid-90 s and since then it has been the focus of extensive research in the fields of A.I., Robotics, real-time systems, Sociology, etc. It has been very popular and attractive because it does not require the use of real robots to participate in the competition and/or the research community. This also brings the opportunity to focus on specific problems that does not require the knowledge of mechanics and/or electronics. Figure 1. Schematic view of the soccerserver simulator. 4 4

The main tool of the Simulation League is the soccerserver software system, which is in charge of simulating all the environment, actions and specific details of a soccer game. Figure 1 illustrates the schematic view of how the soccerserver is used. The server allows to connect players, coaches, and monitors. A soccer team is formed by eleven players and its coach, thus there will be twenty four clients connected to the server in a regular game. Ideally, each client and the server will run on a separate machine that connects to the server via UDP/IP sockets. It is still possible to run the server and/or clients on the same machine, but the accuracy in the communication might decrease and present some problems. The monitor is a utility program that connects to the server to show graphically the soccer game. There are several monitors available but the most common comes with the soccerserver and is called soccermonitor. It is also possible to connect more than one monitor to the server. Each client (player) 1 is a program that will send commands to the server, then the server will execute those commands on behalf of each client. Also the server will send information about the environment relative to each client so that the client could decide which command to send next based on the particular situation that it is happening in the game. Players must only communicate among them via the soccerserver 2, and in that way cooperate as a team. It is important to understand that the server works in cycles of time, thus it will send information and execute commands only at certain intervals in time. This situation brings up the need of a good synchronization mechanism between the clients and the server. Obviously, the client would want to always execute a command and not remain idle during cycles. Another important observation is that the server adds noise to the actions performed by the client. We must remember that in real life a soccer game involves a very noisy environment, for instance, the ball not always go to the desired direction and speed we want to, because of the field, the way we kick the ball, the weather conditions, etc. The server adds this noise to create uncertainty in our actions and also in the way we perceive the environment. Since the environment is highly dynamic, i.e. what we saw in the previous cycle might be completely different of what we are going to see in the next, therefore the server also adds uncertainty and noise to the things we see and feel on the field. Thus the information with which we are working on to make a decision is incomplete and probably not accurate enough. All these restrictions and characteristics make the soccerserver an interesting and appealing way of developing and evaluating multi-agent systems in a real life-like scenario and since all you need is to install the server on a machine, it is also a practical and an inexpensive way to participate with the rest of the community that shares the same objective of solving this problem. 1.3. Main Objectives of This Project As we saw in the previous section, the soccerserver is an excellent platform to work on the specific problems that does not involve the use of real robots. Still, since the intention of the server is to simulate in the best possible way all the particularities involved in a soccer game, there are a large number of details that we need to take into account and to be able to manage in order to have a working team of players. Although, attacking a big problem such as how to play better requires the knowledge of the specific details, sometimes it is desired to skip all those details and go directly to 1 The terms client and player/coach can be used indistinctly on the rest of this report. 2 It is possible to use sockets to make a communication channel between players, but it is assumed as a gentlemen agreement that such thing is not allowed. 5 5

problems such as, how to create good behaviors for the players and how to develop strategies for the team. In other words, directly create teams of players and see the results immediately. Based on that principle, this is the main objective for this project: The development of a high level language, based on rules, that allows the user to create in an easy and fast way a team of players. Providing a complete set of directives, in the form of conditions and commands, that can be used to create rules and ultimately the behavior of the team and its players on the field. This kind of high level language can be used for research purposes or as an educational framework where students can explore how multi-agent systems work by creating strategies and behaviors for each player in the team. Thus, we can start thinking right away in how we can use the conditions and commands available in order to produce better players, without having to know almost anything about how specific details of the server work. There are certain things that still users might need to know, such as the fact that there is noise involved, or that the server executes only one command per cycle. But this is not the same as having to actually worry about how and when to send the command, or how to read the values given by the server, etc. Since the main objective was to create a high level language, and thus, to focus on the development of a large and functional set of conditions and commands, there was no point in creating from scratch a working soccer system (also because the time needed to accomplish that would be quite large). Therefore, the first part of the project was to search and evaluate available source codes to use one of them and start working on top of that. We chose the UvA Trilearn Robotic Soccer Simulation Team [1] as our platform to build on top of that the code for the language. To this end, we modified certain parts of the code and added some other functionalities that were not available in the original code. 1.4. Report Description The sections conforming this report are as follows: Section 2 describes the RoboCup Soccer Server, how it works, the commands available, the messages handled by the server and an overview of other features contained in the server. Section 3 provides an overview about the UvA Trilearn code, it is important to understand part of its functionality and architecture since our final high level language depends on the way it works and how it handles the information. Section 4 describes the additions and modifications made to the UvA original code, and more specifically it describes the layer of code we built on top of the UvA source code. Section 5 provides a description about the proposed language, its syntax and features. Section 6 gives a description about the directives that were created, this section is divided in the description of the conditions and commands for the language. Section 7 presents some ideas that can be accomplished as future work. Section 8 presents the summary and final conclusions of this project. Section 9 gives the references used in the development of this project. 6 6

2. ROBOCUP SOCCER SERVER The RoboCup Soccer Server (or soccerserver) is a software system that simulates all the functionalities encountered on a soccer game. The simulation is executed in a client-server style. Clients can connect to the server via UDP/IP sockets, thus allowing the clients to be programmed in any language that accepts sockets (e.g. C, C++, Java, etc.). Each client is considered a separate process that connects to some port to the server. A soccer team will normally consist on 11 players (10 players and 1 goalie), and a coach 3. Therefore, at most there will be 22 players and 2 coaches for a regular game. The server executes the simulation internally, i.e., we cannot see what is going on when the server is running the simulation. Hence, we need to make use of a soccer monitor program. This monitor will be connected to the server via sockets, as any other client 4. The monitor receives the information needed to build a graphical representation of what is happening inside the server, and so we can see the virtual game. In this section we will explain how the server works, the commands and sensory information available and other features provided by the server. This section is not intended to give all the details the server involves, for a complete explanation of the soccerserver the reader should refer to the RoboCup Soccer Server Manual [10]. 2.1. Functionality The way the server works is easy to understand. Once all players are connected to the server, the server will send sensory information to the client, this information is divided into three types: Visual, aural and body sensory information. We must have in mind that each player receives this information relative to itself, i.e. each player sees, hears and feels what is around it, we do not know what the other players are receiving from the server. The player must manage then this partial information to decide which action to perform from the ones available to it. This action will be send back to the server, where it will be simulated on behalf of the player and thus will change the status of the environment. With the general idea in mind we can now see some particular details about how everything is done. The server is not just a simulation system, but also a real-time system that works in discrete intervals of time (cycles). First, each type of sensory information is sent to the client at different times and thus adding a degree of uncertainty about the information we currently have. The visual information is sent every 150 ms 5, the body sensor information is sent every 100 ms and the aural information will be sent instantaneously. Also, players are expected to send their selected actions in 100 ms intervals. If a player send more than one action during the 100 ms interval, the server will 3 It is not required to use all players and/or the coach; we could connect only one player, however connecting only the coach is not useful at all. 4 In fact we can connect any number of monitors to the server, not only one. 5 The values for the intervals are the ones specified in the latest server version 10. 7 7

chose randomly among the actions it has received. Obviously, this is not desirable, neither the case of not sending an action during an interval, since we do not want to remain idle. Notice that given these values, we will have a big and complex synchronization problem. Visual information is probably the most important data we can receive since it contains the objects and locations of what we are looking at. Now the problem arises due to the fact that we need all possible information in order to select our next action every 100 ms, but the visual information is sent every 150 ms. Furthermore, the server does not specify the start of the cycle in which we must send our action. Precisely this kind of problems are the ones we would like to avoid if we are just interested in building soccer players. 2.2. Sensor Information As we previously mentioned there are three different types of sensory information which the server sends to the client at different intervals of time. These sensor models are divided in: Visual, aural and body sensor models. 2.2.1. Visual Information The visual sensor detects all the objects the player is currently seeing, such objects could be landmarks in the field (flags, lines, goals), the ball and other players. Also, this sensor acts as a proximity sensor, allowing to know which objects are near the player but not visible. The sensor simulates a visual model in which depending on the quality and width of our field of vision, we will see objects in a better or worse way, and the view frequency can also be affected. For instance, we could chose to see with low quality and then the frequency in which we receive the visual information will increase. Of course, the received information will be incomplete. The server will send visual information to the players every 150 ms with the message see, in the format: (see Time ObjName Distance Direction DistChange DirChange BodyDir NeckDir) where, Time = simulation cycle of the soccer server ObjInfo = ( ObjName Distance Direction [ DistChange DirChange [ BodyDir NeckDir ] ] ) ( ObjName Direction ) ObjName = ( p [ "Teamname" [ UniformNr [ goalie ] ] ] ) (b) (g [l r]) (f c) (f [l c r] [t b]) (f p [l r] [t c b]) (f g [l r] [t b]) (f [l r t b] 0) (f [t b] [l r] [10 20 30 40 50]) (f [l r] [t b] [10 20 30]) (l [l r t b]) (B) (F) (G) 8 8

(P) Distance = positive real number Direction = -180 ~180 degrees DistChange = real number DirChange = real number BodyDir = -180 ~180 degrees NeckDir = -180 ~180 degrees Teamname = string UniformNr = 1 ~11 This message contains information about objects the player is currently seeing. In the case of static objects such as flags in the field, we will only receive its distance and direction. In the case of dynamic objects (ball), we might receive as well the change in its distance and direction. And for dynamic objects such as players, we might receive information about their body and neck angle, their team name, and their player number. Receiving all this information will depend on our quality, angle and distance to the objects. Furthermore, the server adds a certain amount of noise to each value and thus adding more uncertainty to the information. 2.2.2. Body Sense Information The body sensor provides information about the current status of the player. It is sent at intervals of 100 ms using a (sense_body) message with the following format: (sense body Time (view_mode ViewQuality ViewWidth) (stamina Stamina Effort) (speed AmountOfSpeed DirectionOfSpeed) (neck_angle NeckAngle) (kick KickCount) (dash DashCount) (turn TurnCount) (say SayCount) (turn_neck TurnNeckCount) (catch CatchCount) (move MoveCount) (change_view ChangeViewCount)) It contains information about the player itself, such as its current view quality and angle, its body and neck direction, its stamina and speed. Also it contains several counters for all the actions available in the server, these counters can be used for statistics or more important, to know whether or not an action sent in the last cycle was executed. 2.2.3. Aural Information Lastly, the aural model allows the player to hear certain things on the field. This is the only message that will be sent to the player without any delay. It is sent in the form of a (hear) message with the format: (hear Time Sender Message ) 9 9

Sender is the relative direction of the sender, just in the case the sender is another player. If not, it will contain one of the following constants: self, referee, online_coach_left or online_coach_right. It is important to understand that the player will not hear every message that could be produced during the game, just those within a certain hearing range. However, every player will always hear all the messages that come from the referee, the coaches and itself. Also, a player will hear a certain number of messages every cycle, unless it comes from the referee. Thus it is not a good idea to override the communication channel with a large number of messages. A communication protocol is needed where we can communicate in a good manner without overloading the hearing capacity of the players. A player could hear one opponent message every second only. 2.2. Action Commands Table 1 shows the list of the available action commands for the current server version 10 (based on the table shown in [1]). Command Parameters Type Frequency Limit catch angle dash power kick power, direction primary only one command move (x, y) position per cycle tackle power turn moment attentionto team, player num one per cycle change_view width, quality one per cycle pointto distance, direction one per cycle say message string secondary none sense_body - three per cycle score - none turn_neck moment one per cycle Table 1. All action commands available to the players. The first group of commands, which henceforth we will call primary commands, are those commands which are executed only one of them per cycle. In other words, if the player sends two of these commands in one cycle the server will chose randomly only one of them to be executed. Thus we must be careful whenever we send commands of this type. On the other hand, a secondary command can be sent to the server along with a primary command. Notice that some of the them still are restricted to be one per cycle, but there is no problem if we send a primary and a secondary command together in the same cycle. Catch command. The goalie or goalkeeper is the only one who can use this command. We need to specify the desired angle in which the goalie will catch the ball. There are no restrictions in the direction of the catch, thus we can specify any angle within the range 180 to 180 degrees. The server defines a certain catchable area in which the goalie could catch the ball with certain probability. The problem is that if the goalie does not catch the ball, it will not be able to use the command again for a certain period of time defined in the server parameters. 10 10

Dash command. This command is used to move or accelerate the player in direction of its body. The command needs the desired power to perform the dash. The stamina of the player need to be considered first in order to perform a dash, since the amount of stamina determines whether or not the player can dash. Every time a dash is performed the stamina is decreased, and although the stamina increases every cycle also, there is a point in which the stamina will no longer increase. It is possible to dash forward or backward but dashing backward will result in a higher stamina consumption. Kick command. This command allows the player to kick the ball in the direction and with the power specified as parameters. The ball must be within the kickable area around the player in order to perform the kicking, if not the command has no effect. The kick will be more powerful and accurate if the ball is located in front of the player, i.e. in the position in which the body angle of the player is pointing to. If that is not the case, the ball will still be kicked but with much less accuracy. Move command. This command can be used only before the game starts and after a goal is scored. This is because we can position the player in some specific coordinates, and this is not possible to do in a real game. The x coordinate ranges between 52.5 to 52.5 and the y coordinate ranges between 34 to 34. Another use of these command is for the goalie, after it catches a ball it can move inside the penalty area a maximum number of times before it kicks the ball. Tackle command. This command is a relatively new and allows the player to tackle the ball. It can be used when we want to recover a ball and steal it from the opponent. It takes a parameter specifying the power of the tackle. The tackle will be performed in the direction of the player s body. It could be seen as a kick that is always performed in the direction of the player. The difference is that it has a certain duration, thus it can be used for a maximum number of cycles, giving the possibility of dashing and tackling the ball. Turn command. This command is used to turn the body of the agent to a certain angle if it is not moving at all. However, if the player is moving (dashing), the turn command will calculate the turning angle taking into account the inertia of the player, since it is more difficult to turn when we are moving. Attentionto command. This is also a new addition to the list of commands. It is used to listen to a certain player during the game. We must specify the team to which we want to pay attention (opp or our) and the number of the player to which we want to listen to. It is sure that we are going to hear messages coming from this player, so this command might be useful for having a better communication and coordination schemes. Change_view command. The player can chose the way it sees by using this command. We need to specify the width and the quality parameters, which are in fact constants narrow, normal or wide for the width parameter, and high or low for the quality parameter. Pointto command. This is another new addition to the set of available commands and it is used to move the player arm so that it can point to a certain position. It takes two parameters, the distance and direction. It might 11 11

be useful for coordination, although it also has a certain duration, therefore we must see that somebody is pointing somewhere before the time expires. Say command. This command allows the player to communicate strings to the rest of the team. The only parameter is the message string to be communicated. Thus, it is unreliable, and hearing depends on the distance to player who is saying the message. Thanks to the incorporation of the attentionto command we can build a better communication scheme. If not it is difficult to rely on only this command. Sense_body command. The purpose of this command is to communicate to the server that we require a sense_body message. Thus, the server once it receives this command, it will send back the sense_body meesage to the player. This message is exactly the same we previously described, but if we ever need it in half the cycle, we could request it. Score command. It only returns the current score of the game. Turn_neck command. The command takes one parameter that specifies the angle to which we want to turn the neck. The player s neck determines the viewing angle, so we could move in one direction but looking at other things. However, it is important to clarify that executing a turn command will make the neck to turn as well, since the neck does not move independently of its body. 2.3. Referee and Play Modes The soccer simulation server encodes a basic automated referee that is in charge of controlling the game by changing the play modes during the game. Also it is able to recognize certain situations such as offside, goals, free kicks, etc. Nevertheless, a human referee is also needed since there are certain events which are hard to recognize and they require a human assistant. For example, the surrounding or holding of the ball, blocking the goal with too many players, etc. Currently, the automated referee can recognize the following situations: Backpasses: the goalie is not allowed to catch the ball if it was passed by one of its teammates. Free kick faults: the player kicking the ball after a free kick, corner kick, kick in or goalie free kick, is not allowed to immediately kick the ball again. Offside: a player is in offside if it is closer to the opponent goal than the ball, or closer to the opponent goal than at least two defenders. Beginning and end of times: the referee recognizes the time before the game starts, the half time, extra time and after a goal has been scored. Out of field: When the ball goes out of the field, the referee determines where to put the ball and the correct mode of play. For example, if it is a corner kick or a kick in. 12 12

The referee announces all these situations and the change of play modes by using a say command, in that way all the players are going to receive the referee announcement in form of a hear message. Furthermore, the referee messages are heard by all the players always in a reliable way. 2.4. Other Features There are other interesting features available in the soccer server [10]. One of them is the possibility of using a client as the coach of the team. The coach is a privileged client that connects in the same way as the players to the server, but receives complete, global and noise-free information of the game. There are two types of coaches, the online coach and the trainer (or offline coach). The difference is that an online coach is used for official games to give advice to the players and planning plays or strategies. On the other hand, a trainer cannot be used for official games, since it has the ability of changing the play modes during the game, therefore it is used to train the players. Both types of coaches communicate with the players by means of the say command. Furthermore, a special language has been created (CLang) to allow the players to understand the messages from a variety of coaches. Or, in other words, a coach can be used with any coachable team. Another feature is the possibility of selecting heterogeneous players given by the server. At the moment of initializing the players, the server will send a message containing information of the available heterogeneous players. These players are generated randomly and they have different characteristics, e.g. some player could have more stamina than others, or could be of different size, etc. The coach is the one responsible of analyzing these available players and substitute them instead of the regular player before the game starts. The use of heterogeneous players brings new interesting possibilities to the problem of coachable teams. 13 13

3. THE UvA Trilearn SOCCER SIMULATION TEAM As mentioned before, our objective was to develop a high level language capable of understanding directives in the form of conditions and commands. Therefore it was not practical to start a simulation system from scratch, from the point of view of the implementation and for the time constraints for this project. Since the establishment of the RoboCup competition several universities and research teams have spent a lot of time and effort in developing good software agents and then releasing their source code so that other interested people could focus on other problems and so avoiding the overhead of developing a new software system. That is the case of this project, and therefore one of the first tasks was to select a good source code so that we could work on top of that. We were looking for a code that above all could be easy to modify to change or add more functionalities than originally were planned. That condition implies that the code should be very well structured and documented. Also it was important to have a recent code that could work with the latest soccerserver version. And finally, a code that also allows you to implement good soccer players. After reviewing several source codes we selected the UvA Trilearn Simulation code as our working platform, since this code contains all the previous characteristics we were looking for. One of the objectives in the development of the UvA Trilearn Team was precisely to have a good base code that could be used in the following years after its creation in 2001. Thus, a lot of effort was put in solving certain low-level aspects of the server, such as the synchronization problem and the development of a way to represent the environment of a soccer game, by means of the agent s world model. Since our project was built on top of the UvA Trilearn code, this section presents an overview of that approach, the main ideas, the agent s architecture and some important features and characteristics that will help to understand the following sections. For a complete and detailed description of the UvA Trilearn project please refer to [1]. 3.1. System Architecture The UvA Trilearn system was created following a functional architecture that is able to describe the functional behavior that the system should exhibit. Figure 2 shows the agent architecture used for their system. Notice that the architecture does not describe the functionality of the team, but just one agent. The resulting actions from the set of agents will describe the team s behavior or functionality. Therefore, each agent should be built having into account that they are not independent, but they are part of a society. 14 14

Figure 2. UvA Trilearn Functional Architecture (taken from [1]). Its functional architecture is described by merging a hierarchical and behavioral approach, which they call hybrid approach. A hierarchical approach is somehow natural in this kind of problems since we can decompose the problem in several tasks or levels and build the system in that form. A behavioral approach is also desirable because it is possible to explore in parallel several options or behaviors, available to the agent instead of having just one path as in the hierarchical approach. Thus, the hybrid architecture will be composed of both approaches, where a more hierarchical approach will be used at the top levels in the system, and a behavioral approach at the low levels, where communication with the server is performed. The interaction layer is in charge of communicating directly with the server. It is divided in two stages, the perception and actuator control stage. The perception stage is in charge of receiving all messages that come from the server, once it receives a message it will pass it to the upper level for its processing. The actuator control stage is in charge of sending messages to the server, i.e. action commands (primary or secondary). These commands are passed to the actuator stage from the upper level in the hierarchy. At this level in the architecture, the behavioral approach was implemented. It is desirable that both stages can work in parallel, meaning that both stages should be independent from each other in the sense that they can receive and send messages at the same time. Furthermore, it is not desirable that both stages take control of the computation time, thus both process are active just the time needed for receive and send messages. This is accomplished with the use of software threads. If there is nothing they can do, they will block themselves until there is some message to receive or send. The skills layer is the second level in the functional architecture. Again it is divided in two stages, the modeling and the action refinement stages. The modeling stage is connected to the perception stage and therefore it is in charge of processing the messages from the server (see, hear, sense_body messages). This stage should present all this information in a useful and complete way to the next level in the hierarchy. We will see that the modeling stage constructs the world model of the agent. Before we explain the action refinement stage, let s see the last layer first. 15 15

The control layer is the highest layer in the hierarchy, it only contains one stage called the reasoning stage. This stage is in charge of making the decision about which high level action should be done at that moment, based on the information gathered by the modeling stage. Notice the importance of having the interaction layer as threads, in this case the reasoning layer is the one who must have the higher processing time in the computer, that is why it is important to just use the exact amount of time for the interaction with the server. Once the reasoning stage selected a high level action to be performed, that information is passed to the second stage of the skills layer, the action refinement stage. This stage is in charge of selecting the final command that will be sent to the server. Recall that at the end the server only understands a small set of commands (kick, dash, catch, etc.), and the reasoning stage may select a high level action such as dribble with the ball. Therefore, it is this stage the one who will determine the exact low level command to be sent. For instance, a dribble might be composed of the low level commands turn, dash and kick. Once the command has been selected it is passed to the actuator stage so that it can be sent to the server. 3.2. System Classes Now that we have described in general the functionality of the system we will explain more specific details about how each of the previous stages are implemented. The system was implemented following an object-oriented design, which provides a natural way of programming the stages within the architecture. An object-oriented system is composed of a collection of objects, which are defined using classes. The system was programmed using the C++ programming language for Unix 6. This programming language was chosen because it is object-oriented and allows the implementation of threads. Figure 3 illustrates the same agent architecture, but now described in terms of the software components (classes) of the UvA Trilearn system. The figure shows the UML class diagram of their system. The Unified Modeling Language (UML) is used for specifying, visualizing, constructing and documenting software systems 7. 6 The 2003 version of the program can also be used in the Windows environment using Borland C++. 7 For the purpose of this section, it is not necessary to be familiar with the UML language. 16 16

Figure 3. UML class diagram for the UvA Trilearn simulation system (taken from [1]). The interaction layer is now composed of three classes, connection, sense handler and act handler. The sense and act handler will have the same functionality as the perception and actuator control stages. But since both classes make use of sockets for communicating with the server, the connection class provides that functionality. At this point is where threads are used in order to implement the sense and act handler. Thus, both classes will activate only when it is necessary. The skills layer is composed of three classes, object, world model and basic player. The world model class processes all the information received by the sense handler. The world model class makes use of the object class which defines all possible objects that we can find in a soccer game. For instance, it contains classes to represent static objects such as flags in the field, a class to represent the ball, a class to represent the players and a class to represent the agent. The agent class makes use of the stamina class to represent the condition of the agent. Also, the world model class makes use of other classes such as the server settings and soccer types classes. As we said, the purpose of the world model class is to build, in the best way, a complete representation of the agent s world. In this way the agent will be able to make better decisions. The world model class is in fact composed of four different types of methods: retrieval methods, update methods, prediction methods and high level methods. The retrieval and update methods are used to process the information that comes from the server. Prediction methods are able to simulate commands, as if the server were executing them, in order to know possible outcomes of the agent s 17 17

actions in advance and thus making better decisions. Of course, these methods are not 100% accurate, since they use past information to predict the outcomes, but still they are a powerful tool. The high level methods allow the agent to know specific things about the world, that can be used to make better decisions. The control layer is composed of three classes, player settings, formations and player. These form the reasoning stage in the architecture, thus they are used as the high level action selection mechanism in the system. The player class is the one that contains the decision making procedure that will select a high level action to be performed. It makes use of the player settings class, which is just a set of variables and their values needed to specify some actions. They can be changed by modifying the player.conf file 8. It also makes use of the formations class, which defines the positions of the players during the game 9. The player class contains an infinite loop, where it will always be processing the next action to be performed in the agent life time. Although the figure does not show an arrow from the world model to the player class, the player class also uses the world model class. Once the player class selects the high level action to be performed, the skills layer is used again in the form of the basic player class. This class will determine the correct action command to be sent to the server. It makes use of the world model class in order to make such decision. And when it is ready it will send the command to the act handler closing the processing circle. 3.3. Other Features In Section 2, we briefly explain the synchronization problem the soccerserver imposes. In the realization of the UvA Trilearn system, several experiments were conducted in order to solve that problem. The main properties for a good synchronization method are the following: (taken from [1]) An action command must be sent to the server in each cycle. Since the server executes the action commands at the end of each cycle, we must be sure in sending the commands in time to be executed. Make sure that the selected action command is based on current visual information. In [1] they describe several techniques in order to achieve the previous properties. At the end they chose to use the Flexible External Windowing method, which makes sure of sending a command every 100 ms and with enough time to be sure it can be executed at the end of the cycle. Also it takes into account the coming of new visual information in order to make a good action selection. If it calculates that during a cycle no visual information will arrive, then it will send the command selected using past information. 8 This player.conf file is totally different from the player.conf file the soccerserver uses. 9 In the next section we will explain in detail this class. 18 18

4. HIGH LEVEL LANGUAGE IMPLEMENTATION Now that we have reviewed the functionality of both, the soccerserver and the UvA Trilearn system, we can observe certain things. While the idea beneath the soccerserver is easy to comprehend, the construction of just one agent is complex and time consuming. There is a large number of considerations that must be taken into account in order to build a software system that could work with the soccerserver. Hence, if we are just interested in creating strategies for players, it will take too much time to get familiar with the server, and not only time but too much effort also, in order to implement a software system. Even if we just use an available source code such as the UvA Trilearn system, it still requires time to get familiar with their code and also requires programming knowledge in order to make changes and create your own players. Therefore, the availability of a high level language that can be easy to understand and use, will result in an excellent way of allowing a large number of users the possibility of creating their own teams in a fast and easy way, in some cases just requiring a small amount of knowledge about how the soccerserver works. In this section we will describe our proposed high level language, how it works, its syntax and special features that will allow the users to create a complete team of players. 4.1. A Rule-Based Language The proposed high level language is based on rules. A rule is of the form: If ( conditions ) then perform some action Conditions must be satisfied in order to perform the directive described in the then statement. Rules are a natural way to describe situations and perform some actions, and in a soccer game can be easily used. For example: If I can score a goal then kick the ball to the goal. Rules allows to create a more reactive kind of players, in the sense that given the current circumstances the player will decide which action to perform. Rules are easy to read and understand, especially if they can be written in English form as the previous example. And thus, rules are easy to build and modify. Therefore, we tried to implement a language that could have all these properties. The specific syntax for the language is the following: if ( condition(s) ) then command(s); end Where conditions and commands can be any of the available that we have previously defined. There can be more than one condition, and can be compared using logical operators: (&&) for ANDoperation, ( ) for OR-operation and (!) for negation; or relational operators: (<) is greater than, (>) 19 19

is less than, (<=) is greater-equal to, (>=) is less-equal to, (==) for is equal, and (!=) for is different. The use of these operators provides a way to have thousands of combinations among conditions. If the condition(s) is satisfied then the command(s) will be executed. It is possible to execute more than one command because secondary server commands 10 can be executed along with primary server commands. Thus, one of the things the user must know is which high level commands result in a primary server command, and which ones do not. Although we will provide a description of the commands with that information, it is not hard to deduce. Suppose we want to execute a kicktogoal command, this command has the kick server command implicit in its name, so it is easy to know that it will result in a primary server command at the end. However, it is less clear to see that that command could result in a turn server command also, which still is a primary server command. The point with this example is to see that the user just need to know which commands result in a primary server command and which one in secondary server commands. We must have in mind that every high level command used in the language is composed of one or more server commands. For instance, a less obvious command could be interceptball. This command encodes a procedure that calculates which server command should be sent to the server. So, depending on certain conditions, such as the position and direction of the player, it might choose to turn, kick or dash. Again, a user does not need to know those details, but just to know that at the end that command will execute a primary server command. Thus, once we know that information we can execute more than one command in one rule. 4.2. Logic of the Game Now that we learned how a single rule works, let s see how the whole set of rules work, in other words, the logic we must have in mind while creating rules. In section 3, when we describe the classes that form the UvA Trilearn system, we mentioned that the player class contained an infinite loop where the player will be always running in order to determine which command to perform next. Well, that means that the loop will always be examining our rules in order to select the next action. The examination of the rules will be in the order in which the rules were defined, i.e. the order we write our rules is extremely important. First, we must have in mind that every time the system chooses a command to be performed, it will start the loop all over again. This means that if the first rule is always satisfied then it will always execute the command specified in that rule, and will never check the rest of the rules. Therefore, another way to formulate this fact is with the following statement: The ordering of the rules determines their priority. At first sight it might not be a good idea to have the loop start all over again after choosing a command, but if we think about it more carefully we will see that rules must have a priority, we will want to always execute the rule with the highest priority. Therefore, determining the priority with the ordering is a simple way to do it. Let s examine an example: If we are an attacker and in our head there are two rules: if I can kick the ball to the goal then kick the ball if I am not in my position then go to my assigned position 10 To avoid confusion with our commands and the ones that ultimately accepts the server, we will use the word server to specify that those are the low level commands. 20 20

Imagine the loop starts with the first rule and it is satisfied at that time but we do not score any goal. But the ball was kicked back by an opponent and I can kick it again. If we let the loop continue where it left off, we will not kick the ball but return to our assigned position since we are not there!! Now, if we let the loop start all over again, then we will definitely kick the ball and try to score again. The same bad thing will happen if the loop starts always but we swap the rules, then we will always move to our position and will never kick the ball, unless we are in position. Therefore, the ordering is extremely important since it determines the ultimate behavior of the player. 4.3. Modes It can be argued that rules and a reactive behavior in players might not be good in certain circumstances and that a planned strategy could work better. Therefore, our language incorporates the use of modes, which in fact can be used for a number of things. A mode is specified as follows: mode name_of_mode : rules : endmode Modes allow the user to group rules and to activate them later in any situation they might find useful. This is done with the special command activatemode (name_of_mode, frequency). Where frequency can be: ALWAYS: always activate this mode. HALF: with 50% of probability activate this mode. LOW: with low probability activate this mode (10%). The use of modes provides new ways to build player behaviors. One way of looking at this feature is in the creation of plans. We can have separate modes or plans for specific circumstances, such as a corner kick or free kick. This is accomplished by looping over the rules of the particular mode we have activated. Also, with the additional specification of the frequency parameter, we can have nonpredictive players, i.e. opponents could not predict for sure the behavior of the agent, adding a surprise factor to our players. Given a certain situation we might or might not take the risk of changing modes. Those probabilities are randomly selected using the server time. Once we are in a certain mode, we could activate any other mode or go back to the normal mode, using the constant DEFAULT. Thus, the rules that are not grouped in any mode are treated as the normal or default mode of the player and there is no need to specify that in the language. The only restrictions when using modes are that we must define all modes before the default rules, and that there cannot be modes within modes. 21 21

4.4. Player Types It is possible to write rules for 8 different types of players 11, based on the players defined in the UvA Trilearn system. The player types are the following: Goalie (or goalkeeper) Central defender Sweeper defender Wing defender Attacker Wing attacker Center midfielder Wing midfielder The user should specify the rules for each type of player in a specific file. For example, the rules for the goalie should be defined on the goalie.rules file. Thus, there are 8 different files we might use. These files are only intended to provide an organized way of describing players. That means that the attacker file might contain the rules for a defender and vice versa, the files are just names and the ultimate behavior of the player is governed by its rules. Also, it is important to observe that it is not required to specify rules for all the files. We will see in the next section that the user specify the type of player in the formations.conf file, meaning that we can define all the players of type attacker, and within those rules we can define via modes the rules for each player. Obviously, this will produce a really large file of rules, but it is possible to do it. For instance, we can use the condition: if ( getmynumber == 2 ) then activatemode ( defender, ALWAYS). So, player number 2 will always activate and work with the mode called defender. Therefore, users might choose any way they want to build their players and to use the available files to organize the rules. 4.5. Compiler The compiler for the language was programmed in C++ for Unix. Figure 4 illustrates a schematic view of the compilation process. The compiler analyzes the rules specified in the files for each player, if everything was correct it will generate a file named PlayerTeams.cpp which will contain the C++ code which will be linked with the UvA Trilearn system. 11 Do not get confused with the heterogeneous players defined by the server and these types of players. These types are defined just in the UvA Trilearn system, and in fact, they are just names given to the players, its type does not change anything in its functionality. 22 22

Figure 4. Compilation process diagram. As we explained above, it is not necessary to define rules for each file. We can tell the compiler which files we want to compile, using the following syntax: parser option(s) where options are: g - compile Goalie file dc - compile Defender-Central file ds - compile Defender-Sweeper file dw - compile Defender-Wing file at - compile Attacker file aw - compile Attacker-Wing file mc - compile Midfielder-Central file mw - compile Midfielder-Wing file p - compile ALL player files Once the C++ file is generated it can be linked to the UvA Trilearn code and it will be ready to run. 23 23