Investigating Reinforcement Learning in Multiagent Coalition Formation

Transcription

1 Investgatng Renforcement Learnng n Multagent Coalton Formaton Xn L and Leen-Kat Soh Department of Computer Scence and Engneerng Unversty of Nebrasa-Lncoln 115 Ferguson Hall, Lncoln, NE {xnl, lsoh}@cse.unl.edu Abstract In ths paper we nvestgate the use of renforcement learnng to address the multagent coalton formaton problem n dynamc, uncertan, real-tme, and nosy envronments. To adapt to the complex envronmental factors, we equp each agent wth the case-based renforcement learnng ablty whch s the ntegraton of case-based reasonng and renforcement learnng. The agent can use case-based reasonng to derve a coalton formaton plan n a real-tme manner based on the past experence, and then nstantate the plan adaptng to the dynamc and uncertan envronment wth the renforcement learnng on coalton formaton experence. In ths paper we focus on descrbng multple aspects of the applcaton of renforcement learnng n multagent coalton formaton. We classfy two types of renforcement learnng: case-orented renforcement learnng and peerrelated renforcement learnng, correspondng to strategc, off-lne learnng scenaro and tactcal, onlne learnng scenaro respectvely. An agent mght learn about the others ont or ndvdual behavor durng coalton formaton, as a result, we dentfy them as ont-behavor renforcement learnng and ndvdual-behavor renforcement learnng. We embed the learnng approach n a mult-phase coalton formaton model and have mplemented the approach. Introducton Coalton formaton n multagent systems s a process where agents form coaltons and wor together to solve a ont problem va coordnatng ther actons wthn each coalton (Sandholm 1999, Shehory and Kraus 1998). It s useful as t may ncrease the ablty of agents to execute tass and mproves ther payoffs. In general, each autonomous agent s ncapable of performng specfc global tass all by tself. So some agents may form coaltons to allocate tass among them to acheve the global goals. However, n complex real-world envronments, an agent only has ncomplete even naccurate nformaton about the dynamcally changng world and the occurrence of events may requre the agents to react n a real-tme manner. When desgnng multagent systems n dynamc real-world envronments, t s mpossble to foresee all the potental stuatons an agent may encounter and specfy an agent behavor optmally n advance. To ncrease problemsolvng coherence and mprove the total performance of the system as a whole, agents should be equpped wth learnng abltes so that they can learn from ther own behavors as well as ther nteracton patterns and adapt to the envronment (Sycara 1998). In recent years, renforcement learnng of agents behavors has attracted more attenton of the communty of multagent systems because of ts adaptablty to dynamc envronments. It has been appled to multagent problems such as the robotc soccer (e.g., Salustowcz, Werng, and Schmdhuber 1998), the predator-prey pursut game (e.g., Kohr, Matsubayash, and Tooro 1997), and the prsoner s dlemma game (e.g., Sandholm and Crtes 1995). We have mplemented a case-based renforcement learnng (CBRL) approach to multagent coalton formaton problem n dynamc, uncertan, real-tme, and nosy envronments. In such an envronment, (1) each agent only has a partal vew of the dynamcally changng envronment and the uncertan behavor of other agents, (2) the ntal states that prompt the decson mang process may change whle the decson mang process s stll gong on, (3) actons performed are not guaranteed to result n expected outcomes, (4) the coaltons need to be formed n real-tme manner, and (5) the accurate event sensng or peer nteracton s not guaranteed due to the nose durng the agent percepton. As a result, a coalton-ntatng agent cannot exactly now whch peer agents are able, or are wllng to on the coalton to perform a global tas. It also cannot exactly expect the computatonal and communcaton cost of coalton formaton process or coalton s executon outcome. Our CBRL approach ntegrates case-based reasonng (CBR) and renforcement learnng (RL) to utlze the agent s past coalton formaton experence on the current problem and renforce the utlty of ts experence wth the current coalton formaton outcome. Specfcally, we ap-

2 ply case-based reasonng to store and reuse prevous coalton formaton strateges, and ts role n coalton formaton s to provde a bass for sutablty study of a coalton formaton plan gven a partcular tas. We apply renforcement learnng to evaluate and score each plan based on the outcomes, and contnuously learn about other agents behavors n onng coaltons. Its role n coalton formaton s to contnuously ncrease the lelhood of a good plan beng selected n the next coalton formaton tass and dentfy hgh-utlty peer agents as coalton canddates. Here we nvestgate the multple aspects of renforcement learnng applcaton n our CBRL approach: (1) caseorented renforcement learnng vs. peer-related renforcement learnng, (2) strategc renforcement learnng vs. tactcal renforcement learnng, (3) off-lne renforcement learnng vs. onlne renforcement learnng, and (4) ont-behavor renforcement learnng vs. ndvdualbehavor renforcement learnng. The case-orented renforcement learnng s to learn about the utltes of cases n casebase whle the peer-related renforcement learnng s to learn about other agents behavors. They correspond to the strategc, off-lne renforcement learnng scenaro and the tactcal, on-lne scenaro respectvely snce the caseorented renforcement learnng provdes a strategc learnng approach to facltate the plannng of coalton formaton strateges and t occurs outsde of the coalton formaton process whle the peer-related renforcement learnng provdes a tactcal learnng approach to learn how to nstantate the planned strategy and t occurs durng the coalton formaton process. In addton, we classfy the peerrelated renforcement learnng nto ont-behavor and ndvdual-behavor renforcement learnng. The former s to learn about a peer agent s socal characterstcs n onng coaltons, e.g., helpfulness of the peer to the agent, whle the latter s to learn about a peer agent s personal characterstcs, e.g., the avalablty degree of a specfc capablty whch ndcates whether the peer possesses the desred capablty to perform a tas. We embed the case-based renforcement learnng n a mult-phase coalton formaton model. The model conssts of three phases: coalton plannng, coalton nstantaton, and coalton evaluaton. Renforcement learnng s employed n coalton nstantaton and coalton evaluaton. Bacground and Related Wor Renforcement learnng s the process of learnng to behave optmally wth respect to some scalar feedbac value over a perod of tme (Sen and Wess 1999). It can be regarded as a memoryless learnng technque, n whch an agent chooses an acton only based on the last observaton. In the standard renforcement learnng model, on each step of nteracton the agent receves as nput some ndcaton of the current state of the envronment; the agent then chooses an acton to generate as output. The acton changes the state of the envronment, and the value of ths state transton s communcated to the agent through a scalar renforcement sgnal. The agent should choose actons that tend to ncrease the long-run sum of values of the renforcement sgnal. It can learn to do ths over tme by systematc tral and error (Kaelblng, Lttman, and Moore 1996). Wth the bascs of multagent characterstcs and propertes defned n recent years, multagent learnng has become an mportant research ssue (Excelente-Toledo and Jennngs 2002, Sen and Wess 1999, Stone and Veloso 2000). Compared wth sngle-agent systems, the multagent systems are more complex partally because of the dynamc nter-agent nteractons. Agents possbly need to learn from ther prevous behavors and other agents behavors to decde on the next actons (Alonso et al. 2001). There are generally three aspects: (1) an agent learns about the other agents and ther envronments by observaton n order to predct ther behavors or to produce a model of them (e.g., Hu and Wellman 1998, Nagayu, Ish, and Doya 2000); (2) agents learn how to coordnate or cooperate to acheve common goals (e.g., Haynes and Sen 1996, Tan 1993); and (3) an agent meta-learns what partcular coordnaton mechansms to use (e.g., Prasad and Lesser 1997, Soh and L 2003, Soh and Tsatsouls 2001, Sugawara and Lesser 1998). In these areas, few address renforcement learnng to form coaltons among agents. In (Soh and L 2003, Soh and Tsatsouls 2001), the learnng s about how to negotate between two agents, at a lower level com-pared to our proposed approach n ths paper. In tradtonal coalton formaton, a ratonal agent can solve the combnatoral problem optmally wthout payng a penalty for delberaton. In (Sandholm and Lesser 1995), a bounded ratonalty s evdent n whch agents are guded by performance profles and computatonal costs n ther coalton formaton processes. Smlarly, our agents are aware of ther communcaton and computatonal costs as well as tme constrant n the coalton formaton process. Agents learn about peer agents characterstcs to select low-cost but hgh-utlty coalton canddates and thus reduce the communcaton and computatonal cost durng coalton formaton. Multagent Coalton Formaton The applcaton context of our renforcement learnng s multagent coalton formaton. We have desgned a Mult- Phase Coalton Formaton (MPCF) model to address factors such as uncertanty, nose, real-tme and dynamc ssues. The model conssts of three phases: coalton plannng, coalton nstantaton and coalton evaluaton, as

3 depcted n Fgure 1. In coalton plannng, the coaltonntatng agent derves a coalton formaton plan. In coalton nstantaton, the agent carres out the planned formaton strategy, dentfyng and negotatng wth other agents fttng the specfcatons of the plan. In coalton evaluaton, the agent evaluates the coalton formaton process, the formed coalton (f a coalton s successfully formed), and the coalton executon outcome (f the coalton s executed eventually) to determne the utlty of the plan. In coalton plannng, the coalton-ntatng agent derves a specfc coalton formaton plan for the current problem. A coalton formaton plan specfes the number of coalton canddates, the number of expected coalton members, the tme allocated for coalton nstantaton, the allocaton algorthm, and the number of messages recommended. The coalton nstantaton phase mplements the coalton formaton plan to form a coalton. At frst, the coalton-ntatng agent normalzes the tas dvdng the tas nto separate executon unts as dfferent negotaton ssues, computng the potental utltes of ts peers, and ranng the peers based on ther potental utltes. Then the agent concurrently negotates wth each selected peer agent on the set of subtass n an attempt to form the ntended coalton. Each negotaton s argumentatve where the ntatng agent attempts to persuade the respondng agent to perform a tas or provde a resource by provdng support or evdence for ts request (Soh and Tsatsouls 2001). CBR Case Base yes Coalton Executon CBRL success? Tas Coalton Requrement Analyss Coalton Plannng Coalton Instantaton no Coalton Evaluaton Coalton formaton problem Renforcement Learnng nteracton Dynamc Proflng Profles Fgure 1. Learnng-based MPCF model Other Agents The coalton evaluaton phase provdes the bass for an agent to mprove ts coalton formaton plans. Ths phase evaluates both the coalton nstantaton process (n terms of tme spent, number of messages used, number of peers approached, etc.) and the executon outcomes of the subtass agreed upon n the coalton (n terms of the number of subtass performed by hghly-capable peers, etc.). In general, a good plan s one that uses lttle computatonal and communcaton resources wth successful nstantatons and subsequent executons. Renforcement Learnng Our CBRL desgn, as shown n Fgure 2, s amed at (1) dentfyng the stuaton where a plan was successful and renforcng that stuaton-plan par n a case, and (2) learnng about peer agents behavor n onng coaltons and dentfyng peer agents of hgh potental utltes to the current coalton formaton. We dentfy the applcaton of renforcement learnng n our coalton formaton model as case-orented renforcement learnng (CRL) and peerrelated renforcement learnng (PRL). The case-orented renforcement learnng s appled specfc to (1) and the peer-related renforcement learnng specfc to (2). Case Forgettng Case Base Retreve the best case (BC) Adapt plan to new problem Instantate plan Compose a new case (NC) Assgn V(s, to NC Store NC? Problem nstance BC Plan Outcome Evaluate plan based on nstantaton and executon outcome yes Evaluaton result NC no Update V(s, of BC wth CRL PRL Case Retreval Case Adaptaton Case Applcaton Case Constructon Case Retenton Fgure 2. Case-Based Renforcement Learnng. CRL = Caseorented Renforcement Learnng, and PRL = Peer-related Renforcement Learnng. A coalton formaton case n the casebase conssts of a problem descrpton, a soluton, an outcome, and ts utlty. The problem descrpton conssts of an agent s external and nternal envronments and the tas descrpton. The soluton part gves a coalton formaton plan for the tas

4 to conduct the actual coalton formaton process. The outcome part ndcates the subtas allocaton result among agents at the end of the coalton formaton process, subtass executon results n the case that subtass are eventually executed by coalton members, and the evaluaton values to the actual coalton formaton process. The utlty part ndcates the qualty of the case, specfcally, the qualty of the soluton to solvng the current coalton formaton problem. The case-orented renforcement learnng s to ( learn the utlty of a case based on ts coalton formaton plan and how well t was appled, and (b) retreve hgh-utlty cases more often. The peer-related renforcement learnng s to ( learn the potental utlty of each peer agent based on ts coalton formaton behavor, and (b) approach hghutlty peers more often. In our defnton of the renforcement learnng, we follow the multagent renforcement learnng model (Sen and Wess 1999) where agents are gven a descrpton of the current state and have to choose the next acton from a set of possble actons so as to maxmze a scalar renforcement receved after each acton. The tradtonal renforcement learnng can be modeled by a fnte-state Marov decson process (MDP) that can be represented by a 4- tuple <S, A, P, r> where S s a set of states, A s a set of actons, P: S S A [0, 1] gves the probablty of movng from state s 1 to s 2 on performng acton and r: S A R s a scalar reward functon. However, the renforcement learnng n a dynamc multagent envronment of multple learnng agents cannot be formulated exactly as an MDP as the state transton probablty of the envronment changes wth tme due to the uncertan behavor of the other agents. We modfed the tradtonal learnng model to adapt to the complex envronment. Case-Orented Renforcement Learnng Case-based reasonng s to reuse the past soluton n current smlar problems. Due to the dynamc and uncertan envronment, the past good plan may not be a good soluton for the current problem; as a result, the outcome of the plan nstantaton may be not good. Ths can be reflected n the case s utlty for the future case reference n plannng. We employ the case-orented renforcement learnng to learn and renforce the cases utltes. After coalton nstantaton and coalton executon, the agent evaluates the coalton formaton process and the outcome. Couplng the evaluaton and the problem descrpton, the agent matches the new case to ts casebase. If the new case dffers sgnfcantly from the exstng cases, the agent learns the case to ncrease the case space. Otherwse, the agent updates the orgnal best case s utlty usng the evaluaton result to renforce the case. On one hand, ths s because the casebase also should eep bad experence as lessons. On the other hand, the dynamc and uncertan envronment may mae a plan wth bad outcome able to produce good outcomes for other problems n the future. Durng case retenton, the case-orented renforcement learnng updates the utlty of the best case retreved, the soluton of whch was used n a coalton formaton process. The renforcement learnng algorthm s: V t+ 1 ( s, (1 α ) Vt ( s, + α Rt + 1 where the state s corresponds to the current coalton formaton problem; the acton a corresponds to tang a coalton formaton plan whch s adapted from the plan n the best case; V t ( s, s the old utlty value of the best case whle V t+ 1( s, s the new utlty value of best case; α s the learnng rate (0 α 1); R s the performance t+ 1 evaluaton result on the actual nstantaton process (.e., the reward) at tme t+1. Ths reward ncludes the qualty of the coalton formaton process and the qualty of the coalton. The learnng result wll be used n the agent s coalton case selecton acton. Peer-Related Renforcement Learnng At the end of coalton plannng, the coalton formaton plan for the current problem has been decded. The coalton-ntatng agent can now whch type of peer agents should be approached for the coalton formaton. For example, an emergent tas needs hgh-promptness peer agents to reduce coalton nstantaton tme. In coalton nstantaton, the agent wll select peer agents as coalton canddates accordng to the planned canddate type, and concurrently negotate wth each of them on the set of subtass n an attempt to form the ntended coalton. However, an agent only has ncomplete nformaton about the envronment and other agents, so the coalton-ntatng agent needs to learn about the dynamc and uncertan behavor of peer agents n past coalton formaton actvtes to perceve ther characterstcs and potental utltes as coalton canddates. We employ peer-related renforcement learnng to learn the characterstcs of each peer agent and the correspondng potental utlty through nteractons between agents. The potental utlty of a peer agent as a coalton canddate perceved by an agent s based on the cooperaton relatonshp between the peer and the agent (Soh and Tsatsouls 2001), and the peer s coalton-derved behavors, negotaton-derved behavors, and estmated capabltes. They are recorded n the neghborhood profle. The parameters profled nclude: (1) the helpfulness of the peer to the agent ndcatng the satsfacton degree of requests to the peer, (2) the helpfulness of the agent to the peer ndcatng the satsfacton degree of requests from the peer, (3) the relance of the agent on the peer n terms of the rato of

5 sendng requests to the peer among all peers, (4) the relance of the peer on the agent n terms of the rato of recevng requests from the peer among all peers, (5) the peer s tardness degree ndcatng the communcaton delay between the agent and the peer, n terms of the message round-trp tme (RTT) between agents, (6) the peer s hestaton degree ndcatng how readly the peer s to agree to a request, n terms of the number of evdence messages the agent needs to provde to persuade the peer, (7) the avalablty degree of capablty ndcatng whether the peer possesses the desred capablty to solve tas, (8) the relablty degree n coalton formaton actvtes based on the standard devatons of the peer s behavors, and so forth. These parameters reflect varetes of characterstcs of the peer agent. After each nteracton (negotaton), the agent A updates the potental utlty (for future coalton formaton actvty) of ts peer A s th characterstc C A n the followng manner: PU ( s, t + 1) (1 β ) PU ( s, t) + β C ( A, t + 1) A, C A, C A A where the state s corresponds to the current coalton formaton problem; the acton a corresponds to coalton canddate selecton; PU ( s, t), s the old potental A C A utlty of C A and PU ( s, t + 1) s the updated one; β A, C A s the learnng rate (0 β 1); and CA ( A, t + 1) s the peer A s th characterstc as measured by A. The learnng result wll be used n the agent s coalton canddate selecton acton. The potental utlty of peer A s the weghted sum of the potental utltes of ts characterstcs. It s computed when agent A selects coalton canddates. The weght values adapt to the canddate type requrement. Wth ths renforcement, an agent prefers peer agents that have been helpful and coalton-worthy. The peer-related renforcement learnng n our coalton formaton model can be dentfed as ont-behavor renforcement learnng and ndvdual-behavor renforcement learnng further accordng to the peer agent s dfferent characterstcs revealed through the nteractons. The ontbehavor renforcement learnng s to learn about a peer agent s socal characterstcs n onng coaltons such as helpfulness degree, relance degree, relablty degree, and so forth. The ndvdual-behavor renforcement learnng s to learn about a peer agent s nherent characterstcs such as tardness degree, hestaton degree, avalablty degree of capablty. We dstngush the ont-behavor renforcement learnng and ndvdual-behavor renforcement learnng to dentfy the dfferent roles of dfferent types of characterstcs of peer agents n coalton formaton. In prncple, the ont-behavor renforcement learnng s enough for a coalton-ntatng agent to dentfy the potental utlty of a A peer agent as coalton canddate based on the past cooperaton experence. In the dynamc and uncertan envronment, however, peer agents nherent characterstcs may change as the tme progresses. It wll nfluence the peer s partcpaton to coalton formaton actvtes. So we also employ ndvdual-behavor renforcement learnng to address the envronmental factors. Case-Orented RL vs. Peer-Related RL The case-orented renforcement learnng occurs at the coalton evaluaton phase and the learnng result s appled at the coalton plannng phase. We apply t nto the coalton formaton model as a strategc learnng approach to facltate the plannng of a coalton, adaptng to the realtme and envronmental requrements. The peer-related renforcement learnng occurs at the coalton nstantaton phase and the learnng result s also appled at the coalton nstantaton phase. We apply t nto the coalton formaton model as a tactcal learnng approach to address how to nstantate a coalton formaton plan, tang nto account uncertan and dynamc behavors of the peer agents. In the dynamc, uncertan, and nosy envronment, a good coalton formaton plan of an agent s not guaranteed to succeed as planned. Thus, a conceptually good plan may not be a practcally good plan. The contnual tactcal learnng durng coalton formaton s necessary to address the change of the envronment. Correspondng to the dfferent occurrences, the caseorented renforcement learnng and the peer-related renforcement learnng are n the off-lne renforcement learnng scenaro and the onlne scenaro respectvely. The case-orented renforcement learnng can be off-lne because t does not drectly need agents nteractons. The peer-related renforcement learnng must be onlne because t drectly needs the nteractons between agents. We combne off-lne learnng and onlne learnng nto our renforcement learnng to address the real-tme, dynamc, uncertan, and nosy envronmental factors. Then agents can learn n both scenaros of wthout nteractons and wth nteractons. In short, the applcaton of case-orented renforcement learnng n coalton formaton s to meta-learn coalton formaton processes as a strategc learnng approach to conduct the actual coalton formaton process. Its off-lne applcaton can reduce the agents computatonal costs spent on learnng durng coalton formaton. The applcaton of peer-related renforcement learnng n coalton formaton s to dynamcally learn other agents behavor n the course of contnual nteractons as a tactcal approach to learn how to nstantate the planned coalton formaton strategy. The computatonal costs for ths onlne learnng scenaro are necessary.

6 Experments and Results We have mplemented a multagent system where each agent s capable of performng multple tass and has multple resources. Here we present our experments and results to evaluate the performance of our renforcement learnng approach. Partcularly, our experments were to nvestgate the mpact of learnng on the success rate of coalton formaton, and on the qualty of the coalton formaton process. We report our experments on four versons of our multagent desgn. The frst verson was CRLPRL n whch both case-orented renforcement learnng and peer-related renforcement learnng are used. The second verson, OnlyPRL, used only the peer-related renforcement learnng. The thrd verson, OnlyCRL, used only case-orented renforcement learnng. The fourth verson, NoLearnng, dd not use any learnng at all. In our experments, there were 9 agents n the system, A 1 ~ A 9, each of whch could ntate coalton formaton actvtes for tas fulfllment. We randomly smulated a seres of 40 tass (tmed to occur at dfferent tmes) for each agent. Each tas conssted of dfferent subtass and requred dfferent resources. All tass requred the agent to ntate coaltons. We ran the experments about a dozen tmes to obtan the average values used n the followng dscussons. Impact of Learnng on Coalton Formaton Success Rate Fgure 3 shows the mpact of renforcement learnng on an agent s success rate n formng coaltons. Success Rate (%) CRLPRL OnlyPRL OnlyCRL NoLearnng Varous Learnng Fgure 3. Success rates of coalton formaton for dfferent agent versons of learnng mechansms Note that the success rate of each verson of our agent learnng desgn was low (<60%). Ths was due to the envronment the agents operated n. Snce the agents were handlng ther own tass whch overlapped temporally, t was possble for a peer agent not able to entertan or agree to a coalton request. Real-tme constrants also played a role. Each tas was tmed and thus a coalton formaton process that had run too long would be termnated. An agent that had been too slow n respondng or communcaton would cause such delays. From Fgure 3, we observe the followng: The agent desgn wth both case-orented renforcement learnng and peer-related renforcement learnng (CRLPRL) outperformed all the other versons n terms of the success rates of coalton formaton. Ths shows that our agents were able to learn to mprove ther performance n coalton formaton. Wth learnng, the agents were able to form coaltons more successfully. Ths s based on the observaton that NoLearnng yelded the lowest success rate at 35%. Peer-related renforcement learnng (OnlyPRL) yelded better success rate than case-orented renforcement learnng (OnlyCRL), 48.75% to 40%. Ths ndcates that the peer-related renforcement learnng played a more sgnfcant role than the case-orented renforcement learnng n our envronment. In the dynamc, uncertan envronment, the tactcal, onlne renforcement learnng on peer agents behavor played a more sgnfcant role on the effectveness of coalton formaton than the strategc, off-lne renforcement learnng on the utltes of past coalton formaton cases. Impact of Learnng on Coalton Formaton Qualty Fgure 4 shows the mpact of learnng on the coalton formaton qualty, whch s an average of how well the coalton formaton processes were (e.g., n terms of the number of messages, tme spent vs. tme expected, etc.) and the qualty of the actual coalton formed (e.g., whether an expert peer contrbuted to the coalton, etc.). Average Qualty (%) CRLPRL OnlyPRL OnlyCRL NoLearnng Varous Learnng Fgure 4. Average coalton formaton qualtes for dfferent agent versons of learnng mechansms In general, we observe smlar patterns as n Fgure 3. CRLPRL outperformed all others wth an average qualty of 80%. NoLearnng and OnlyCRL were far behnd, hov-

7 erng around 40%. Ths clearly showed that even when there was a good plan (derved from case-based reasonng), the actual coalton formaton process or the coalton qualty mght not be better than that wthout a good plan (OnlyPRL, for example) n our envronment. In the dynamc, uncertan envronment, the tactcal, onlne renforcement learnng on peer agents behavor played a more sgnfcant role on the effcency of coalton formaton than the strategc, off-lne renforcement learnng on the utltes of past coalton formaton cases. Conclusons We have descrbed the use of renforcement learnng n a mult-phase coalton formaton model. We nvestgated multple aspects of the renforcement learnng applcaton n multagent coalton formaton. We have conducted several prelmnary experments and the results have been promsng n provng the feasblty of renforcement learnng. Wth renforcement learnng, our agents form coaltons more effectvely and effcently. Our future wor wll focus on desgnng further experments to test the mpact of renforcement learnng on the effectveness and effcency of coalton formaton based on dfferent degrees of heterogenety n the agents characterstcs. References Alonso, E., D Inverno, M., Kudeno, D., Luc, M., and Noble, J Learnng n Mult-Agent Systems. The Knowledge Engneerng Revew 16(3): Excelente-Toledo, C. B., and Jennngs, N. R Learnng to Select a Coordnaton Mechansm. In Proceedngs of the Frst Internatonal Jont Conference on Autonomous Agents and Mult-Agent Systems (AAMAS 2002), , Bologn Italy. Haynes, T., and Sen, S Learnng Cases to Resolve Conflcts and Improve Group Behavor. Internatonal Journal of Human-Computer Studes 48(1): Hu, J., and Wellman, M. P Onlne Learnng about Other Agents n a Dynamc Multagent System. In Proceedngs of the Second Internatonal Conference on Autonomous Agents, , Mnneapols, MN. Kaelblng, L.P., Lttman, M.L., and Moore, A. W Renforcement Learnng: A Survey. Journal of Artfcal Intellgence Research 4: Kohr, T., Matsubayash, K., and Tooro, M An Adaptve Archtecture for Modular Q-Learnng. In Proceedngs of the Ffteenth Internatonal Jont Conference on Artfcal Intellgence (IJCAI 1997), , Nagoy Japan. Nagayu, Y., Ish, S., and Doy K Mult-Agent Renforcement Learnng: An Approach Based on the Other Agent s Internal Model. In Proceedngs of the Fourth Internatonal Conference on Mult-Agent Systems, , Kyoto, Japan. Prasad, M. V. N., and Lesser, V. R The Use of Meta-Level Informaton n Learnng Stuaton-Specfc Coordnaton. In Proceedngs of the Ffteenth Internatonal Jont Conference on Artfcal Intellgence (IJCAI 1997), , Nagoy Japan. Salustowcz, R. P., Werng, M. A., and Schmdhuber, J Learnng Team Strateges: Soccer Case Studes. Machne Learnng 33(2/3): Sandholm, T. W Dstrbuted Ratonal Decson Mang. In G. Wess (Ed.), Multagent Systems: A Modern Approach to Dstrbuted Artfcal Intellgence , the MIT Press. Sandholm, T. W., and Crtes, R. H Multagent Renforcement Learnng n the Iterated Prsoner s Dlemmas. Bosystems 37: Sandholm, T. W., and Lesser, V. R Coalton Formaton amongst Bounded Ratonal Agents. In Proceedngs of the Fourteenth Internatonal Jont Conference on Artfcal Intellgence (IJCAI 1995), , Montreal, Canada. Sen, S., and Wess, G Learnng n Multagent Systems. In Wess, G. (ed.), Multagent Systems: A Modern Approach to Dstrbuted Artfcal Intellgence , the MIT Press. Shehory, O., and Kraus, S Methods for Tas Allocaton va Agent Coalton Formaton. Artfcal Intellgence 101: Soh, L.-K. and L, X An Integrated Mult-Level Learnng Approach to Multagent Coalton Formaton. In Proceedngs of the Eghteenth Internatonal Jont Conference on Artfcal Intellgence (IJCAI 2003), , Acapulco, Mexco. Soh, L.-K., and Tsatsouls, C Reflectve Negotatng Agents for Real-Tme Multsensor Target Tracng. In Proceedngs of the Seventeenth Internatonal Jont Conference on Artfcal Intellgence (IJCAI 2001), , Seattle, WA. Stone, P., and Veloso, M Multagent Systems: A Survey from a Machne Learnng Perspectve. Autonomous Robotcs 8(3): Sugawar T., and Lesser, V Learnng to Improve Coordnated Actons n Cooperatve Dstrbuted Problem-Solvng Envronments. Machne Learnng 33(2/3): Sycar K. P Multagent Systems. AI Magazne 19(2): Tan, M Mult-Agent Renforcement Learnng: Independent vs. Cooperatve Agents. In Proceedngs of the Tenth Internatonal Conference on Machne Learnng (ICML 1993), , Amherst, MA.