How Geo-distributed Data Centers Do Demand Response: A Game-Theoretic Approach

IEEE TRANSACTIONS ON SMART GRIDS 1 How Geo-dstrbuted Data Centers Do Demand Response: A Game-Theoretc Approach Nguyen H. Tran, Member, IEEE, Da H. Tran, Shaole Ren, Member, IEEE, Zhu Han, Fellow, IEEE, Eu-Nam Huh, Member, IEEE, and Choong Seon Hong, Senor Member, IEEE Abstract We study the demand response DR) of geodstrbuted data centers DCs) usng smart grd s prcng sgnals set by local electrc utltes. The geo-dstrbuted DCs are sutable canddates for the DR programs due to ther huge energy consumpton and flexblty to dstrbute ther energy demand across tme and locaton, whereas the prce sgnal s well-known for DR programs to reduce the peak-to-average load rato. There are two dependences that make the prcng desgn dffcult: ) dependency among utltes, and ) dependency between DCs and ther local utltes. Our proposed prcng scheme s constructed based on a two-stage Stackelberg game n whch each utlty sets a real-tme prce to maxmze ts own proft n Stage I; and based on these prces, the DCs servce provder mnmzes ts cost va workload shftng and dynamc server allocaton n Stage II. For the frst dependency, we show that there exsts a unque Nash equlbrum. For the second dependency, we propose an teratve and dstrbuted algorthm that can converge to ths equlbrum, where the rght prces are set for the rght demands. We also verfy our proposal by trace-based smulatons and results show that our prcng scheme sgnfcantly outperforms other baselne schemes n terms of flattenng the power demand over tme and space. Index Terms Demand response, Data centers, Smart grds, Stackelberg games, Nash equlbrum. I. INTRODUCTION Data centers DCs) are well-known as large-scale consumers of electrcty e.g. DCs consumed 1.5% of the worldwde electrcty supply n 2011 and ths fracton s expected to grow to 8% by 2020 [1]). A recent study shows that many DC operators pad more than $10M [2] on ther annual electrcty blls, whch contnues to rse wth the flourshng of cloudcomputng servces. Therefore, t s necessary for DC operators to both cut costs and ncrease performances. Recent works have shown that DC operators can save more than 5% 45% [3] operaton cost by leveragng tme and locaton dverstes of electrcty market prces to optmze geo-dstrbuted DCs. However, most of the exstng research s based on one mportant assumpton: the electrcty prce applyng to DCs does not change wth demand. Ths assumpton may not be true snce an ndvdual DC wth enormous energy consumpton e.g., Facebook s DC n Crook County, Oregon can contrbuted up to 50% of the total load of ts dstrbuton grd [4]) wll mpact N. H. Tran, D. H. Tran, E. Huh, and C. S. Hong are wth the Department of Computer Engneerng, Kyung Hee Unversty, Korea emal: {nguyenth, da.tran, johnhuh, cshong}@khu.ac.kr). S. Ren s wth School of Computng and Informaton Scences, Florda Internatonal Unversty, Florda, USA sren@cs.fu.edu). Z. Han s wth the Electrcal and Computer Engneerng Department, Unversty of Houston, Houston, Texas, USA emal: zhan2@uh.edu). to the supply-demand balance of ts local utlty, whch n turn can alter the utlty s prce as shown n recent studes [5] [7]. Furthermore, the power grd can be negatvely affected due to ths assumpton. For example, blackouts mght happen due to overloads n these areas where the DCs operator shfts all of ts energy demand to a local utlty wth a low prce and a hgh enough background load. To make the power grd more relable and robust, tremendous research and ndustry efforts have focused on buldng the next-generaton power grds, known as smart grds. Due to ts effcency and potental, many studes consder how DC operators can run ther geo-dstrbuted DCs on smart grds that support two-way nformaton exchange between utltes and customers [5], [8], [21]. An mportant feature of smart grds s demand response DR). DR programs seek to provde ncentves to nduce dynamc demand management of customers electrcty load n response to power supply condtons. For example, just before the peak load hours, a utlty can send the warnng sgnal to customers smart meters whch wll automatcally schedule ther demands to reduce the power consumpton. Due to ther huge and rapdly ncreasng energy consumpton, DCs should be sgnfcantly encouraged to partcpate n the DR programs. Furthermore, wth the recent trend n dynamc server capacty provson and flexblty of workload shftng, geo-dstrbuted DCs have a great potental to easly adapt the DR programs. One of the DR programs s usng real-tme prcng schemes to reduce the peak-toaverage PAR) load rato by encouragng customers to shft ther energy demand away from peak hours. The challenge of an effectve prcng scheme s how to charge the customers wth a rght prce not only at the rght tme and rght place but also on the rght amount of customers demand. A real-tme prcng scheme s consdered effectve f t can mtgate the large fluctuaton of energy consumpton between peak and offpeak hours to ncrease power grd s relablty and robustness. In ths paper, we consder the problem of usng realtme prcng of utltes to enable the geo-dstrbuted DCs partcpaton nto the DR program. In ths program, whle geo-dstrbuted DCs employ workload shftng and dynamc server provsonng n response to the prce sgnal, the role of local utltes s how to set the real-tme prces to flatten the customers demand load. It can be observed that there s an nteracton between geo-dstrbuted DCs and ther local utltes; and t s the frst challenge of ths DR problem that we call vertcal dependency. Specfcally, when partcpatng n the DR program, a DCs operator wll dstrbute ts energy demand geographcally based on the electrc prces adjusted

IEEE TRANSACTIONS ON SMART GRIDS 2 ntellgently by the local utltes. However, the utltes set ther prces based on the total demand ncludng the DCs demand, whch s only known when the prce s avalable. We clearly see that ths dependency makes t dffcult for both DCs and utltes to make ther decsons. The second mportant challenge, whch s less obvous, s an nteracton among local utltes feedng power to the geo-dstrbuted DCs; and we call t horzontal dependency. Specfcally, the DCs decsons on workload shftng and server allocaton depend on the electrc prces set by local utltes; therefore, f any sub-set of the local utltes change ther prces, t can lead to the DCs decson changng. Snce the utltes are noncooperatve.e., no nformaton exchange) n practce, how to desgn a prcng mechansm that can enable an equlbrum prce settng profle s the bottleneck of ths DR program. To tackle the above dscussed challenges, our contrbutons can be summarzed as follows We transform the functonal space of the geo-dstrbuted DCs DR program nto a mathematcal space of a formulated two-stage Stackelberg game. In ths game, each utlty wll set a real-tme prce to maxmze ts own proft n Stage I; and gven these prces, the DCs operator wll mnmze ts cost va workload shftng and dynamc server allocaton n Stage II. We also utlze the backward nducton method to fnd the Stackelberg equlbra of ths two-stage game. Based on the Stackelberg equlbra, our proposed scheme can deal wth the nherent challenges of ths DR as follows: Frst, the horzontal dependency between utltes are characterzed as a strategc game n Stage I, and we show that there exsts a Nash equlbrum n ths game. Second, we propose an teratve and dstrbuted algorthm to acheve the Stackelberg equlbrum. In ths algorthm, the DCs and utltes exchange ther nformaton.e. DCs demand and utltes prces) teratvely untl the algorthm converges. We also examne the algorthm s convergence where the rght prces are set for the rght demands as a soluton for the vertcal dependency ssue. Fnally, we perform a real-world trace-based smulaton to soldfy the analyss. The results show that our proposed prcng scheme can flatten the workload not only over tme but also over space to mprove the power grd s relablty and robustness. The rest of ths paper s organzed as follows. Secton II s about related work. Secton III presents the system model and the two-stage Stackelberg game. We analyze ths game and propose a dstrbuted algorthm n Secton IV. Secton V provdes the trace-based smulaton results and Secton VI concludes our work. II. RELATED WORK DR s ndentfed as one of hgh-prortzed areas for future smart grds [9] [11] wth ts potental to reduce up to 20% of the total peak electrcty demand of US [12]. Most DR proposals, whch try to ncentvze customers to manage ther demand dynamcally n response to the power supply condtons, mostly targeted to resdental customers [13] [16]. On the other hand, most of the exstng research on DCs, whch can be classfed as medum or large ndustral customers, manly focus on ther cost mnmzaton that takes the electrcty prce for granted [3], [17], [18], whch does not follow any DR programs. However, due to the mportant role of DCs n DR programs, DRs of DCs recently receve sgnfcant attenton [4], [7], [8], [19] [22]. For those work consderng DR of geo-dstrbuted DCs, based on the nteractons between DCs and utltes, we smply dvde them nto two categores. 1) One-way nteracton: One of the most popular DR programs of DCs s Concdent Peak Prcng CPP), whch s studed n [20]. CPP charges very hgh prces for power usage durng the concdent peak hour at whch the most electrc demands s requested to the utlty. By predctng the upcomng potental peak hours, the utltes send a warnng sgnal.e., not a prce) to help customers schedule ther power consumpton. However, current DCs do not respond actvely to the warnng sgnals due to the uncertanty of these warnngs [20], whch motvates researchers to devse more effectve DR approaches. The authors n [7] use a predcton-based method where the customers DCs) respond to the prces whch are chosen based on a supply functon. Ths supply functon can be modeled usng some data fttng methods based on hstory. Hence, n ths work only customers respond to a predcted prce whle there s no acton from the power supplers to set the prces correspondng to the demand. 2) Two-way nteracton: There are three recent papers [5], [8], [21] n ths category. The frst two papers, whch are hghly related to our work, consder dynamc prcng mechansms wth the couplng between utltes and DCs, whereas the last one proposed that DCs can partcpate n the spot market va a broker, whch s a sgnfcant departure from our model. Moreover, the system model of [8] assumes that all utltes cooperate to solve a socal optmzaton problem, whch s not relevant to current practce snce there s no nformaton exchange between utltes n realty. On the other hand, the prcng scheme of [21] s based on a heurstc approach, whch cannot maxmze the utltes proft as well as mnmze ther cost. Our work falls nto ths category of two-way nteracton, yet s dfferent from others n terms of ts two-stage game-theoretc approach to tackle the vertcal and horzontal couplng ssues, whch are not addressed n the lterature, between geo-dstrbuted DCs and local utltes. III. SYSTEM MODEL AND PROBLEM FORMULATION We consder one-perod demand response as n [17], [23], where ts duraton, whch s controlled by a utlty/load servng entty, matches an nterval at whch the DCs decsons and utltes real-tme prces can be updated such as 15 mnutes or 1 hour). Let I = {1,..., I} denote the set of stes wth dfferent electrcal utltes servce regons where DCs are located. Such geo-dstrbuted DCs are very common n practce, e.g., Google, Amazon, etc. Each DC s powered by a local utlty company and have S homogeneous servers. A DC wth heterogeneous types of servers can be vewed as multple vrtual DCs each havng homogeneous servers. For the ease of presentaton, Table I lsts key notatons of ths paper.

IEEE TRANSACTIONS ON SMART GRIDS 3 Demand Response of Data Centers usng Smart Grd Utlty Smart Meters Data Center Workload Dstrbuton Two-stage Stackelberg Game Stage I: Utlty proft maxmzaton Dstrbuted Algorthm Stage II: Data Centers cost mnmzaton Real-tme Prcng Game between Utltes Dynamc Server Provsonng Fg. 1. The functonal space of the geo-dstrbuted DCs demand response on the left and ts transformed mathematcal space as a two-stage Stackelberg game on the rght. Notaton I Λ λ d γ e d e b e ω TABLE I SUMMARY OF NOTATIONS Descrpton Number of tenants Total workload at front-end server Workload at DC Transmsson delay from front-end server to DC Weght of utlty cost Interactve-job energy Batch-job energy Total energy of DC Weght factor of mgraton cost p Prce at DC µ Servce rate of DC s B α, β D C Number of actve servers Background energy demand of utlty Parameters of background demand B model Maxmum average delay of DC Capacty of utlty We ncorporate the role of utlty nto the DR programs of DCs to regulate the power demand at each local ste for load balancng the power grd. We llustrate a functonal space and a mathematcal space of ths DR program n Fg. 1. In the functonal space, we leverage the dea of usng the advanced two-way communcaton of smart grd to facltate the nformaton exchange between utltes and DCs at each local ste va smart meters. Whle utltes set prces to ncentvze DCs to flatten the demand over tme and locatons to ncrease the power grd s relablty, as the prcetakers the DCs wll mnmze ther costs. In the mathematcal space, we observe that there exsts a specal mutual nteracton between DCs and utltes where utltes set prces based on the total demand, and DCs mnmze ther costs based on the prces. Therefore, we transform ths DCs DR program nto a leader-follower game that can be studed usng a two-stage Stackelberg game. Specfcally, the utltes are the leaders that set the prces to maxmze ther profts n Stage I and DCs wll make ther decsons on workload shftng and dynamc server provsonng to mnmze ther costs n Stage II. We present ths two-stage game formulaton n the reverse sequence, 1 startng wth Stage-II optmzaton problem. A. DCs Cost Mnmzaton n Stage II We frst descrbe the workload model of a typcal DC. We then elaborate the DCs cost focusng on the energy cost and delay cost model. Fnally, we formulate the Stage-II DC s cost mnmzaton. 1) Workload Model: Even though DCs can support a wde range of workloads, we generally classfy them nto two typcal types of workload: nteractve non-nterruptve) jobs and batch nterruptve) jobs. Whle the former s delaysenstve e.g. computng search, onlne game, etc.), the later s delay-tolerant e.g. backup tasks, MapReduce, etc.). We assume that each DC processes ts batch jobs locally.e. batch jobs cannot be re-drected to other DCs for load balancng) snce wthout strngent delay constrant, they are flexble to be scheduled across a large tme wndow at a local ste, lke [18]. For nteractve jobs, we denote the total arrval rate to the DCs front-end server,.e. all DCs are managed by a DCs servce provder DCs provder)) by Λ and ths front-end server s responsble for splttng the total ncomng workload Λ nto separate workloads of geo-dspersed DCs, denoted by {λ } I. Even though we only consder workload shftng, the other control knobs for DR such as power load reducton e.g., scalng down CPU frequences and/or turnng off unused servers) can also be ntegrated nto our framework. 2) DC s Cost and SLA Model: We assume that the DCs provder tres not only to mnmze ts energy cost and mgraton cost but also to guarantee the Servce Level Agreement SLA) requrements for the nteractve jobs. Energy Cost: Snce batch jobs are flexble to schedule n tme doman, batch jobs processng s consdered to consume an amount of energy e b of each DC wth ther dedcated servers. On the other hand, the energy consumpton of nteractve jobs at DC s [2] e d ) = s Pdle + P peak P dle )U + η 1)P peak 1) where s s the number of actve servers, µ s the servce rate of a server, P peak and P dle are the server s peak and dle power, respectvely, U = λ s µ s the average server utlzaton, and η s the power usage effectveness PUE) measurng the energy effcency of the DC. We can rewrte e d as follows e d = a λ + b s, I, 2) where a = P peak P dle )/µ and b = P dle +η 1)P peak. Therefore, denotng the total energy by e = e d + e b, 3) and gven a prce p, the energy cost of DC s e p. Mgraton Cost: Snce mgratng the workload from frontend server to geo-dstrbuted DCs can be very costly e.g. mgratng vrtual machnes or vdeo content requests over the Internet could be expensve due to reservng bandwdth from an ISP), we model the mgraton cost to DC as ωd c λ ), where d s the transmsson delay from the front-end server to DC, ω s a weght factor and c λ ) s a functon whch s assumed to be strctly ncreasng and convex. Snce d s

IEEE TRANSACTIONS ON SMART GRIDS 4 proportonal to the dstance, t s assumed to be a constant and we see that mgratng more requests from the front-end server to a more dstant DC s more costly. For analyss tractablty, we choose a quadratc functon c λ ) = λ 2 snce t s wdely used n many felds such as control, sgnal processng, communcaton networks, etc. to model a cost functon [24]. SLA Constrant: We assume that each delay-senstve request mposes a maxmum delay D that the DCs provder has to guarantee when shftng ths request to DC. Therefore, the SLA constrant n terms of delay guarantee can be modeled as follows 1 + d D,, 4) s µ λ where 1/s µ λ ) s the average delay tme of a request processed n DC wth arrval rate λ and servce rate s µ by queueng theory, whch has been wdely used as an analytc vehcle to provde a reasonable approxmaton for the actual servce process [18], [25]. 3) Problem Formulaton: Our model focuses on two key controllng knobs of DCs cost mnmzaton: the workload shftng to DC λ and the number of actve servers provsoned s at ste,. Then, the Stage-II DC cost mnmzaton s gven by DC : mnmze I e p + ωd λ 2 5) =1 subject to constrants 2), 3), 4), I λ = Λ, 6) =1 0 s S,, 7) 0 λ s µ,, 8) varables s, λ,. 9) Whle constrants 2), 3) and 4) are the defntons of the objectve functon and the SLA contrant, the remanng constrants are straght-forward. In 6), all of the ncomng workload must be served by some DCs. Moreover, 7) lmts the number of actve servers and 8) means that the total workload assgned to a DC must be less than ts capacty. Wth thousands of servers n a DC, we can further relax the nteger varables s as contnuous varables so that ths problem s tractable [17]. B. Non-Cooperatve Prcng Game n Stage I In ths stage, we frst present the market structure. We next descrbe the utlty s revenue and cost models and fnally formulate the non-cooperatve prcng game. 1) DR Retal Prce: Tradtonally utltes nvolve many complex electrcty markets. As buyers, utltes can partcpate n a wholesale market day-head, real-tme balancng) to buy electrcty from the generatng companes wth wholesale prces. As sellers, utltes make proft by sellng retal to ther customers wth proper retal rates [5]. Snce conventonal customers.e. no DR) have nelastc demand wth predctable patterns, utltes can predct and buy energy from wholesale Conventonal Retal Prce for non-dr customers Wholesale Prcng Utlty Retal Prcng DR Retal Prce for DR-enabled customer Fg. 2. Besdes conventonal wholesale and retal prcng, the utltes DR real-tme prcng s proposed for geo-dstrbuted DCs and other DR-enabled customers. market, then resell t at the conventonal retal rates. However, DCs wth workload shftng represent a new type of elastcdemand customers, whch makes utltes dffcult to predct ther demands, mpactng the grd s stablty. Therefore, we propose a new DR retal prcng scheme for utltes to serve the unpredcted and elastc customers, e.g. not just loadshftng DCs but also for all DR-enabled customers. The basc dea of ths scheme s that utltes and these DR-enabled customers can coordnate va smart-grd nfrastructure to match supply wth demand. Fg. 2 llustrates that utltes can apply the conventonal and DR retal prces to ther correspondng customers, whch are complementary to each other so that the proposed scheme wll not affect to the conventonal scheme, smlar to [8], [16], [26]. Snce conventonal markets and customers are orthogonal to our model, henceforth we only consder utlty s proft model and the proposed real-tme prcng scheme for DR-enable customers. 2) Utlty s Revenue and Cost Model: We see that the optmal energy consumpton of DCs that can be obtaned from solvng DC depends on all utltes prces. Denote the correspondng optmal power demand by e p), where p := {p } I. We further assume that due to the grd regulatons at each regon, the lower and upper bound of the real-tme prce should be mposed and denoted by p l and p u,, t, respectvely. Furthermore, besdes the power demand of DCs, each utlty has ts own background load e.g. resdental/commercal/ndustral demand). Snce there are consderable works focusng on the resdental DR programs, we assume that the background load of utlty, denoted by B p ), also responds to the prce and can be modeled by the followng functon B l, p p l ; B p) = α β p, p l p p u ; 10) B u, p p u, where B l and Bu are the mnmum of maxmum background demands of ste due to the physcal constrants of consumers.e. mnmum and maxmum power of electrc devces or vehcles). Ths functon, whch follows the lnear demand model n [27], shows an nherent response of customers to the prce: decrease the demand down to a lower-bound constrant when the prce ncreases, and vce versa, where β s the decreasng slope and α models the physcal upper-bound demand wthout prce. Based on the hstory of customer s usage data, utltes can estmate α and β usng some data 1

IEEE TRANSACTIONS ON SMART GRIDS 5 fttng methods, smlar to [7]. Based on the total power requested by DCs and background s demands, the revenue of utlty s gven by R p) = e p) + B p ) ) p. 11) On the other hand, every utlty ncurs a cost when t serves the customers load. When the load ncreases, the utlty s cost also ncreases snce normally blackouts happen due to overload, whch s a dsaster to any utlty. Hence, we can model the utlty s cost based on a wdely-used electrc load ndex ELI) as follows ) 2 C p) = γeli := γr 2 e p) + B p ) C = C, 12) where C s utlty capacty, γ reflects the weght of the cost, and r s a load rato that measures the power load levels. A very hgh r can rsk the utlty s stablty. ELI s motvated by the ndex measurement technques used for load flattenng n a power grd [8], [28], [29]. We see that ELI can weght dfferent utltes load rato r by ther capactes, provdng feeder load-balancng capablty. On the other hand, a utlty wth hgh γ shows that t s more concerned about the effect of ELI to the relablty, whle a utlty wth low γ has more nterest n makng revenue and less concerned about the nstablty s threat. 3) Stage-I Prcng Game Formulaton: In realty, the geodstrbuted utltes usually have no communcaton exchange to optmze the socal performance. Instead, each utlty has ts own goal to maxmze ts proft, whch s defned as the dfference between revenue and cost as follows C u p, p ) = R p) C p), 13) where p denotes the prce vector of other utltes except. Ths notaton comes from an observaton that there s a game between utltes because the proft of each utlty not only depends on ts energy prce but also on the others. Hence, the Stage-I utlty proft maxmzaton game, denoted by UP = I, {p } I, {u } I ), s defned as follows Players: the utltes n the set I; Strategy: p l p p u, I; Payoff functon: u p, p ), I. IV. TWO-STAGE STACKELBERG GAME: EQUILIBRIA AND ALGORITHM In ths secton, we frst apply the backward nducton method to solve the Stackelberg game. Then, we propose an teratve algorthm to reach an equlbrum of ths game. A. Backward Inducton Method 1) Optmal Solutons at Stage II: We realze that the Stage-II DCs cost mnmzaton can be decomposed nto ndependent problems. Henceforth, we only consder a specfc tme perod and drop the tme dependence notaton for ease of presentaton. In ths stage, DCs cooperate wth each other to mnmze the total cost by determnng the workload allocaton λ and the number of actve servers s at each DC. It s easy to see that the DCs cost mnmzaton s a convex optmzaton problem. Frst, we observe that constrant 4) must be actve because otherwse the DCs provder can decrease ts energy cost by reducng s. Hence, we have 4) s equvalent to [ 1 s λ ) = λ + µ D )] S 1, 14) where [.] y x s the projecton onto the nterval [x, y] and D := D d. In practce, most DCs can have a suffcent number of servers to serve all requests at the same tme due to the lluson of nfnte capacty of DCs [17]. Therefore, we adopt s λ ) = 1 µ λ + D ) 1 n the sequel. By substtutng ths s λ ) nto the objectve of DC, we have an equvalent problem DC as follows DC : mn. λ s.t. I I 0 =1 f λ ) 15) λ = Λ, =1 16) λ 0,, 17) where f λ ) := ωd λ 2 + p a + b ) λ + p e b + b D ) 1. µ µ It can be seen that DC s a strctly convex problem, whch has a unque soluton. Snce DCs provder lkes to have λ > 0,, n order to utlze all DCs resources, we characterze the unque soluton of DC and a necessary condton to acheve ths soluton wth the optmal λ > 0,, as the followng result. Lemma 1. Gven a prce vector p, we have the unque solutons of Stage-II DC problem: λ = ν p A > 0, and s = 1 λ + 2ωd µ D ) 1,, only f 18) ω > ωth 1 := ˆd max {p A } I ) p A /d /2Λ, 19) =1 where ˆd := I =1 1/d, A := a + b µ and ν = 2ωΛ + I =1 p A /d ). 1ˆd Snce all parameters to calculate ωth 1 are avalable to DC, we can consder condton 19) as a gudelne for a DCs provder to choose an approprate weght factor ω to ensure that all DCs have postve request rates. 2) Nash Equlbrum at Stage I: We contnue to characterze the Nash equlbrum of the Stage-I game based on the Stage-II solutons. From 13), we have ) ) e u p, p ) = e 2 p) + B p ) p γc p) + B p ), C 20)

IEEE TRANSACTIONS ON SMART GRIDS 6 The nteractons between front-end server, local DCs and local utltes where e p) = a λ + b s ) + eb wth λ and s obtaned from Lemma 1) and can be presented as follows e p, p ) = 21) A 2 p 1 1) + A A j p j 2ωd ˆdd 2ω ˆdd + A Λ + b + e d j ˆdd b. µ D j In the non-cooperatve game, one of the most mportant questons s whether there exsts a unque Nash equlbrum. In the case of Stage-I game, we have the followng defnton of a Nash equlbrum. Defnton 1. A prce vector p e := {p e } I s sad to be a Nash equlbrum f no utlty can mprove ts proft by unlaterally devatng ts prce from the Nash equlbrum: u p e, p e ) u p, p e ), p l p p u,. 22) Theorem 1. Exstence) There exst a Nash equlbrum of the Stage-I UP game. In ths Stage-I game, gven all other utltes strateges p, a natural strategy of utlty s the best response strategy as follows BR p ) = arg max u p, p ),, 23) p P where P := [ p l, ] pu. In order to fnd the best response, we set up) p = 0. Then, the teratve best response updates can be obtaned as follows ) ) p k+1) = BR p k) = 1/2 γn h p /C k),, 1 γn /C N ) P 24) where [.] P denotes the projecton onto P, k represents the teratons, N := A2 2ωd ˆdd 1 1) β, and hp ) := A 2ω ˆdd j A j p j d j + A Λ ˆdd + b µ D + e b + α,. 25) When all utltes play best response strateges, a Nash equlbrum p e s a profle that satsfes p e = BR p e ),,.e. every utlty s strategy s ts best response to others strateges. However, there are two ssues here: ) There s no condton for general games such that the best responses converge to a Nash equlbrum; ) Snce multple Nash equlbra can exst n the UP game, how the best response can converge to a unque Nash equlbra. Hence, we next examne the convergence property of the best response 24) to a unque Nash equlbrum by usng the concept contracton mappng. We brefly ntroduce contracton mappng and ts propertes, all of whch can be found n chapter 3 of [30]. Snce many teratve algorthms have the form x k+1) = Tx k) ), k = 0, 1,..., where x k) X R n, the mappng T : X X s called a contracton f there s a scalar 0 σ < 1 such that Tx) Ty) σ x y, x, y X, 26) Utlty 1 Locaton 1 DC 1 e 1 p) p 1 Utlty 2 Locaton 2 p 2 DC 2 Internet e 2 p) p 1,, p I ) e 1 p),, e I p) Front-end Server Λ p I e I p) DC I Locaton I Utlty I Fg. 3. Detaled operatons of Algorthm 1, where red arrows represent steps Fgure 9: The nteractons between front-end server, local DCs and utltes 3 and 5 and blue arrows correspond to step 4. where. s some norm defned on X. Furthermore, the mappng T s called a pseudo-contracton f T has a fxed pont x X.e. x = Tx )) and Tx) x σ x x, x X. 27) Both contracton and pseudo-contracton have the geometrc convergence rate property: suppose the mappng T has a fxedpont, the sequence {x k) } generated by x k+1) = Tx k) ) converges to a unque fxed pont x geometrcally satsfyng x k) x σ k x 0) x, k 0, 28) wth any ntal value x 0) X. Based on the above propertes of contracton mappng and Theorem 1, f we can show that the best response update 24) s a contracton mappng, then we can guarantee ts convergence to a unque Nash equlbrum. Therefore, we establsh the followng suffcent condton. Theorem 2. Convergence and Unqueness) If { A ω ωth 2 j := max A j/d j A 2 ˆd1 } 1/d ˆd)), 2β ˆdd 29) then startng from any ntal pont, the best response updates 24) of the Stage-I UP game s a contracton mappng that converges to a unque Nash equlbrum p e geometrcally. B. Dstrbuted Algorthm We frst descrbe the detaled operatons of the proposed algorthm. Next, we dscuss practcal mplementaton ssues of the algorthm. 1) Proposed Algorthm s Operatons and Convergence: We contnue proposng a dstrbuted algorthm, shown n Algorthm 1 Alg. 1), whch can acheve the Nash equlbrum. The detaled operatons of Alg. 1 are llustrated n Fg. 3. We assume that Alg. 1 operates at the begnnng of each prcng update perod.e. one hour) and the algorthm runs for many teratons communcaton rounds wth a parameter k) untl t converges to a prce settng equlbrum. Here, based on the total ncomng workload, the front-end server of the DCs provder frst collects all prces from ts local DCs and

IEEE TRANSACTIONS ON SMART GRIDS 7 Algorthm 1 Demand Response of Data Center wth Real-tme Prcng 1: ntalze: k = 0, ɛ s arbtrarly small, p 0) ω satsfes 29); 2: repeat 3: Utlty broadcasts ts p k) = p u,, and to all customers; 4: The front-end server collects p k) from all DCs, updates e p)k) as 21) and sends t back to DC, ; 5: Each DC reports ts e p)k) to the local utlty; 6: Utlty receves the demand responses from the local DC e p)k) and background users B p) k), then updates p k+1) = BR p k) ) as 24); 7: untl p k+1) p k) < ɛ. calculates the optmal energy consumpton as 21) step 4). After that, the front-end server wll feedback these energy consumpton data to ts local DCs, whch then forwards ts own nformaton to the local utlty step 5). Each utlty solves ts own proft maxmzaton problem best response updates) to fnd an optmal prce, then broadcasts ths prce to ts local DCs and background customers step 6). The process repeats untl the game converges to the unque Nash equlbrum accordng to Theorem 2 step 7). At ths state the prce settng s fnalzed and appled to the whole consdered perod. Even though Alg. 1 s presented n a scalable and synchronous fashon.e. all local utltes update and broadcast ther prces at the same tme), asynchronous dstrbuted algorthm s preferred snce n realty, the message-passng among front-end server, DCs and utltes usually ncurs heterogeneous delays. Fortunately, wth condton 29), Alg. 1 can also work asynchronously snce 29) s derved from establshng a contracton mappng wth respect to a maxmum norm., whch guarantees the asynchronous convergence of the mappng sequence [30] pp. 431). 2) Practcal Issues and Implementaton Dscusson: We dscuss two ssues here: the workload shftng assumpton and the message-passng. In terms of the former, we assume the DCs provder deploys a front-end server to dstrbute the ncomng workload to DCs. Ths can be done by usng varous practcal solutons such as ncorporatng the authortatve DNS servers whch s used by Akama) or HTTP ngress proxes whch s used by Google and Yahoo) nto the front-end servers. Furthermore, n realty there s only a sub-set of DCs to whch a workload type can be routed to due to the avalablty resource constrant of each DC. Ths ssue can be easly addressed by ncorporatng addtonal constrants nto our model such as [32], and n practce we can mplement t by classfyng the workload types at the front-end server before routng. In terms of the later, we assume that the two-way communcaton between a DC and ts local utlty can be enabled va communcaton networks of future smart grd. Regardng to the communcatons between DCs and ts front-end server, a DC reports ts utlty s prce by choosng one of the egress lnks of ts Internet Servce Provder ISP) to send ts packet through the Internet to the front-end server, and vce versa. Specfcally, the total tme of one teraton conssts of the transmsson tme and computatonal tme. Whle the transmsson tme from utltes to DCs and vce versa) s from 1 to 10 ms over a broadband speed of 100 Mbps, t s from 50 to 100 ms for a one-way communcaton between DCs and the frontend servers over a current ISP s path. The computatonal tme depends on the processng power of the front-end server and smart meters on calculatng the optmal energy 21) and maxmzng the convex proft functon 21), whch are both low-complexty problems and can be n the tme-scale of mcrosecond [24]. V. TRACE-BASED SIMULATIONS In ths secton, we conduct trace-based smulatons, mplemented n the Python language wth exstng lbrares ncludng NumPy, ScPy, and Matplotlb, to valdate our analyss and evaluate the performance of Alg. 1. A. Setups We consder sx geo-dstrbuted DCs powered by ther local utltes at the followng ordered locatons: 1. The Dalles, OR; 2. Councl Bluffs, IA; 3. Mayes County, OK; 4. Lenor, NC; 5. Berkeley County, SC and 6. Douglas County, GA. These locatons correspond to real Google s DCs [33]. All DCs PUEs are set to 1.5 over tme perods. The homogeneous servers have peak power of 200 W and dle power of 100 W, and the servce rate of each server s chosen unformly between 1.1 and 1.2. The mgraton weght ω s set to 1 unless otherwse stated. The delay SLA D are dstrbuted unformly between 100 and 300 ms and d s scaled by the vector [1.9, 1.0, 1.3, 2.5, 2.8, 2.3] n whch we assume that the front-end server s placed at Colorado. We use realstc traces for the ncomng workload Λ at the front-end server and the power demand of delay-tolerant batch jobs e b at each DC. All of them are scaled wth respectve to servce rates. We use an nteractve workload trace collected from Mcrosoft Research MSR) [34]. The workload can be predcted to a farly reasonable accuracy usng, e.g., regresson technques [3], [34]. Furthermore, we use Google trace for the power demand of delay-tolerant batch jobs e b n recent study [35]. The batch job power demand and workload seres spans over 30 days correspondng to a typcal utlty bllng cycle and each pont of seres s a one-hour perod. Snce lackng the publc nformaton of local utltes, we assume that all utltes have the capactes C unformly n the range of 25 and 30 MW, whch s a standard measure for a medum-sze utlty. Whle γ s set to 1 unless otherwse stated, α and β parameters are chosen unformly n the range of [25, 30] and [0.25, 0.30], respectvely. We consder two baselne prcng schemes for comparson. The frst baselne s based on the proposed dynamc prcng scheme of [21], whch s brefly descrbed as follows p t + 1) = δp D t) P S t)) + p t), 30) where P D and P S are the power demand and supply of utlty. We set δ to 0.5 n all smulaton scenaros. Ths baselne serves as a recent related benchmark.

IEEE TRANSACTIONS ON SMART GRIDS 8 Prces $/MW) 200 150 100 50 0 200 150 100 50 0 Fg. 4. Compared prces at sx locatons. The Dalles, OR Mayes County, OK 200 Berkeley County, SC 150 100 50 0 0 100 200 300 400 500 600 700 Hour Councl Bluffs, IA Lenor, NC Douglas County, GA Baselne 1 Baselne 2 Alg. 1 0 100 200 300 400 500 600 700 Hour 1.0 0.8 0.6 0.4 0.2 0.0 1.0 0.8 0.6 0.4 0.2 Proporton of local DCs demand over utltes demand Baselne 1 Baselne 2 Alg. 1 0.0 0 10 20 30 40 50 60 70 80 90 Hour Fg. 5. Proporton of local DC demand over utlty total demand at Mayes County above) and Lenor below). TABLE II AVERAGE OPTIMAL PRICES COMPARISONS WITH γ EFFECT Stes MSR Baselne Alg. 1 Alg. 1 Alg. 1 1 γ = 1 γ = 4 γ = 8 1 40.32 20.75 21.20 21.77 2 88.23 35.74 36.66 37.81 3 62.78 28.32 29.01 29.88 4 31.82 17.02 17.34 17.75 5 28.54 15.82 16.10 16.45 6 33.96 18.02 18.38 18.83 The second baselne s based on the Google s contract wth ther local utltes. Accordng to the emprcal study n [33], there are sx Google s DCs at sx mentoned locatons, where Google s DCs are nfered to have long-term contracts wth ther local utltes as the followng fxed rates.e. energy charges) [32.57, 42.73, 36.41, 40.68, 44.44, 39.97] $/MWh, respectvely. Ths baselne serves as an n-realty benchmark. We manly use ths baselne for the PAR comparsons snce ) the Google long-term contract often negotates a monthly electrcty bll scheme that combnes energy charges and DCs cost $) Utltes proft $) 5000 4500 4000 3500 8500 8000 7500 7000 6500 6000 5500 5000 4500 4000 1 2 3 4 5 6 7 8 γ Fg. 6. Effect of γ to average DCs cost and utltes proft. γ Baselne 1 Alg. 1 demand charges that we do not know exactly, whch can then nfluence the DCs cost and utltes proft, and ) t s not far to compare a dynamc prcng scheme to a snapshot statc prcng scheme n terms of cost and proft. B. Results We frst provde the sample-path optmal prces of three schemes at sx locatons n Fg. 4. In all perods, we observe that Alg. 1 can converge n less than ten teratons, where the stoppng condton ɛ = 10 4. Snce Baselne 1 and Alg. 1 employ dynamc prcng mechansms, we observe that the utltes prces of these two schemes vary accordng to the workload pattern. We also observe the effect of mgraton cost to the optmal prces n ths fgure. Snce the nearest DCs to the front-end server are stes 2 and 3, Fg. 4 shows that all dynamc prcng schemes set hgh prces at these two stes compared wth the other stes. Ths can be explaned as follows, due to the small mgraton cost at these stes whch leads to hgh demand, the dynamc schemes set hgh prces to balance between energy cost and mgraton cost. Furthermore, we observe that Alg. 1 can contrbute less load to utltes than other schemes do most of the tme; for example, ths can be seen n Fg. 5 that shows the proporton of DCs demand over utltes total demand varatons n three days at two stes.

IEEE TRANSACTIONS ON SMART GRIDS 9 PAR PAR PAR 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 2.44 2.40 1.66 The Dalles, OR 2.44 The Dalles, OR 2.44 2.40 2.40 1.67 1.67 The Dalles, OR 3.59 3.25 2.66 Councl Bluffs, IA 3.59 3.25 Councl Bluffs, IA 3.59 3.25 2.68 2.69 Councl Bluffs, IA 3.03 2.86 2.23 2.06 Mayes County, OK Lenor, NC 3.03 a) γ = 1 2.86 2.24 2.06 Mayes County, OK Lenor, NC 3.03 b) γ = 4 2.86 2.25 2.06 Mayes County, OK Lenor, NC c) γ = 8 2.27 2.23 1.96 1.52 1.49 2.17 Baselne 1 Baselne 2 Alg 1 2.35 1.58 Berkeley County, SC Douglas County, GA 2.27 2.23 1.96 1.51 1.48 2.17 Baselne 1 Baselne 2 Alg 1 2.35 1.57 Berkeley County, SC Douglas County, GA 2.27 2.23 1.96 1.49 1.46 2.17 Baselne 1 Baselne 2 Alg 1 2.35 1.55 Berkeley County, SC Douglas County, GA Fg. 7. PAR wth respect to MSR trace at sx locatons wth dfferent γ. Furthermore, we also nvestgate the effect of γ to the prcng schemes. Table II shows that f we ncrease γ, then the Alg. 1 s optmal prces also ncrease snce the hgher the weght utltes ELI cost factor s, the more conservatve utltes are n terms of relablty by rasng the prces. Fnally, we can see that Baselne 1 always overprces Alg. 1 n all scenaros snce Baselne 1 s more aggressve than Alg. 1 n terms of balancng the supply and demand. However, t could lead to hgh demand fluctuatons.e. hgh PAR) as shown n the followng results. We also observe that the average prces of Alg. 1 are not affected by ω. We also evaluate the effect of parameter γ to average DCs cost and utltes proft n Fg. 6. Frst, we can see that Baselne 1 wth hgher prces has hgher DCs cost and utltes proft than those of Alg. 1. In detals, the share of DCs energy cost of Alg.1 s 36.3%, 37.8%, and 38.7% when γ = 1, 4, and 8, respectvely, whereas that of Baselne 1 wthout γ mpact) s 44.8%. Therefore, Alg. 1 can gve more ncentves to encourage the DCs to jon the DR program. Second, we can see that when γ ncreases, the utltes proft of both schemes decrease accordng to 20). Snce the prcng scheme of Baselne 1 s ndependent wth γ, we can see that γ has no effect to the DCs cost of ths baselne. However, we see that DCs cost of Alg. 1 ncreases when γ ncreases due to the correspondng ncrease of the optmal prces c.f. 20)). Wth Alg. 1, we see that small γ s favorable because t can provde low DCs cost and hgh utltes proft. Furthermore, due to the background demand, we see that DCs cost ncludng the mgraton cost s lower than utltes proft. The fnal factor that we examne s the power demand PAR at each ste, whch s one of the most mportant metrcs to measure the effectveness of desgns for smart grd snce the fluctuaton of energy consumpton between peak and off-peak hours ndcate power grd s relablty and robustness. PAR s calculated as maxt{e pt))+bpt))}t T. Reducng PAR s the t=1 e pt))+bpt)) mportant goal of any DR program desgns. Therefore, we extensvely compare the PAR of three schemes wth dfferent γ n Fgs. 7a, 7b, and 7c. The most mportant observaton s that PAR s performance of Alg. 1 outperforms those of other schemes, ether statc or dynamc prcng, over tme and space sgnfcantly. Specfcally, consderng the case γ = 1, Fg. 7a shows that for all stes 1 to 6, Alg. 1 can acheve the lowest PAR value as expected, reducng the PAR to 32.3%, 27.0%, 28.1%, 28.0%, 25.8%, and 29.4% compared to Baselne 1, and 31.6%, 16.7%, 22.2%, 33.5%, 34.0% and 34.0% compared to Baselne 2, respectvely. We conclude that Alg. 1 can spread out the demand not only over tme but also over locatons. VI. CONCLUSION AND FUTURE WORK We have nvestgated the demand response of geodstrbuted data centers wth the help of emergence technques of smart grd. We frst characterze the challenged dependences of ths geo-dstrbuted DCs DR program where a utlty decsons not only depends on that of DCs, and vce versa, but also mpacts on other utltes decsons. We then formulate ths DR program nto a two-stage game to model these dependences. In ths game, the role of each utlty s settng a prce to maxmze ts proft, whle the DCs mnmze ts cost by workload shftng and dynamc server allocaton. We then characterze the exstence and unqueness of the Nash equlbrum of ths game, and develop an teratve and dstrbuted algorthm to reach ths equlbrum. By usng tracebased smulatons, we valdate and complement our proposal wth the smulaton results, whch shows that our prcng schemes based on the two-stage game can flatten the energy demand of DCs over tme and locatons to ncrease the power grd s relablty and robustness. APPENDIX A PROOF OF THEOREM 1 Snce the strategy space of each utlty s a nonempty compact and convex subset of Eucldean space, t s suffcent for us to show that the contnuous functon u p, p ) on ths

IEEE TRANSACTIONS ON SMART GRIDS 10 strategy space s a quas-concave functon,, such that there exsts a Nash equlbrum for Stage-I game [36]. From 21) and 10), t can be seen that e p) and B p ) are affne functons of p. Therefore, e p) + B p )) 2 s a convex functon [24]. Furthermore, we have 2 B p )p ) = β < 0, and 2 e p)p) = A2 p 2 1 1 < 0,, snce ˆdd ˆdd > 1,. Hence, both e p)p and B p )p are concave functons. Therefore, from 20) we see that u p, p ) s the sum of two concave functons so that s also a concave and hence quas-concave as well) functon. 2ωd APPENDIX B PROOF OF THEOREM 2 We frst seek the condton such that the best response update 24) s a contracton mappng. Defne a Cartesan product space P = Π I P and a vector BRp) := BR p )) I. Snce BRp) s contnuous and dfferentable on by P, by the mean value theorem, we have BRp 1 ) BRp 2 ) = BRp) p ) p 2 p 1 p 2, 31) p 1, p 2 P and p s on the segment connectng p 1 and p 2. Furthermore, the Jacoban BRp) p s as follows BR p ) p j = { 0, j = ; 1/2 γn /C A A j N )1 γn /C ) 2ω ˆdd, d j j. Then, by usng the norm. of the Jacoban, from 26) and 31), we see that 24) s a contracton mappng when BRp) p = { } max 1/2 γn /C A A j j N )1 γn /C ) 2ω ˆdd d < 1. 32) j It s straghtforward { to see that the } suffcent condton to satsfy 32) s max 1, whch s equvalent to ω max A 2ω ˆdd N j A j d j { A j A j/d j A 2 ˆd1 } 1/d ˆd)). 33) 2β ˆdd We have shown that wth condton 33), the best response update s a contracton mappng. Furthermore, accordng to Theorem 1, we have the exstence of a fxed-pont of the mappng 24). Hence, based on the convergence property of contracton mappng, we complete the proof. REFERENCES [1] J. Koomey, Growth n data center electrcty use 2005 to 2010, Oakland, CA Anal. Press. August, vol. 1, 2011. [2] A. Quresh, R. Weber, H. Balakrshnan, J. Guttag, and B. Maggs, Cuttng the electrc bll for nternet-scale systems, n Proc. ACM SIGCOMM, Barcelona, Span, Aug. 2009, pp. 123 134. [3] L. Rao, X. Lu, L. Xe, and W. Lu, Mnmzng electrcty cost: Optmzaton of dstrbuted Internet data centers n a mult-electrctymarket envronment, n Proc. IEEE INFOCOM, San Dego, CA, Mar. 2010. [4] Z. Lu, I. Lu, S. Low, and A. Werman, Prcng data center demand response, ACM SIGMETRICS Perform. Eval. Rev., vol. 42, no. 1, pp. 111 123, Jun. 2014. [5] J. Camacho, Y. Zhang, M. Chen, and D. M. Chu, Balance your bds before your bts: the economcs of geographc load-balancng, n e- Energy 14, New York, NY, Jun. 2014, pp. 75 85. [6] A. H. Mahmud and S. Ren, Onlne capacty provsonng for carbonneutral data center wth demand-responsve electrcty prces, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 2, p. 26, Aug. 2013. [7] P. Wang, L. Rao, X. Lu, and Y. Q, D-Pro: Dynamc data center operatons wth demand-responsve electrcty prces n smart grd, IEEE Trans. Smart Grd, vol. 3, no. 4, pp. 1743 1754, Dec. 2012. [8] H. Wang, J. Huang, X. Ln, and H. Mohsenan-Rad, Explorng smart grd and data center nteractons for electrc power load balancng, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 3, pp. 89 94, Jan. 2014. [9] NIST Framework and Roadmap for Smart Grd Interoperablty Standards, 2010. [10] X. Fang, S. Msra, G. Xue, and D. Yang, Smart Grd The New and Improved Power Grd: A Survey, IEEE Commun. Surv. Tutorals, vol. 14, no. 4, pp. 944 980, Jan. 2012. [11] F. Rahm, S. Member, I. Eee, and A. Ipakch, Demand response as a market resource under the smart grd paradgm, IEEE Trans. Smart Grd, vol. 1, no. 1, pp. 82 88, 2010. [12] Natonal assessment of demand response potental. 2009. [13] W. Saad, Z. Han, H. Poor, and T. Basar, Game-Theoretc Methods for the Smart Grd: An Overvew of Mcrogrd Systems, Demand-Sde Management, and Smart Grd Communcatons, IEEE Sgnal Process. Mag., vol. 29, no. 5, pp. 86 105, Sep. 2012. [14] A.-H. Mohsenan-Rad, V. W. S. Wong, S. Member, J. Jatskevch, R. Schober, and A. Leon-Garca, Autonomous Demand-Sde Management Based on Game-Theoretc Energy Consumpton Schedulng for the Future Smart Grd, IEEE Trans. Smart Grd, vol. 1, no. 3, pp. 320 331, Dec. 2010. [15] Z. Baharloue, M. Hashem, H. Narman, and H. Mohsenan-Rad, Achevng Optmalty and Farness n Autonomous Demand Response: Benchmarks and Bllng Mechansms, IEEE Trans. Smart Grd, vol. 4, no. 2, pp. 968 975, Jun. 2013. [16] N. L, L. Chen, and S. H. Low, Optmal demand response based on utlty maxmzaton n power networks, n IEEE Power Energy Soc. Gen. Meet., San Dego, CA, Jul. 2011, pp. 1 8. [17] Z. Lu, M. Ln, A. Werman, S. H. Low, and L. L. H. Andrew, Greenng geographcal load balancng, n Proc. ACM SIGMETRICS, San Jose, CA, Jun. 2011, pp. 233 244. [18] S. Ren and Y. He, COCA: onlne dstrbuted resource management for cost mnmzaton and carbon neutralty n data centers, n Proc. SC13, Denver, Colorado, Nov. 2013. [19] G. Ghatkar, V. Gant, N. Matson,, and M. A. Pette, Demand Response Opportuntes and Enablng Technologes for Data Centers: Fndngs From Feld Studes, Tech. Rep., Apr. 2012. [20] Z. Lu, A. Werman, Y. Chen, B. Razon, and N. Chen, Data center demand response: avodng the concdent peak va workload shftng and local generaton, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 1, p. 341, Jun. 2013. [21] Y. L, D. Chu, C. Lu, and L. T. X. T. X. Phan, Towards dynamc prcng-based collaboratve optmzatons for green data centers, n IEEE ICDEW, Brsbane, Australa, 2013, pp. 272 278. [22] A. Werman, Z. Lu, and H. Mohsenan-Rad, Opportuntes and Challenges for Data Center Demand Response, n IEEE IGCC, 2014. [23] M. Ghamkhar and H. Mohsenan-Rad, Proft maxmzaton and power management of green data centers supportng multple slas, n IEEE ICNC, Jan. 2013, pp. 465 469. [24] S. Boyd and L. Vandenberghe, Convex Optmzaton. Cambrdge Unversty Press, Mar. 2004. [25] Z. Lu, M. Ln, A. Werman, S. H. Low, and L. L. H. Andrew, Geographcal load balancng wth renewables, ACM SIGMETRICS Perform. Eval. Rev., vol. 39, no. 3, pp. 62 66, Dec. 2011. [26] L. P. Qan, Y. J. A. Zhang, J. Huang, and Y. Wu, Demand Response Management va Real-Tme Electrcty Prce Control n Smart Grds, IEEE J. Sel. Areas Commun., vol. 31, no. 7, pp. 1268 1280, Jul. 2013. [27] A. Mas-Colell, M. Whnston, and J. Green, Mcroeconomc Theory. Oxford Unversty Press, 1995. [28] Whe-Mn Ln and Hong-Chan Chn, A current ndex based load balance technque for dstrbuton systems, n IEEE POWERCON, vol. 1, 1998, pp. 223 227. [29] M. Kashem and V. Ganapathy, Three-phase load balancng n dstrbuton systems usng ndex measurement technque, Int. J. Electr. Power Energy Syst., vol. 24, no. 1, pp. 31 40, Jan. 2002.

IEEE TRANSACTIONS ON SMART GRIDS 11 [30] D. P. Bertsekas and J. N. Tstskls, Parallel and dstrbuted computaton: numercal methods. Prentce-Hall, Inc., Jan. 1989. [31] N. H. Tran, S. Ren, Z. Han, S. man Jang, S. I. Moon, and C. S. Hong, Demand Response of Data Centers: A Real-tme Prcng Game between Utltes n Smart Grd, n USENIX Feed. Comput. Work., Phladelpha, PA, 2014. [32] P. X. Gao, A. R. Curts, B. Wong, and S. Keshav, It s not easy beng green, n Proc. ACM SIGCOMM, Helsnk, Fnland, Aug. 2012, pp. 211 222. [33] H. Xu and B. L, Reducng electrcty demand charge for data centers wth partal executon, n Proc. ACM e-energy 14, Cambrdge, UK, Jun. 2014, pp. 51 61. [34] M. Ln, A. Werman, L. L. H. Andrew, and E. Thereska, Dynamc rghtszng for power-proportonal data centers, n Proc. IEEE INFOCOM, Shangha, Chna, Apr. 2011, pp. 1098 1106. [35] C. Wang, B. Urgaonkar, and Q. Wang, Data center cost optmzaton va workload modulaton under real-world electrcty prcng, Arxv Prepr. arxv1308.0585, pp. 1 14, 2013. [36] J. B. Rosen, Exstence and Unqueness of Equlbrum Ponts for Concave N-Person Games, Econometrca, vol. 33, no. 3, pp. 520 534, Jul. 1965. Zhu Han S 01-M 04-SM 09-F 14) receved the B.S. degree n electronc engneerng from Tsnghua Unversty, n 1997, and the M.S. and Ph.D. degrees n electrcal engneerng from the Unversty of Maryland, College Park, n 1999 and 2003, respectvely. From 2000 to 2002, he was an R&D Engneer of JDSU, Germantown, Maryland. From 2003 to 2006, he was a Research Assocate at the Unversty of Maryland. From 2006 to 2008, he was an assstant professor n Bose State Unversty, Idaho. Currently, he s an Assocate Professor n Electrcal and Computer Engneerng Department at the Unversty of Houston, Texas. Hs research nterests nclude wreless resource allocaton and management, wreless communcatons and networkng, game theory, wreless multmeda, securty, and smart grd communcaton. Dr. Han s an Assocate Edtor of IEEE Transactons on Wreless Communcatons snce 2010. Dr. Han s the wnner of IEEE Fred W. Ellersck Prze 2011. Dr. Han s an NSF CAREER award recpent 2010. Dr. Han s IEEE dstngushed lecturer snce 2015. Nguyen H. Tran S 10-M 11) receved the BS degree from Hochmnh Cty Unversty of Technology and Ph.D degree from Kyung Hee Unversty, n electrcal and computer engneerng, n 2005 and 2011, respectvely. Snce 2012, he has been an Assstant Professor n the Department of Computer Engneerng, Kyung Hee Unversty. Hs research nterest s usng queueng theory, optmzaton theory, control theory and game theory to desgn, analyze and optmze the cuttng-edge applcatons n communcaton networks, ncludng cogntve rado, cloud-computng data center, smart grd, heterogeneous networks and femto cell. Da H. Tran receved the BS degree from Fontys Unversty of Appled Scence n Endhoven, The Netherlands, n Informaton Technology, n 2011. Snce 2013, he has been wth the Department of Computer Engneerng, Kyung Hee Unversty as a Master student. Hs research nterests s about extendng the capabltes of Moble Systen through Moble Cloud Computng. Shaole Ren M 13) receved hs B.E. from Tsnghua Unversty n 2006, M.Phl. from Hong Kong Unversty of Scence and Technology n 2008, and Ph.D. from Unversty of Calforna, Los Angeles, n 2012, all n electrcal engneerng. Snce 2012, he has been an Assstant Professor n the School of Computng and Informaton Scences, Florda Internatonal Unversty, where he also holds a jont appontment wth Department of Electrcal and Computer Engneerng. Hs research nterests nclude sustanable computng, data center resource management, and network economcs. He receved the Best Paper Award at Internatonal Workshop on Feedback Computng co-located wth USENIX ICAC) n 2013 and the Best Paper Award at IEEE Internatonal Conference on Communcatons n 2009. Eu-Nam Huh receved the B.S. degree from Busan Natonal Unversty, Pusan, Korea, the M.S. degree n computer scence from the Unversty of Texas, Austn, n 1995, and the Ph.D. degree from Oho Unversty, Athens, n 2002. Durng 2001 and 2002, he was the Drector of the Computer Informaton Center and an Assstant Professor wth Sahmyook Unversty, Seoul, Korea. He has also been an Assstant Professor wth Seoul Womens Unversty. He s currently a Professor wth the Department of Computer Engneerng, Kyung Hee Unversty, Suwon, Korea. He has been an Edtor of the Journal of Korean Socety for Internet Informaton. Hs research nterests nclude hghperformance networks, sensor networks, dstrbuted real-tme systems, grds, cloud computng, and network securty. Prof. Huh was the Program Char of the Workshop on Parallel and Dstrbuted Real-Tme Systems/Internatonal Parallel and Dstrbuted Processng Symposum n 2003. Snce 2002, he has been the Korea Grd Standard Group Char. Choong Seon Hong S 95, M 97, SM 11) receved hs B.S. and M.S. degrees n electronc engneerng from Kyung Hee Unversty, Seoul, Korea, n 1983, 1985, respectvely. In 1988 he joned KT, where he worked on Broadband Networks as a member of the techncal staff. From September 1993, he joned Keo Unversty, Japan. He receved the Ph.D. degree at Keo Unversty n March 1997. He had worked for the Telecommuncatons Network Lab., KT as a senor member of techncal staff and as a drector of the networkng research team untl August 1999. Snce September 1999, he has worked as a professor of the department of computer engneerng, Kyung Hee Unversty. He has served as a General Char, TPC Char/Member, or an Organzng Commttee Member for Internatonal conferences such as NOMS, IM, APNOMS, E2EMON, CCNC, ADSN, ICPP, DIM, WISA, BcN, TINA, SAINT, and ICOIN. Also, he s now an assocate edtor of IEEE Transactons on Servces and Networks Management, Internatonal Journal of Network Management, Journal of Communcatons and Networks, and an assocate techncal edtor of IEEE Communcatons Magazne. And he s a senor member of IEEE, and a member of ACM, IEICE, IPSJ, KIISE, KICS, KIPS and OSIA. Hs research nterests nclude Future Internet, Ad hoc Networks, Network Management, and Network Securty.