IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 2, MARCH

IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 2, MARCH 2016 937 How Geo-Dstrbuted Data Centers Do Demand Response: A Game-Theoretc Approach Nguyen H. Tran, Member, IEEE, Da H. Tran, Shaole Ren, Member, IEEE, Zhu Han, Fellow, IEEE, Eu-Nam Huh, Member, IEEE, and Choong Seon Hong, Senor Member, IEEE Abstract We study the demand response (DR) of geodstrbuted data centers (DCs) usng smart grd s prcng sgnals set by local electrc utltes. The geo-dstrbuted DCs are sutable canddates for the DR programs due to ther huge energy consumpton and flexblty to dstrbute ther energy demand across tme and locaton, whereas the prce sgnal s well-known for DR programs to reduce the peak-to-average load rato. There are two dependences that make the prcng desgn dffcult: 1) dependency among utltes; and 2) dependency between DCs and ther local utltes. Our proposed prcng scheme s constructed based on a two-stage Stackelberg game n whch each utlty sets a real-tme prce to maxmze ts own proft n Stage I and based on these prces, the DCs servce provder mnmzes ts cost va workload shftng and dynamc server allocaton n Stage II. For the frst dependency, we show that there exsts a unque Nash equlbrum. For the second dependency, we propose an teratve and dstrbuted algorthm that can converge to ths equlbrum, where the rght prces are set for the rght demands. We also verfy our proposal by trace-based smulatons, and results show that our prcng scheme sgnfcantly outperforms other baselne schemes n terms of flattenng the power demand over tme and space. Index Terms Data centers (DCs), demand response (DR), Nash equlbrum, smart grds, Stackelberg games. I. INTRODUCTION DATA CENTERS (DCs) are well-known as large-scale consumers of electrcty (e.g., DCs consumed 1.5% of the worldwde electrcty supply n 2011 and ths fracton s expected to grow to 8% by 2020 [1]). A recent study shows that many DC operators pad more than $10M [2] on ther annual electrcty blls, whch contnues to rse wth the flourshng of cloud-computng servces. Therefore, t s Manuscrpt receved October 6, 2014; revsed February 2, 2015; accepted March 30, 2015. Date of publcaton May 6, 2015; date of current verson February 17, 2016. Ths research was supported n part by the Basc Scence Research Program through the Natonal Research Foundaton of Korea (NRF) funded by the Mnstry of Educaton (NRF-2014R1A2A2A01005900), and n part by the Mnstry of Scence, ICT, and Future Plannng (MISP), Korea, under the Informaton Technology Research Center (ITRC) support program (IITP-2015-(H8501-15-1015) supervsed by the IITP. Paper no. TSG-00990-2014. (Correspondng author: Choong Seon Hong.) N. H. Tran, D. H. Tran, E.-N. Huh, and C. S. Hong are wth the Department of Computer Engneerng, Kyung Hee Unversty, Yongn 446-701, Korea (e-mal: cshong@khu.ac.kr). S. Ren s wth the School of Computng and Informaton Scences, Florda Internatonal Unversty, Mam, FL 33199 USA. Z. Han s wth the Department of Electrcal and Computer Engneerng, Unversty of Houston, Houston, TX 77004 USA. Color versons of one or more of the fgures n ths paper are avalable onlne at http://eeexplore.eee.org. Dgtal Object Identfer 10.1109/TSG.2015.2421286 necessary for DC operators to both cut costs and ncrease performances. Recent works have shown that DC operators can save more than 5% 45% [3] operaton cost by leveragng tme and locaton dverstes of electrcty market prces to optmze geo-dstrbuted DCs. However, most of the exstng research s based on one mportant assumpton: the electrcty prce applyng to DCs does not change wth demand. Ths assumpton may not be true snce an ndvdual DC wth enormous energy consumpton (e.g., Facebook s DC n Crook County, Oregon can contrbuted up to 50% of the total load of ts dstrbuton grd [4]) wll mpact to the supply demand balance of ts local utlty, whch n turn can alter the utlty s prce as shown n recent studes [5] [7]. Furthermore, the power grd can be negatvely affected due to ths assumpton. For example, blackouts mght happen due to overloads n these areas where the DCs operator shfts all of ts energy demand to a local utlty wth a low prce and a hgh enough background load. To make the power grd more relable and robust, tremendous research and ndustry efforts have focused on buldng the next-generaton power grds, known as smart grds. Due to ts effcency and potental, many studes consder how DC operators can run ther geo-dstrbuted DCs on smart grds that support two-way nformaton exchange between utltes and customers [5], [8], [9]. An mportant feature of smart grds s demand response (DR). DR programs seek to provde ncentves to nduce dynamc demand management of customers electrcty load n response to power supply condtons. For example, just before the peak load hours, a utlty can send the warnng sgnal to customers smart meters whch wll automatcally schedule ther demands to reduce the power consumpton. Due to ther huge and rapdly ncreasng energy consumpton, DCs should be sgnfcantly encouraged to partcpate n the DR programs. Furthermore, wth the recent trend n dynamc server capacty provson and flexblty of workload shftng, geo-dstrbuted DCs have a great potental to easly adapt the DR programs. One of the DR programs s usng real-tme prcng schemes to reduce the peak-to-average (PAR) load rato by encouragng customers to shft ther energy demand away from peak hours. The challenge of an effectve prcng scheme s how to charge the customers wth a rght prce not only at the rght tme and rght place but also on the rght amount of customers demand. A real-tme prcng scheme s consdered effectve f t can mtgate the large fluctuaton of energy consumpton between peak and off-peak hours to ncrease power grd s relablty and robustness. 1949-3053 c 2015 IEEE. Personal use s permtted, but republcaton/redstrbuton requres IEEE permsson. See http://www.eee.org/publcatons_standards/publcatons/rghts/ndex.html for more nformaton.

938 IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 2, MARCH 2016 In ths paper, we consder the problem of usng realtme prcng of utltes to enable the geo-dstrbuted DCs partcpaton nto the DR program. In ths program, whle geo-dstrbuted DCs employ workload shftng and dynamc server provsonng n response to the prce sgnal, the role of local utltes s how to set the real-tme prces to flatten the customers demand load. It can be observed that there s an nteracton between geo-dstrbuted DCs and ther local utltes; and t s the frst challenge of ths DR problem that we call vertcal dependency. Specfcally, when partcpatng n the DR program, a DCs operator wll dstrbute ts energy demand geographcally based on the electrc prces adjusted ntellgently by the local utltes. However, the utltes set ther prces based on the total demand ncludng the DCs demand, whch s only known when the prce s avalable. We clearly see that ths dependency makes t dffcult for both DCs and utltes to make ther decsons. The second mportant challenge, whch s less obvous, s an nteracton among local utltes feedng power to the geo-dstrbuted DCs; and we call t horzontal dependency. Specfcally, the DCs decsons on workload shftng and server allocaton depend on the electrc prces set by local utltes; therefore, f any subset of the local utltes change ther prces, t can lead to the DCs decson changng. Snce the utltes are noncooperatve (.e., no nformaton exchange) n practce, how to desgn a prcng mechansm that can enable an equlbrum prce settng profle s the bottleneck of ths DR program. To tackle the above dscussed challenges, our contrbutons can be summarzed as follows. 1) We transform the functonal space of the geo-dstrbuted DCs DR program nto a mathematcal space of a formulated two-stage Stackelberg game. In ths game, each utlty wll set a real-tme prce to maxmze ts own proft n Stage I; and gven these prces, the DCs operator wll mnmze ts cost va workload shftng and dynamc server allocaton n Stage II. We also utlze the backward nducton method to fnd the Stackelberg equlbra of ths two-stage game. 2) Based on the Stackelberg equlbra, our proposed scheme can deal wth the nherent challenges of ths DR as follows: frst, the horzontal dependency between utltes are characterzed as a strategc game n Stage I, and we show that there exsts a Nash equlbrum n ths game. Second, we propose an teratve and dstrbuted algorthm to acheve the Stackelberg equlbrum. In ths algorthm, the DCs and utltes exchange ther nformaton (.e., DCs demand and utltes prces) teratvely untl the algorthm converges. We also examne the algorthm s convergence where the rght prces are set for the rght demands as a soluton for the vertcal dependency ssue. 3) Fnally, we perform a real-world trace-based smulaton to soldfy the analyss. The results show that our proposed prcng scheme can flatten the workload not only over tme but also over space to mprove the power grd s relablty and robustness. The rest of ths paper s organzed as follows. Secton II s about related work. Secton III presents the system model and the two-stage Stackelberg game. We analyze ths game and propose a dstrbuted algorthm n Secton IV. Secton V provdes the trace-based smulaton results. Fnally, Secton VI concludes ths paper. II. RELATED WORK DR s dentfed as one of hgh-prortzed areas for future smart grds [10] [12] wth ts potental to reduce up to 20% of the total peak electrcty demand of the U.S. [13]. Most DR proposals, whch try to ncentvze customers to manage ther demand dynamcally n response to the power supply condtons, mostly targeted to resdental customers [14] [17]. On the other hand, most of the exstng research on DCs, whch can be classfed as medum or large ndustral customers, manly focus on ther cost mnmzaton that takes the electrcty prce for granted [3], [18], [19], whch does not follow any DR programs. However, due to the mportant role of DCs n DR programs, DRs of DCs recently receve sgnfcant attenton [4], [7] [9], [20] [22]. For those work consderng DR of geo-dstrbuted DCs, based on the nteractons between DCs and utltes, we smply dvde them nto two categores. 1) One-Way Interacton: One of the most popular DR programs of DCs s concdent peak prcng (CPP), whch s studed n [21]. CPP charges very hgh prces for power usage durng the concdent peak hour at whch the most electrc demands s requested to the utlty. By predctng the upcomng potental peak hours, the utltes send a warnng sgnal (.e., not a prce) to help customers schedule ther power consumpton. However, current DCs do not respond actvely to the warnng sgnals due to the uncertanty of these warnngs [21], whch motvates researchers to devse more effectve DR approaches. Wang et al. [7] used a predcton-based method where the customers (DCs) respond to the prces whch are chosen based on a supply functon. Ths supply functon can be modeled usng some data fttng methods based on hstory. Hence, n ths paper only customers respond to a predcted prce whle there s no acton from the power supplers to set the prces correspondng to the demand. 2) Two-Way Interacton: There are three recent papers [5], [8], [9] n ths category. The frst two papers, whch are hghly related to ths paper, consder dynamc prcng mechansms wth the couplng between utltes and DCs, whereas the last one proposed that DCs can partcpate n the spot market va a broker, whch s a sgnfcant departure from our model. Moreover, the system model of [9] assumes that all utltes cooperate to solve a socal optmzaton problem, whch s not relevant to current practce snce there s no nformaton exchange between utltes n realty. On the other hand, the prcng scheme of [8] s based on a heurstc approach, whch cannot maxmze the utltes proft as well as mnmze ther cost. Ths paper falls nto ths category of two-way nteracton, yet s dfferent from others n terms of ts two-stage game-theoretc approach to tackle the vertcal and horzontal couplng ssues, whch are not addressed n the lterature, between geo-dstrbuted DCs and local utltes.

TRAN et al.: HOW GEO-DISTRIBUTED DCs DO DR: GAME-THEORETIC APPROACH 939 TABLE I SUMMARY OF NOTATIONS to flatten the demand over tme and locatons to ncrease the power grd s relablty, as the prce-takers the DCs wll mnmze ther costs. In the mathematcal space, we observe that there exsts a specal mutual nteracton between DCs and utltes where utltes set prces based on the total demand, and DCs mnmze ther costs based on the prces. Therefore, we transform ths DCs DR program nto a leader follower game that can be studed usng a two-stage Stackelberg game. Specfcally, the utltes are the leaders that set the prces to maxmze ther profts n Stage I and DCs wll make ther decsons on workload shftng and dynamc server provsonng to mnmze ther costs n Stage II. We present ths twostage game formulaton n the reverse sequence, startng wth Stage-II optmzaton problem. Fg. 1. Functonal space of the geo-dstrbuted DCs DR on the left and ts transformed mathematcal space as a two-stage Stackelberg game on the rght. III. SYSTEM MODEL AND PROBLEM FORMULATION We consder one-perod DR as n [18] and [23], where ts duraton, whch s controlled by a utlty/load servng entty, matches an nterval at whch the DCs decsons and utltes real-tme prces can be updated (such as 15 mn or 1 h). Let I ={1,...,I} denote the set of stes wth dfferent electrcal utltes servce regons where DCs are located. Such geodstrbuted DCs are very common n practce, e.g., Google, Amazon, etc. Each DC s powered by a local utlty company and have S homogeneous servers. A DC wth heterogeneous types of servers can be vewed as multple vrtual DCs each havng homogeneous servers. For the ease of presentaton, Table I lsts key notatons of ths paper. We ncorporate the role of utlty nto the DR programs of DCs to regulate the power demand at each local ste for load balancng the power grd. We llustrate a functonal space and a mathematcal space of ths DR program n Fg. 1. Inthe functonal space, we leverage the dea of usng the advanced two-way communcaton of smart grd to facltate the nformaton exchange between utltes and DCs at each local ste va smart meters. Whle utltes set prces to ncentvze DCs A. DCs Cost Mnmzaton n Stage II We frst descrbe the workload model of a typcal DC. We then elaborate the DCs cost focusng on the energy cost and delay cost model. Fnally, we formulate the Stage-II DCs cost mnmzaton. 1) Workload Model: Even though DCs can support a wde range of workloads, we generally classfy them nto two typcal types of workload: nteractve (nonnterruptve) jobs and batch (nterruptve) jobs. Whle the former s delay-senstve (e.g., computng search, onlne game, etc.), the latter s delaytolerant (e.g., backup tasks, MapReduce, etc.). We assume that each DC processes ts batch jobs locally (.e., batch jobs cannot be redrected to other DCs for load balancng) snce wthout strngent delay constrant, they are flexble to be scheduled across a large tme wndow at a local ste, lke [19]. For nteractve jobs, we denote the total arrval rate to the DCs front-end server [.e., all DCs are managed by a DCs servce provder (DCs provder)], by and ths front-end server s responsble for splttng the total ncomng workload nto separate workloads of geo-dspersed DCs, denoted by {λ } I. Even though we only consder workload shftng, the other control knobs for DR such as power load reducton (e.g., scalng down CPU frequences and/or turnng off unused servers) can also be ntegrated nto our framework. 2) DCs Cost and SLA Model: We assume that the DCs provder tres not only to mnmze ts energy cost and mgraton cost but also to guarantee the servce level agreement (SLA) requrements for the nteractve jobs. a) Energy cost: Snce batch jobs are flexble to schedule n tme doman, batch jobs processng s consdered to consume an amount of energy e b of each DC wth ther dedcated servers. On the other hand, the energy consumpton of nteractve jobs at DC s [2] e d ( = s Pdle + ( ) ) P peak P dle U + (η 1)P peak (1) where s s the number of actve servers, μ s the servce rate of a server, P peak and P dle are the server s peak and dle power, respectvely, U = λ /s μ s the average server utlzaton, and η s the power usage effectveness (PUE) measurng the energy effcency of the DC. We can rewrte e d as follows: e d = a λ + b s, I (2)

940 IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 2, MARCH 2016 where a = (P peak P dle )/μ and b = P dle + (η 1)P peak. Therefore, denotng the total energy by e = e d + e b (3) and gven a prce p, the energy cost of DC s e p. b) Mgraton cost: Snce mgratng the workload from front-end server to geo-dstrbuted DCs can be very costly [e.g., mgratng vrtual machnes or vdeo content requests over the Internet could be expensve due to reservng bandwdth from an Internet servce provder (ISP)], we model the mgraton cost to DC as ωd c (λ ), where d s the transmsson delay from the front-end server to DC, ω s a weght factor and c (λ ) s a functon whch s assumed to be strctly ncreasng and convex. Snce d s proportonal to the dstance, t s assumed to be a constant and we see that mgratng more requests from the front-end server to a more dstant DC s more costly. For analyss tractablty, we choose a quadratc functon c (λ ) = λ 2 snce t s wdely used n many felds such as control, sgnal processng, communcaton networks, etc. to model a cost functon [24]. c) SLA constrant: We assume that each delay-senstve request mposes a maxmum delay D that the DCs provder has to guarantee when shftng ths request to DC. Therefore, the SLA constrant n terms of delay guarantee can be modeled as follows: 1 + d D, (4) s μ λ where 1/(s μ λ ) s the average delay tme of a request processed n DC wth arrval rate λ and servce rate s μ by queueng theory, whch has been wdely used as an analytc vehcle to provde a reasonable approxmaton for the actual servce process [19], [25]. 3) Problem Formulaton: Our model focuses on two key controllng knobs of DCs cost mnmzaton: the workload shftng to DC λ and the number of actve servers provsoned s at ste,. Then, the Stage-II DC cost mnmzaton s gven by DC : mnmze I e p + ωd λ 2 (5) =1 subject to constrants (2) (4) I λ = (6) =1 0 s S, (7) 0 λ s μ (8) varables s,λ,. (9) Whle constrants (2) (4) are the defntons of the objectve functon and the SLA constrant, the remanng constrants are straght forward. In (6), all of the ncomng workload must be served by some DCs. Moreover, (7) lmts the number of actve servers and (8) means that the total workload assgned to a DC must be less than ts capacty. Wth thousands of servers n a DC, we can further relax the nteger varables s as contnuous varables so that ths problem s tractable [18]. Fg. 2. Besdes conventonal wholesale and retal prcng, the utltes DR real-tme prcng s proposed for geo-dstrbuted DCs and other DR-enabled customers. B. Noncooperatve Prcng Game n Stage I In ths stage, we frst present the market structure. We next descrbe the utlty s revenue and cost models and fnally formulate the noncooperatve prcng game. 1) DR Retal Prce: Tradtonally utltes nvolve many complex electrcty markets. As buyers, utltes can partcpate n a wholesale market (day head, real-tme balancng) to buy electrcty from the generatng companes wth wholesale prces. As sellers, utltes make proft by sellng retal to ther customers wth proper retal rates [5]. Snce conventonal customers (.e., no DR) have nelastc demand wth predctable patterns, utltes can predct and buy energy from wholesale market, then resell t at the conventonal retal rates. However, DCs wth workload shftng represent a new type of elastc-demand customers, whch makes utltes dffcult to predct ther demands, mpactng the grd s stablty. Therefore, we propose a new DR retal prcng scheme for utltes to serve the unpredcted and elastc customers, e.g., not just loadshftng DCs but also for all DR-enabled customers. The basc dea of ths scheme s that utltes and these DR-enabled customers can coordnate va smart-grd nfrastructure to match supply wth demand. Fg. 2 llustrates that utltes can apply the conventonal and DR retal prces to ther correspondng customers, whch are complementary to each other so that the proposed scheme wll not affect to the conventonal scheme, smlar to [9], [17], and [26]. Snce conventonal markets and customers are orthogonal to our model, henceforth, we only consder utlty s proft model and the proposed real-tme prcng scheme for DR-enable customers. 2) Utlty s Revenue and Cost Model: We see that the optmal energy consumpton of DCs that can be obtaned from solvng DC depends on all utltes prces. Denote the correspondng optmal power demand by e (p), where p := {p } I. We further assume that due to the grd regulatons at each regon, the lower and upper bound of the real-tme prce should be mposed and denoted by p l and p u,, t, respectvely. Furthermore, besdes the power demand of DCs, each utlty has ts own background load (e.g., resdental/commercal/ ndustral demand). Snce there are consderable works focusng on the resdental DR programs, we assume that the background load of utlty, denoted by B (p ), also responds to the prce and can be modeled by the followng functon: B l, p p l B (p) = α β p, p l p p u (10) B u, p p u

TRAN et al.: HOW GEO-DISTRIBUTED DCs DO DR: GAME-THEORETIC APPROACH 941 where B l and B u are the mnmum of maxmum background demands of ste due to the physcal constrants of consumers (.e., mnmum and maxmum power of electrc devces or vehcles). Ths functon, whch follows the lnear demand model n [27], shows an nherent response of customers to the prce: decrease the demand down to a lower-bound constrant when the prce ncreases, and vce versa, where β s the decreasng slope and α models the physcal upper-bound demand wthout prce. Based on the hstory of customer s usage data, utltes can estmate α and β usng some data fttng methods, smlar to [7]. Basedonthetotalpower requested by DCs and background s demands, the revenue of utlty s gven by R (p) = (e (p) + B (p ))p. (11) On the other hand, every utlty ncurs a cost when t serves the customers load. When the load ncreases, the utlty s cost also ncreases snce normally blackouts happen due to overload, whch s a dsaster to any utlty. Hence, we can model the utlty s cost based on a wdely used electrc load ndex (ELI) as follows: ( ) C (p) = γ ELI := γ r 2 C e (p) + B (p ) 2 = C (12) where C s utlty capacty, γ reflects the weght of the cost, and r s a load rato that measures the power load levels. Averyhghr can rsk the utlty s stablty. ELI s motvated by the ndex measurement technques used for load flattenng n a power grd [9], [28], [29]. We see that ELI can weght dfferent utltes load rato r by ther capactes, provdng feeder load-balancng capablty. On the other hand, a utlty wth hgh γ shows that t s more concerned about the effect of ELI to the relablty, whle a utlty wth low γ has more nterest n makng revenue and less concerned about the nstablty s threat. 3) Stage-I Prcng Game Formulaton: In realty, the geodstrbuted utltes usually have no communcaton exchange to optmze the socal performance. Instead, each utlty has ts own goal to maxmze ts proft, whch s defned as the dfference between revenue and cost as follows: u (p, p ) = R (p) C (p) (13) where p denotes the prce vector of other utltes except. Ths notaton comes from an observaton that there s a game between utltes because the proft of each utlty not only depends on ts energy prce but also on the others. Hence, the Stage-I utlty proft maxmzaton game, denoted by UP = (I, {p } I, {u } I ), s defned as follows. 1) Players: The utltes n the set I. 2) Strategy: p l p p u, I. 3) Payoff functon: u (p, p ), I. IV. TWO-STAGE STACKELBERG GAME: EQUILIBRIA AND ALGORITHM In ths secton, we frst apply the backward nducton method to solve the Stackelberg game. Then, we propose an teratve algorthm to reach an equlbrum of ths game. C A. Backward Inducton Method 1) Optmal Solutons at Stage II: We realze that the Stage-II DCs cost mnmzaton can be decomposed nto ndependent problems. Henceforth, we only consder a specfc tme perod and drop the tme dependence notaton for ease of presentaton. In ths stage, DCs cooperate wth each other to mnmze the total cost by determnng the workload allocaton λ and the number of actve servers s at each DC. It s easy to see that the DCs cost mnmzaton s a convex optmzaton problem. Frst, we observe that constrant (4) must be actve because otherwse the DCs provder can decrease ts energy cost by reducng s. Hence, we have (4) s equvalent to [ 1 ( 1 s (λ ) = λ + D )] S (14) μ 0 where [.] y x s the projecton onto the nterval [x, y] and D := D d. In practce, most DCs can have a suffcent number of servers to serve all requests at the same tme due to the lluson of nfnte capacty of DCs [18]. Therefore, we adopt 1 s (λ ) = 1/μ (λ + D ) n the sequel. By substtutng ths s (λ ) nto the objectve of DC, we have an equvalent problem DC as follows: I DC :mn. f (λ ) (15) λ s.t. =1 I λ = (16) λ 0, (17) where ( f (λ ) := ωd λ 2 + p a + b )λ + p (e b + b D ) 1. μ μ It can be seen that DC s a strctly convex problem, whch has a unque soluton. Snce DCs provder lkes to have λ > 0,, n order to utlze all DCs resources, we characterze the unque soluton of DC and a necessary condton to acheve ths soluton wth the optmal λ > 0,, as the followng result. Lemma 1: Gven a prce vector p, we have the unque solutons of Stage-II DC problem λ = ν p A > 0, and s = 1 ( λ 1 + D ), (18) 2ωd μ only f =1 ω>ω 1 th := (ˆd max {p A } ) I / / p A d 2 (19) where ˆd := I =1 1/d, A := a + b /μ, and ν = 1/ˆd(2ω + I =1 p A /d ). Snce all parameters to calculate ωth 1 are avalable to DC, we can consder condton (19) as a gudelne for a DCs provder to choose an approprate weght factor ω to ensure that all DCs have postve request rates. =1

942 IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 2, MARCH 2016 2) Nash Equlbrum at Stage I: We contnue to characterze the Nash equlbrum of the Stage-I game based on the Stage-II solutons. From (13), we have u (p, p ) = ( e (p) + B (p ) ) ( e ) p γ C (p) + B (p ) 2 C (20) where e (p) = (a λ + b s ) + eb (wth λ and s obtaned from Lemma 1) and can be presented as follows: e (p, p ) = A2 p ( ) 1 1 2ωd ˆdd + A 2ωˆdd j = A j p j d j + A ˆdd + b μ D + e b. (21) In the noncooperatve game, one of the most mportant questons s whether there exsts a unque Nash equlbrum. In the case of Stage-I game, we have the followng defnton of a Nash equlbrum. Defnton 1: A prce vector p e := {p e } I s sad to be a Nash equlbrum f no utlty can mprove ts proft by unlaterally devatng ts prce from the Nash equlbrum ( u p e, p e ( ) u p, p e ), p l p p u,. (22) Theorem 1: (Exstence) There exst a Nash equlbrum of the Stage-I UP game. In ths Stage-I game, gven all other utltes strateges p, a natural strategy of utlty s the best response strategy as follows: BR (p ) = arg max u (p, p ), (23) p P where P := [ p l, ] pu. In order to fnd the best response, we set u (p)/ p = 0. Then, the teratve best response updates can be obtaned as follows: ( ) p (k+1) = BR p (k) = 1/2 γ N /C ( h 1 γ N /C p (k) ), ( N ) P (24) where [.] P denotes the projecton onto P, k represents the teratons, N := A 2 /2ωd (1/ˆdd 1) β, and h(p ) := A A j p j + A + b + e b + α,. 2ωˆdd d j ˆdd μ D j = (25) When all utltes play best response strateges, a Nash equlbrum p e s a profle that satsfes p e = BR (p e ),,.e., every utlty s strategy s ts best response to others strateges. However, there are two ssues here. 1) There s no condton for general games such that the best responses converge to a Nash equlbrum. 2) Snce multple Nash equlbra can exst n the UP game, how the best response can converge to a unque Nash equlbra. Hence, we next examne the convergence property of the best response (24) to a unque Nash equlbrum by usng the concept contracton mappng. Fg. 3. Detaled operatons of Algorthm 1, where red arrows represent steps 3 and 5 and blue arrows correspond to step 4. We brefly ntroduce contracton mappng and ts propertes, all of whch can be found n [30, Ch. 3]. Snce many teratve algorthms have the form x (k+1) = T(x (k) ), k = 0, 1,..., where x (k) X R n, the mappng T : X X s called a contracton f there s a scalar 0 σ<1 such that T(x) T(y) σ x y, x, y X (26) where. s some norm defned on X. Furthermore, the mappng T s called a pseudo-contracton f T has a fxed pont x X [.e., x = T(x )] and T(x) x σ x x, x X. (27) Both contracton and pseudo-contracton have the geometrc convergence rate property: suppose the mappng T has a fxed-pont, the sequence {x (k) } generated by x (k+1) = T(x (k) ) converges to a unque fxed pont x geometrcally satsfyng x (k) x σ k x (0) x, k 0 (28) wth any ntal value x (0) X. Based on the above propertes of contracton mappng and Theorem 1, f we can show that the best response update (24) s a contracton mappng, then we can guarantee ts convergence to a unque Nash equlbrum. Therefore, we establsh the followng suffcent condton. Theorem 2: (Convergence and Unqueness) If A j = A j/d j A 2 ˆd ( ω ωth 2 1 1 /( )) d ˆd := max 2β ˆdd (29) then startng from any ntal pont, the best response updates (24) of the Stage-I UP game s a contracton mappng that converges to a unque Nash equlbrum p e geometrcally. B. Dstrbuted Algorthm We frst descrbe the detaled operatons of the proposed algorthm. Next, we dscuss practcal mplementaton ssues of the algorthm. 1) Proposed Algorthm s Operatons and Convergence: We contnue proposng a dstrbuted algorthm, shown n Algorthm 1, whch can acheve the Nash equlbrum. The detaled operatons of Algorthm 1 are llustrated n Fg. 3.

TRAN et al.: HOW GEO-DISTRIBUTED DCs DO DR: GAME-THEORETIC APPROACH 943 Algorthm 1 DR of DC Wth Real-tme Prcng 1: ntalze: k = 0, ɛ s arbtrarly small, p (0) = p u,, and ω satsfes (29); 2: repeat 3: Utlty broadcasts ts p (k) to all customers; 4: The front-end server collects p (k) from all DCs, updates e (p)(k) as (21) and sends t back to DC, ; 5: Each DC reports ts e (p)(k) to the local utlty; 6: Utlty receves the demand responses from the local DC e (p)(k) and background users B (p) (k), then updates p (k+1) = BR (p (k) ) as (24); 7: untl p (k+1) p (k) <ɛ. We assume that Algorthm 1 operates at the begnnng of each prcng update perod (.e., 1 h) and the algorthm runs for many teratons (communcaton rounds wth a parameter k) untl t converges to a prce settng equlbrum. Here, based on the total ncomng workload, the front-end server of the DCs provder frst collects all prces from ts local DCs and calculates the optmal energy consumpton as (21) (step 4). After that, the front-end server wll feedback these energy consumpton data to ts local DCs, whch then forwards ts own nformaton to the local utlty (step 5). Each utlty solves ts own proft maxmzaton problem (best response updates) to fnd an optmal prce, then broadcasts ths prce to ts local DCs and background customers (step 6). The process repeats untl the game converges to the unque Nash equlbrum accordng to Theorem 2 (step 7). At ths state the prce settng s fnalzed and appled to the whole consdered perod. Even though Algorthm 1 s presented n a scalable and synchronous fashon (.e., all local utltes update and broadcast ther prces at the same tme), asynchronous dstrbuted algorthm s preferred snce n realty, the message-passng among front-end server, DCs and utltes usually ncurs heterogeneous delays. Fortunately, wth condton (29), Algorthm 1 can also work asynchronously snce (29) s derved from establshng a contracton mappng wth respect to a maxmum norm., whch guarantees the asynchronous convergence of the mappng sequence [30, p. 431]. 2) Practcal Issues and Implementaton Dscusson: We dscuss two ssues here: the workload shftng assumpton and the message-passng. In terms of the former, we assume the DCs provder deploys a front-end server to dstrbute the ncomng workload to DCs. Ths can be done by usng varous practcal solutons such as ncorporatng the authortatve DNS servers (whch s used by Akama) or HTTP ngress proxes (whch s used by Google and Yahoo) nto the front-end servers. Furthermore, n realty there s only a sub-set of DCs to whch a workload type can be routed to due to the avalablty resource constrant of each DC. Ths ssue can be easly addressed by ncorporatng addtonal constrants nto our model such as [31], and n practce we can mplement t by classfyng the workload types at the front-end server before routng. In terms of the later, we assume that the two-way communcaton between a DC and ts local utlty can be enabled va communcaton networks of future smart grd. Regardng to the communcatons between DCs and ts front-end server, a DC reports ts utlty s prce by choosng one of the egress lnks of ts ISP to send ts packet through the Internet to the front-end server, and vce versa. Specfcally, the total tme of one teraton conssts of the transmsson tme and computatonal tme. Whle the transmsson tme from utltes to DCs (and vce versa) s from 1 to 10 ms over a broadband speed of 100 Mb/s, t s from 50 to 100 ms for a one-way communcaton between DCs and the front-end servers over a current ISPs path. The computatonal tme depends on the processng power of the front-end server and smart meters on calculatng the optmal energy (21) and maxmzng the convex proft functon (21), whch are both low-complexty problems and can be n the tme-scale of mcrosecond [24]. V. TRACE-BASED SIMULATIONS In ths secton, we conduct trace-based smulatons, mplemented n the Python language wth exstng lbrares ncludng NumPy, ScPy, and Matplotlb, to valdate our analyss and evaluate the performance of Algorthm 1. A. Setups We consder sx geo-dstrbuted DCs powered by ther local utltes at the followng ordered locatons: 1) the Dalles, OR; 2) Councl Bluffs, IA; 3) Mayes County, OK; 4) Lenor, NC; 5) Berkeley County, SC; and 6) Douglas County, GA. These locatons correspond to real Google s DCs [32]. All DCs PUEs are set to 1.5 over tme perods. The homogeneous servers have peak power of 200 W and dle power of 100 W, and the servce rate of each server s chosen unformly between 1.1 and 1.2. The mgraton weght ω s set to 1 unless otherwse stated. The delay SLA D are dstrbuted unformly between 100 and 300 ms and d s scaled by the vector [1.9, 1.0, 1.3, 2.5, 2.8, and 2.3] n whch we assume that the front-end server s placed at Colorado. We use realstc traces for the ncomng workload at the front-end server and the power demand of delay-tolerant batch jobs e b at each DC. All of them are scaled wth respectve to servce rates. We use an nteractve workload trace collected from Mcrosoft Research (MSR) [33]. The workload can be predcted to a farly reasonable accuracy usng, e.g., regresson technques [3], [33]. Furthermore, we use Google trace for the power demand of delay-tolerant batch jobs e b n recent study [34]. The batch job power demand and workload seres spans over 30 days correspondng to a typcal utlty bllng cycle and each pont of seres s a 1-h perod. Snce lackng the publc nformaton of local utltes, we assume that all utltes have the capactes C unformly n the range of 25 and 30 MW, whch s a standard measure for a medum-sze utlty. Whle γ s set to 1 unless otherwse stated, α and β parameters are chosen unformly n the range of [25, 30] and [0.25, 0.30], respectvely. We consder two baselne prcng schemes for comparson. The frst baselne s based on the proposed dynamc prcng scheme of [8], whch s brefly descrbed as follows: p (t + 1) = δ(pd (t) PS (t)) + p (t) (30)

944 IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 2, MARCH 2016 Fg. 4. Compared prces at sx locatons. TABLE II AVERAGE OPTIMAL PRICES COMPARISONS WITH γ EFFECT Fg. 5. Proporton of local DC demand over utlty total demand at Mayes County (top) and Lenor (bottom). where PD and PS are the power demand and supply of utlty. We set δ to 0.5 n all smulaton scenaros. Ths baselne serves as a recent related benchmark. The second baselne s based on the Google s contract wth ther local utltes. Accordng to the emprcal study n [32], there are sx Google s DCs at sx mentoned locatons, where Google s DCs are nferred to have long-term contracts wth ther local utltes as the followng fxed rates (.e., energy charges) [32.57, 42.73, 36.41, 40.68, 44.44, and 39.97] $/Wh, respectvely. Ths baselne serves as an n-realty benchmark. We manly use ths baselne for the PAR comparsons snce: 1) the Google long-term contract often negotates a monthly electrcty bll scheme that combnes energy charges and demand charges that we do not know exactly, whch can then nfluence the DCs cost and utltes proft and 2) t s not far to compare a dynamc prcng scheme to a snapshot statc prcng scheme n terms of cost and proft. B. Results We frst provde the sample-path optmal prces of three schemes at sx locatons n Fg. 4. In all perods, we observe that Algorthm 1 can converge n less than ten teratons, where the stoppng condton ɛ = 10 4. Snce Baselne 1 and Algorthm 1 employ dynamc prcng mechansms, we observe that the utltes prces of these two schemes vary accordng to the workload pattern. We also observe the effect of mgraton cost to the optmal prces n ths Fg. 4. Snce the nearest DCs to the front-end server are stes 2 and 3, Fg. 4 shows that all dynamc prcng schemes set hgh prces at these two stes compared wth the other stes. Ths can be explaned as follows, due to the small mgraton cost at these stes whch leads to hgh demand, the dynamc schemes set hgh prces to balance between energy cost and mgraton cost. Furthermore, we observe that Algorthm 1 can contrbute less load to utltes than other schemes do most of the tme; for example, ths can be seen n Fg. 5 that shows the proporton of DCs demand over utltes total demand varatons n three days at two stes. Furthermore, we also nvestgate the effect of γ to the prcng schemes. Table II shows that f we ncrease γ, then the Algorthm 1s optmal prces also ncrease snce the hgher the weght utltes ELI cost factor s, the more conservatve utltes are n terms of relablty by rasng the prces. Fnally, we can see that Baselne 1 always overprces Algorthm 1 n all scenaros snce Baselne 1 s more aggressve than Algorthm 1 n terms of balancng the supply and demand. However, t could lead to hgh demand fluctuatons (.e., hgh PAR) as shown n the followng results. We also observe that the average prces of Algorthm 1 are not affected by ω.

TRAN et al.: HOW GEO-DISTRIBUTED DCs DO DR: GAME-THEORETIC APPROACH 945 Fg. 6. Effect of γ to average DCs cost and utltes proft. utltes proft than those of Algorthm 1. In detals, the share of DCs energy cost of Algorthm 1 s 36.3%, 37.8%, and 38.7% when γ = 1, 4, and 8, respectvely, whereas that of Baselne 1 (wthout γ mpact) s 44.8%. Therefore, Algorthm 1 can gve more ncentves to encourage the DCs to jon the DR program. Second, we can see that when γ ncreases, the utltes proft of both schemes decrease accordng to (20). Snce the prcng scheme of Baselne 1 s ndependent wth γ, we can see that γ has no effect to the DCs cost of ths baselne. However, we see that DCs cost of Algorthm 1 ncreases when γ ncreases due to the correspondng ncrease of the optmal prces [see(20)]. Wth Algorthm 1, we see that small γ s favorable because t can provde low DCs cost and hgh utltes proft. Furthermore, due to the background demand, we see that DCs cost ncludng the mgraton cost s lower than utltes proft. The fnal factor that we examne s the power demand PAR at each ste, whch s one of the most mportant metrcs to measure the effectveness of desgns for smart grd snce the fluctuaton of energy consumpton between peak and off-peak hours ndcate power grd s relablty and robustness. PAR s calculated as max t {e (p(t))+ B (p (t))}t/ T t=1 e (p(t)) + B (p (t)). Reducng PAR s the mportant goal of any DR program desgns. Therefore, we extensvely compare the PAR of three schemes wth dfferent γ n Fg. 7(a) (c). The most mportant observaton s that PARs performance of Algorthm 1 outperforms those of other schemes, ether statc or dynamc prcng, over tme and space sgnfcantly. Specfcally, consderng the case γ = 1, Fg. 7(a) shows that for all stes 1 6, Algorthm 1 can acheve the lowest PAR value as expected, reducng the PAR to 32.3%, 27.0%, 28.1%, 28.0%, 25.8%, and 29.4% compared to Baselne 1, and 31.6%, 16.7%, 22.2%, 33.5%, 34.0%, and 34.0% compared to Baselne 2, respectvely. We conclude that Algorthm 1 can spread out the demand not only over tme but also over locatons. Fg. 7. PAR wth respect to MSR trace at sx locatons wth dfferent γ. (a) γ = 1. (b) γ = 4. (c) γ = 8. We also evaluate the effect of parameter γ to average DCs cost and utltes proft n Fg. 6. Frst, we can see that Baselne 1 wth hgher prces has hgher DCs cost and VI. CONCLUSION We have nvestgated the DR of geo-dstrbuted DCs wth the help of emergence technques of smart grd. We frst characterze the challenged dependences of ths geo-dstrbuted DCs DR program where a utlty decsons not only depends on that of DCs, and vce versa, but also mpacts on other utltes decsons. We then formulate ths DR program nto a two-stage game to model these dependences. In ths game, the role of each utlty s settng a prce to maxmze ts proft, whle the DCs mnmze ts cost by workload shftng and dynamc server allocaton. We then characterze the exstence and unqueness of the Nash equlbrum of ths game, and develop an teratve and dstrbuted algorthm to reach ths equlbrum. By usng trace-based smulatons, we valdate and complement our proposal wth the smulaton results, whch shows that our prcng schemes based on the two-stage game can flatten the energy demand of DCs over tme and locatons to ncrease the power grd s relablty and robustness.

946 IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 2, MARCH 2016 APPENDIX A PROOF OF THEOREM 1 Snce the strategy space of each utlty s a nonempty compact and convex subset of Eucldean space, t s suffcent for us to show that the contnuous functon u (p, p ) on ths strategy space s a quas-concave functon,, such that there exsts a Nash equlbrum for Stage-I game [35]. From (10) and (21), t can be seen that e (p) and B (p ) are affne functons of p. Therefore, (e (p) + B (p )) 2 s a convex functon [24]. Furthermore, we have 2 (B (p )p )/ p 2 = β < 0, and 2 (e (p)p )/ p 2 = A 2 /2ωd (1/ˆdd 1) <0,, snce ˆdd > 1,. Hence, both e (p)p and B (p )p are concave functons. Therefore, from (20) we see that u (p, p ) s the sum of two concave functons so that s also a concave (and hence quas-concave as well) functon. APPENDIX B PROOF OF THEOREM 2 We frst seek the condton such that the best response update (24) s a contracton mappng. Defne a Cartesan product space P = I P and a vector BR(p) := (BR (p )) I. Snce BR(p) s contnuous and dfferentable on by P, bythe mean value theorem, we have BR(p 1 ) BR(p 2 ) = BR(p) p p 1 p 2 (31) p 1, p 2 P, and p s on the segment connectng p 1 and p 2. Furthermore, the Jacoban BR(p)/ p s as follows: { BR (p ) 0, j = = 1/2 γ N p /C A A j j ( N )(1 γ N /C ), j =. 2ωˆdd d j Then, by usng the norm. of the Jacoban, from (26) and (31), we see that (24) s a contracton mappng when BR(p) p 1/2 γ N /C A A j = max < 1. (32) ( N j = )(1 γ N /C ) 2ωˆdd d j It s straghtforward to see that the suffcent condton to satsfy (32) smax {A / 2ωˆdd N j = A j/d j } 1, whch s equvalent to A j = A j/d j A 2 ˆd ( ( )) 1 1/ d ˆd ω max 2β ˆdd. (33) We have shown that wth condton (33), the best response update s a contracton mappng. Furthermore, accordng to Theorem 1, we have the exstence of a fxed-pont of the mappng (24). Hence, based on the convergence property of contracton mappng, we complete the proof. REFERENCES [1] J. Koomey, Growth n Data Center Electrcty Use 2005 to 2010, vol.1. Oakland, CA, USA: Anal. Press, Aug. 2011. [2] A. Quresh, R. Weber, H. Balakrshnan, J. Guttag, and B. Maggs, Cuttng the electrc bll for Internet-scale systems, n Proc. ACM SIGCOMM, Barcelona, Span, Aug. 2009, pp. 123 134. [3] L. Rao, X. Lu, L. Xe, and W. Lu, Mnmzng electrcty cost: Optmzaton of dstrbuted Internet data centers n a mult-electrctymarket envronment, n Proc. IEEE INFOCOM, San Dego, CA, USA, Mar. 2010, pp. 1 9. [4] Z. Lu, I. Lu, S. Low, and A. Werman, Prcng data center demand response, ACM SIGMETRICS Perform. Eval. Rev., vol. 42, no. 1, pp. 111 123, 2014. [5] J. Camacho, Y. Zhang, M. Chen, and D. M. Chu, Balance your bds before your bts: The economcs of geographc load-balancng, n Proc. 5th Int. Conf. Future Energy Syst. (e-energy), Cambrdge, U.K., Jun. 2014, pp. 75 85. [6] A. H. Mahmud and S. Ren, Onlne capacty provsonng for carbonneutral data center wth demand-responsve electrcty prces, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 2, p. 26 37, Aug. 2013. [7] P. Wang, L. Rao, X. Lu, and Y. Q, D-Pro: Dynamc data center operatons wth demand-responsve electrcty prces n smart grd, IEEE Trans. Smart Grd, vol. 3, no. 4, pp. 1743 1754, Dec. 2012. [8] Y. L et al., Towards dynamc prcng-based collaboratve optmzatons for green data centers, n Proc. IEEE 29th Int. Conf. Data Eng. Workshop (ICDEW), Brsbane, QLD, Australa, Apr. 2013, pp. 272 278. [9] H. Wang, J. Huang, X. Ln, and H. Mohsenan-Rad, Explorng smart grd and data center nteractons for electrc power load balancng, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 3, pp. 89 94, Jan. 2014. [10] Offce of the Natonal Coordnator for Smart Grd Interoperablty, NIST framework and roadmap for smart grd nteroperablty standards, Nat. Inst. Stand. Technol., Gathersburg, MD, USA, Tech. Rep. 1108, 2010. [11] X. Fang, S. Msra, G. Xue, and D. Yang, Smart Grd The new and mproved power grd: A survey, IEEE Commun. Surveys Tuts., vol. 14, no. 4, pp. 944 980, Jan. 2012. [12] F. Rahm and A. Ipakch, Demand response as a market resource under the smart grd paradgm, IEEE Trans. Smart Grd, vol. 1, no. 1, pp. 82 88, Jun. 2010. [13] A. Faruqu, Natonal assessment of demand response potental, Fed. Energy Reg. Comm., Washngton, DC, USA, Tech. Rep., 2009. [Onlne]. Avalable: http://www.ferc.gov/legal/staff-reports/ 06-09-demand-response.pdf [14] W. Saad, Z. Han, H. Poor, and T. Basar, Game-theoretc methods for the smart grd: An overvew of mcrogrd systems, demand-sde management, and smart grd communcatons, IEEE Sgnal Process. Mag., vol. 29, no. 5, pp. 86 105, Sep. 2012. [15] A.-H. Mohsenan-Rad et al., Autonomous demand-sde management based on game-theoretc energy consumpton schedulng for the future smart grd, IEEE Trans. Smart Grd, vol. 1, no. 3, pp. 320 331, Dec. 2010. [16] Z. Baharloue, M. Hashem, H. Narman, and H. Mohsenan-Rad, Achevng optmalty and farness n autonomous demand response: Benchmarks and bllng mechansms, IEEE Trans. Smart Grd, vol. 4, no. 2, pp. 968 975, Jun. 2013. [17] N. L, L. Chen, and S. H. Low, Optmal demand response based on utlty maxmzaton n power networks, n Proc. IEEE Power Energy Soc. Gen. Meetng, San Dego, CA, USA, 2011, pp. 1 8. [18] Z. Lu, M. Ln, A. Werman, S. H. Low, and L. L. H. Andrew, Greenng geographcal load balancng, n Proc. ACM SIGMETRICS Jont Int. Conf. Meas. Model. Comput. Syst., San Jose, CA, USA, 2011, pp. 233 244. [19] S. Ren and Y. He, COCA: Onlne dstrbuted resource management for cost mnmzaton and carbon neutralty n data centers, n Proc. Int. Conf. Hgh Perform. Comput. Network. Storage Anal. (SC), Denver,CO, USA, Nov. 2013, pp. 1 12. [20] G. Ghatkar, V. Gant, N. Matson, and M. A. Pette, Demand response opportuntes and enablng technologes for data centers: Fndngs from feld studes, Lawrence Berkeley Nat. Lab., Berkeley, CA, USA, Tech. Rep. LBNL-5763E, 2012. [Onlne]. Avalable: http://drrc.lbl.gov/stes/all/fles/lbnl-5763e.pdf [21] Z. Lu, A. Werman, Y. Chen, B. Razon, and N. Chen, Data center demand response: Avodng the concdent peak va workload shftng and local generaton, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 1, p. 341 342, Jun. 2013. [22] A. Werman, Z. Lu, and H. Mohsenan-Rad, Opportuntes and challenges for data center demand response, n Proc. IEEE Int. Green Comput. Conf. (IGCC), Dallas, TX, USA, 2014, pp. 1 10. [23] M. Ghamkhar and H. Mohsenan-Rad, Proft maxmzaton and power management of green data centers supportng multple SLAs, n Proc. IEEE Int. Comput. Netw. Commun. (ICNC), San Dego, CA, USA, 2013, pp. 465 469.

TRAN et al.: HOW GEO-DISTRIBUTED DCs DO DR: GAME-THEORETIC APPROACH 947 [24] S. Boyd and L. Vandenberghe, Convex Optmzaton. Cambrdge, U.K.: Cambrdge Unv. Press, Mar. 2004. [25] Z. Lu, M. Ln, A. Werman, S. H. Low, and L. L. H. Andrew, Geographcal load balancng wth renewables, ACM SIGMETRICS Perform. Eval. Rev., vol. 39, no. 3, pp. 62 66, Dec. 2011. [26] L. P. Qan, Y. J. A. Zhang, J. Huang, and Y. Wu, Demand response management va real-tme electrcty prce control n smart grds, IEEE J. Sel. Areas Commun., vol. 31, no. 7, pp. 1268 1280, Jul. 2013. [27] A. Mas-Colell, M. Whnston, and J. Green, Mcroeconomc Theory. New York, NY, USA: Oxford Unv. Press, 1995. [28] W.-M. Ln and H.-C. Chn, A current ndex based load balance technque for dstrbuton systems, n Proc. IEEE Int. Conf. Power Syst. Technol. (POWERCON), vol. 1. Bejng, Chna, 1998, pp. 223 227. [29] M. Kashem and V. Ganapathy, Three-phase load balancng n dstrbuton systems usng ndex measurement technque, Int. J. Elect. Power Energy Syst., vol. 24, no. 1, pp. 31 40, Jan. 2002. [30] D. P. Bertsekas and J. N. Tstskls, Parallel and Dstrbuted Computaton: Numercal Methods. Englewood Clffs, NJ, USA: Prentce-Hall, Jan. 1989. [31] P. X. Gao, A. R. Curts, B. Wong, and S. Keshav, It s not easy beng green, n Proc. ACM SIGCOMM Conf. Appl. Technol. Archt. Protocols Comput. Commun., Helsnk, Fnland, Aug. 2012, pp. 211 222. [32] H. Xu and B. L, Reducng electrcty demand charge for data centers wth partal executon, n Proc. 5th Int. Conf. Future Energy Syst. (e-energy), Cambrdge, U.K., Jun. 2014, pp. 51 61. [33] M. Ln, A. Werman, L. L. H. Andrew, and E. Thereska, Dynamc rghtszng for power-proportonal data centers, n Proc. IEEE INFOCOM, Shangha, Chna, Apr. 2011, pp. 1098 1106. [34] C. Wang, B. Urgaonkar, and Q. Wang, Data center cost optmzaton va workload modulaton under real-world electrcty prcng, n Proc. 6th IEEE/ACM Int. Conf. Utlty Cloud Comput. (UCC), Dec. 2013, pp. 1 14. [Onlne]. Avalable: http://csl.cse.psu.edu/publcatons/ucc13.pdf [35] J. B. Rosen, Exstence and unqueness of equlbrum ponts for concave N-person games, Econometrca, vol. 33, no. 3, pp. 520 534, Jul. 1965. Nguyen H. Tran (S 10 M 11) receved the B.S. degree from the Vetnam Unversty of Technology, Ho Ch Mnh Cty, Vetnam, n 2005, and the Ph.D. degree from Kyung Hee Unversty, Yongn, Korea, n 2011, both n electrcal and computer engneerng. Snce 2012, he has been an Assstant Professor wth the Department of Computer Engneerng, Kyung Hee Unversty. Hs current research nterests nclude queueng theory, optmzaton theory, control theory, and game theory to desgn, analyze, and optmze the cuttng-edge applcatons n communcaton networks, such as cogntve rado, cloud-computng data center, smart grd, heterogeneous networks, and femto cell. Da H. Tran receved the B.S. degree n nformaton technology from the Fontys Unversty of Appled Scence, Endhoven, The Netherlands, n 2011. He s currently pursung the Master s degree wth Kyung Hee Unversty, Yongn, Korea. Snce 2013, he has been wth the Department of Computer Engneerng, Kyung Hee Unversty. Hs current research nterests nclude extendng the capabltes of moble system through moble cloud computng. Shaole Ren (M 13) receved the B.E. degree from Tsnghua Unversty, Bejng, Chna, n 2006; the M.Phl. degree from the Hong Kong Unversty of Scence and Technology, Hong Kong, n 2008; and the Ph.D. degree from the Unversty of Calforna Los Angeles, Los Angeles, CA, USA, n 2012, all n electrcal engneerng. Snce 2012, he has been an Assstant Professor wth the School of Computng and Informaton Scences, Florda Internatonal Unversty, Mam, FL, USA, and holds a jont appontment wth the Department of Electrcal and Computer Engneerng. Hs current research nterests nclude sustanable computng, data center resource management, and network economcs. Dr. Ren was a recpent of the Best Paper Award at the Internatonal Workshop on Feedback Computng (co-located wth USENIX ICAC) n 2013 and the IEEE Internatonal Conference on Communcatons n 2009. Zhu Han (S 01 M 04 SM 09 F 14) receved the B.S. degree n electronc engneerng from Tsnghua Unversty, Bejng, Chna, n 1997, and the M.S. and Ph.D. degrees n electrcal engneerng from the Unversty of Maryland, College Park, MD, USA, n 1999 and 2003, respectvely. From 2000 to 2002, he was a Research and Development Engneer wth JDSU, Germantown, MD. From 2003 to 2006, he was a Research Assocate wth the Unversty of Maryland. From 2006 to 2008, he was an Assstant Professor wth Bose State Unversty, Bose, ID, USA. He s currently an Assocate Professor wth the Department of Electrcal and Computer Engneerng, Unversty of Houston, Houston, TX, USA. Hs current research nterests nclude wreless resource allocaton and management, wreless communcatons and networkng, game theory, wreless multmeda, securty, and smart grd communcaton. Dr. Han was a recpent of the Natonal Scence Foundaton CAREER Award n 2010 and the IEEE Fred W. Ellersck Prze Award n 2011. He has been an Assocate Edtor of the IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS snce 2010 and an IEEE Dstngushed Lecturer snce 2015. Eu-Nam Huh (M 15) receved the B.S. degree from Busan Natonal Unversty, Pusan, Korea; the M.S. degree n computer scence from the Unversty of Texas, Austn, TX, USA, n 1995; and the Ph.D. degree n computer engneerng from Oho Unversty, Athens, OH, USA, n 2002. From 2001 to 2002, he was the Drector of the Computer Informaton Center and an Assstant Professor wth Sahmyook Unversty, Seoul, Korea. He was an Assstant Professor wth Seoul Women s Unversty, Seoul. He s currently a Professor wth the Department of Computer Engneerng, Kyung Hee Unversty, Yongn, Korea. Hs current research nterests nclude hgh-performance networks, sensor networks, dstrbuted real-tme systems, grds, cloud computng, and network securty. Prof. Huh was the Program Char of the Workshop on Parallel and Dstrbuted Real-Tme Systems/Internatonal Parallel and Dstrbuted Processng Symposum n 2003. Snce 2002, he has been the Char of the Korea Grd Standard Group. He was an Edtor of the Journal of Korean Socety for Internet Informaton. Choong Seon Hong (S 95 M 97 SM 11) receved the B.S. and M.S. degrees n electronc engneerng from Kyung Hee Unversty, Yongn, Korea, n 1983 and 1985, respectvely, and the Ph.D. degree n computer engneerng from Keo Unversty, Mnato, Japan, n 1997. In 1988, he joned Korea Telecom (KT), Seoul, Korea, where he was a Techncal Staff Member of Broadband Networks. In 1993, he joned Keo Unversty. He was wth Telecommuncatons Network Laboratory, KT, as a Techncal Staff Senor Member and the Drector of the Networkng Research Team untl 1999. Snce 1999, he has been a Professor wth the Department of Computer Engneerng, Kyung Hee Unversty. Hs current research nterests nclude future Internet, ad hoc networks, network management, and network securty. Dr. Hong has served as a General Char, the TPC Char/Member, or an Organzng Commttee Member for nternatonal conferences such as NOMS, IM, APNOMS, E2EMON, CCNC, ADSN, ICPP, DIM, WISA, BcN, TINA, SAINT, and ICOIN. He s an Assocate Edtor of the IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, theinternatonal Journal of Network Management, andthejournal of Communcatons and Networks, and an Assocate Techncal Edtor of the IEEE COMMUNICATIONS MAGAZINE. He s a member of ACM, IEICE, IPSJ, KIISE, KICS, KIPS, and OSIA.