arxiv: v1 [cs.lg] 15 Mar 2019

Transcription

1 A Ranng Model Motivated by Nonnegative Matrix Factorization with Applications to Tennis Tournaents Rui Xia 1, Vincent Y. F. Tan 1, Louis Filstroff 2, and Cédric Févotte 2 1 Departent of Matheatics, National University of Singapore rui.xia@u.nus.edu, vtan@nus.edu.sg 2 IRIT, Université de Toulouse, CNRS, France {louis.filstroff, cedric.fevotte}@irit.fr arxiv: v1 [cs.lg] 15 Mar 2019 Abstract. We propose a novel ranng odel that cobines the Bradley-Terry-Luce probability odel with a nonnegative atrix factorization fraework to odel and uncover the presence of latent variables that influence the perforance of top tennis players. We derive an efficient, provably convergent, and nuerically stable ajorizationiniization-based algorith to axiize the likelihood of datasets under the proposed statistical odel. The odel is tested on datasets involving the outcoes of atches between 20 top ale and feale tennis players over 14 ajor tournaents for en (including the Grand Slas and the ATP Masters 1000) and 16 ajor tournaents for woen over the past 10 years. Our odel autoatically infers that the surface of the court (e.g., clay or hard court) is a key deterinant of the perforances of ale players, but less so for feales. Top players on various surfaces over this longitudinal period are also identified in an objective anner. Keywords: BTL ranng odel, Nonnegative atrix factorization, Majorization-iniization, Low-rank approxiation, Sports analytics. 1 Introduction The international ranngs for both ale and feale tennis players are based on a rolling 52-week, cuulative syste, where ranng points are earned fro players perforances at tournaents. However, due to the liited observation window, such a ranng syste is not sufficient if one would like to copare doinant players over a long period (say 10 years) as players peak at different ties. The ranng points that players accrue depend only on the stage of the tournaents reached by hi or her. Unlike the well-studied Elo rating syste for chess [1], one opponent s ranng is not taken into account, i.e., one will not be awarded with bonus points by defeating a top player. Furtherore, the current ranng syste does not take into account the players perforances under different conditions (e.g., surface type of courts). We propose a statistical odel to aeliorate the above-entioned shortcoings by (i) understanding the relative ranng of players over a longitudinal period and (ii) discovering the existence of any latent variables that influence players perforances. The statistical odel we propose is an aalgaation of two well-studied odels in the ranng and dictionary learning literatures, naely, the Bradley- Terry-Luce (BTL) odel [2, 3] for ranng a population of ites (in this case, tennis players) based on pairwise coparisons and nonnegative atrix factorization (NMF) [4, 5]. The BTL odel posits that given a pair of players (i, j) fro a population of players {1,..., N}, the probability that the pairwise coparison i beats j is true is given by Pr(i beats j) = 1 λ i λ i + λ j. (1)

2 M tournaents Wibledon Australian Open French Open. Rafal Nadal Novak Djokovic Roger Federer... N players Λ M K W K N H Figure 1: The BTL-NMF Model Thus, λ i R + := [0, ) can be interpreted as the sll level of player i. The row vector λ = (λ 1,..., λ N ) R 1 N + thus paraetrizes the BTL odel. Other ore general ranng odels are discussed in [6] but the BTL odel suffices as the outcoes of tennis atches are binary. NMF consists in the following proble. Given a nonnegative atrix Λ R M N +, one would like to find two atrices W R M K + and H R K N + such that their product WH serves as a good low-rank approxiation to Λ. NMF is a linear diensionality reduction technique that has seen a surge in popularity since the seinal papers by Lee and Seung [4, 7]. Due to the non-subtractive nature of the decoposition, constituent parts of objects can be extracted fro coplicated datasets. The atrix W, known as the dictionary atrix, contains in its coluns the parts, and the atrix H, known as the coefficient atrix, contains in its rows activation coefficients that encode how uch of each part is present in the coluns of the data atrix Λ. NMF has also been used successfully to uncover latent variables with specific interpretations in various applications, including audio signal processing [8], text ining analysis [9], and even analyzing soccer players playing style [10]. We cobine this fraework with the BTL odel to perfor a sports analytics task on top tennis players. 1.1 Main Contributions Model: In this paper, we aalgaate the aforeentioned odels to rank tennis players and uncover latent factors that influence their perforances. We propose a hybrid BTL-NMF odel (see Fig. 1) in which there are M different sll vectors λ, {1,..., M}, each representing players relative sll levels in various tournaents indexed by. These row vectors are stacked into an M N atrix Λ which is the given input atrix in an NMF odel. Algoriths and Theory: We develop coputationally efficient and nuerically stable ajorization-iniization (MM)-based algoriths [11] to obtain a decoposition of Λ into W and H that axiizes the likelihood of the data. Furtherore, by using ideas fro [12, 13], we prove that not only is the objective function onotonically non-decreasing along iterations, additionally, every liit point of the sequence of iterates of the dictionary and coefficient atrices is a stationary point of the objective function. Experients: We collected rich datasets of pairwise outcoes of N = 20 top ale and feale players and M = 14 (or M = 16) top tournaents over 10 years. Based on these datasets, our algorith yielded factor atrices W and H that allowed us to draw interesting conclusions about the existence of latent variable(s) and relative ranngs of doinant players over the past 10 years. In particular, we conclude that ale players perforances are influenced, to a large extent, by the surface of the court. In other words, the surface turns out to be the pertinent latent variable for ale players. This effect is, however, less pronounced for feale players. Interestingly, we are also able to validate via our odel, datasets, and algorith that Nadal is undoubtedly the King of Clay ; Federer, a precise and accurate server, is doinant on grass (a non-clay surface other than hard court) as evidenced by his winning of Wibledon on ultiple occasions; and Djokovic is a ore balanced top player regardless of 2

3 surface. Conditioned on playing on a clay court, the probability that Nadal beats Djokovic is larger than 1/2. Even though the results for the woen are less pronounced, our odel and longitudinal dataset confirs objectively that S. Willias, Sharapova, and Azarenka (in this order) are consistently the top three players over the past 10 years. Such results (e.g., that Sharapova is so consistent that she is second best over the past 10 years) are not directly deducible fro official ranngs because these ranngs are essentially instantaneous as they are based on a rolling 52-week cuulative syste. 1.2 Related Work Most of the works that incorporate latent factors in statistical ranng odels (e.g., the BTL odel) ake use of ixture odels. See, for exaple, [14 16]. While such odels are able to take into account the fact that subpopulations within a large population possess different sll sets, it is difficult to ake sense of what the underlying latent variable is. In contrast, by erging the BTL odel with the NMF fraework the latter encouraging the extraction of parts of coplex objects we are able to observe latent features in the learned dictionary atrix W (see Table 3) and hence to extract the seantic eaning of latent variables. In our particular application, it is the surface type of the court for ale tennis players. See Sec. 4.5 where we also show that our solution is ore stable and robust (in a sense to be ade precise) than that of the ixture-btl odel. The paper ost closely related to the present one is [17] in which a topic odelling approach was used for ranng. However, unlike our work in which continuous-valued sll levels in Λ are inferred, perutations (i.e., discrete objects) and their corresponding ixture weights were learned. We opine that our odel and results provide a ore nuanced and quantitative view of the relative sll levels between players under different latent conditions. 1.3 Paper Outline In Sec. 2, we discuss the proble setup, the statistical odel, and its associated likelihood function. In Sec. 3, we derive efficient MM-based algoriths to axiize the likelihood. In Sec. 4, we discuss nuerical results of extensive experients on real-world tennis datasets. We conclude our discussion in Sec Proble Setup, Statistical Model, and Likelihood 2.1 Proble Definition and Model Given N players and M tournaents over a fixed nuber of years (in our case, this is 10), we consider a dataset D := { } M {0, 1, 2,...} : (i, j) P, =1 where P denotes the set of gaes between pairs of players that have played at least once in tournaent, and is the nuber of ties that player i has beaten player j in tournaent over the fixed nuber of year. To odel the sll levels of each player, we consider a nonnegative atrix Λ of diensions M N. The (, i) th eleent [Λ] i represents the sll level of player i in tournaent. Our goal is to design an algorith to find a factorization of Λ into two nonnegative atrices W R M K + and H R K N + such that the likelihood of D under the BTL odel in (1) is axiized; this is the so-called axiu likelihood fraework. Here K in{m, N} is a sall integer so the factorization is low-rank. In Sec. 3.3, we discuss different strategies to noralize W and H so that they are easily interpretable, e.g., as probabilities. Roughly speang, the eventual interpretation of W and H is as follows. Each colun of the dictionary atrix W encodes the likelihood that a certain tournaent {1,..., M} belongs to a certain latent class (e.g., type of surface). Each row of the coefficient atrix encodes the player s sll level in a tournaent of a certain latent class. For exaple, referring to Fig. 1, if the latent classes indeed correspond to surface types, the (1, 1) entry of W could represent the likelihood that Wibledon is a tournaent that is played on clay. The (1, 1) entry of H could represent Nadal s sll level on clay. 3

4 2.2 Likelihood of the BTL-NMF Model According to the BTL odel and the notations above, the probability that player i beats player j in tournaent is Pr(i beats j in tournaent ) = [Λ] i [Λ] i + [Λ] j. We expect that Λ is close to a low-rank atrix as the nuber of latent factors governing players sll levels is sall. We would like to exploit the utual inforation or correlation between tournaents of siilar characteristics to find a factorization of Λ. If Λ were unstructured, we could solve M independent, tournaent-specific probles to learn (λ 1,..., λ M ). We replace Λ by WH and the likelihood over all gaes in all tournaents (i.e., of the dataset D), assuing conditional independence across tournaents and gaes, is p(d W, H) = M =1 (i,j) P ( [WH] i [WH] i + [WH] j ) b (). It is often ore tractable to iniize the negative log-likelihood. In the sequel, we regard this as our objective function which can be expressed as f(w, H) := log p(d W, H) M = =1 (i,j) P [ log ( [WH] i ) + log ( [WH]i + [WH] j ) ]. (2) 3 Algoriths and Theoretical Guarantees In this section, we describe the algorith to optiize (2), together with accopanying theoretical guarantees. We also discuss how we aeliorate nuerical probles while aintaining the desirable guarantees of the algorith. 3.1 Majorization-Miniization (MM) Algorith We now describe how we use an MM algorith [11] to optiize (2). The MM fraework iteratively solves the proble of iniizing a certain function f(x), but its utility is ost evident when the direct of optiization of f(x) is difficult. One proposes an auxiliary function or ajorizer u(x, x ) that satisfies the following two properties: (i) f(x) = u(x, x), x and (ii) f(x) u(x, x ), x, x (ajorization). In addition for a fixed value of x, the iniization of u(, x ) is assued to be tractable (e.g., there exists a closed-for solution for x = arg in x u(x, x )). Then, one adopts an iterative approach to find a sequence {x (l) } l=1. One observes that if x(l+1) = arg in x u(x, x (l) ) is a iniizer at iteration l + 1, then f(x (l+1) ) (ii) u(x (l+1), x (l) ) u(x (l), x (l) ) (i) = f(x (l) ). (3) Hence, if such an auxiliary function u(x, x ) can be found, it is guaranteed that the sequence of iterates results in a sequence of non-increasing objective values. Applying MM to our odel is slightly ore involved as we are trying to find two nonnegative atrices W and H. Borrowing ideas fro using MM in NMFs probles (see for exaple the works [18,19]), the procedure first updates W by keeping H fixed, then updates H by keeping W fixed to its previously updated value. We will describe, in the following, how to optiize the original objective in (2) with respect to W with fixed H as the other optiization proceeds in an alost 1 syetric fashion since Λ T = H T W T. As entioned above, the MM algorith requires us to construct an auxiliary function u 1 (W, W H) that ajorizes log p(d W, H). 1 The updates for W and H are not copletely syetric because the data is in the for of a 3-way tensor { }; this is also apparent in the objective in (2) and the updates in (4). 4

5 The difficulty in optiizing the original objective function in (2) is twofold. The first concerns the coupling of the two ters [WH] i and [WH] j inside the logarith function. We resolve this using a technique introduced by Hunter in [20]. It is known that for any concave function f, its first-order Taylor approxiation overestiates it, i.e., f(y) f(x) + f(x) T (y x). Since the logarith function is concave, we have the inequality log y log x + 1 x (y x) which is an equality when x = y. These two properties ean that the following is a ajorizer of the ter log([wh] i + [WH] j ) in (2): log ( [W (l) H] i + [W (l) H] j ) + [WH] i + [WH] j [W (l) H] i + [W (l) H] j 1. The second difficulty in optiizing (2) concerns the ter log([wh] i ) = log( k w h ). By introducing the ters γ (l) i := w(l) h /[W (l) H] i for k {1,..., K} (which have the property that k γ(l) i = 1) to the su in log( k w h ) as was done by Févotte and Idier in [18, Theore 1], and using the convexity of log x and Jensen s inequality, we obtain the following ajorizer of the ter log([wh] i ) in (2): k w (l) h ( w log [W (l) H] [W (l) H] i w (l) i ). The sae procedure can be applied to find an auxiliary function u 2 (H, H W) for the optiization for H. Miniization of the two auxiliary functions with respect to W and H leads to the following MM updates: w (l+1) h (l+1) (i,j) P (i,j) P j i:(i,j) P w (l) h(l) [W (l) H (l) ] i h (l) + h(l) kj [W (l) H (l) ] i + [W (l) H (l) ] j j i:(i,j) P ( b () w (l+1) h(l) [W (l+1) H (l) ] i, (4a) + ) w (l+1) ji [W (l+1) H (l) ] i + [W (l+1) H (l) ] j. (4b) Note that since we first update W, H is given and fixed which eans that it is indexed by the previous iteration l; as for the update of H, the newly calculated W at iteration l + 1 will be used. 3.2 Resolution of Nuerical Probles While the above updates guarantee that the objective function does not decrease, nuerical probles ay arise in the ipleentation of (4). Indeed, it is possible that [WH] i becoes extreely close to zero for soe (, i). To prevent such nuerical probles fro arising, our strategy is to add a sall nuber ɛ > 0 to every eleent of H in (2). The intuitive explanation that justifies this is that we believe that each player has soe default sll level in every type of tournaent. By odifying H to H + ɛ1, where 1 is the K N all-ones atrix, we have the following new objective function: f ɛ (W, H) := M =1 (i,j) P [ log ( [W(H + ɛ1)] i ) + log ( [W(H + ɛ1)] i + [W(H + ɛ1)] j ) ]. (5) Note that f 0 (W, H) = f(w, H), defined in (2). Using the sae ideas involving MM to optiize f(w, H) as in Sec. 3.1, we can find new auxiliary functions, 5

6 denoted siilarly as u 1 (W, W H) and u 2 (H, H W), leading to following updates w (l+1) h (l+1) (i,j) P (i,j) P w (l) (h(l) + ɛ) [W (l) (H (l) + ɛ1)] i h (l) + h(l) kj + 2ɛ [W (l) (H (l) + ɛ1)] i + [W (l) (H (l) + ɛ1)] j j i:(i,j) P j i:(i,j) P w (l+1) (h(l) + ɛ) [W (l+1) (H (l) + ɛ1)] i ( + ji )w (l+1) [W (l+1) (H (l) + ɛ1)] i + [W (l+1) (H (l) + ɛ1)] j, (6a) ɛ. (6b) Notice that although this solution successfully prevents division by zero (or sall nubers) during the iterative process, for the new update of H, it is possible h (l+1) becoes negative because of the subtraction by ɛ in (6b). To ensure h is nonnegative as required by the nonnegativity of NMF, we set h (l+1) ax { h(l+1), 0 }. (7) After this truncation operation, it is, however, unclear whether the likelihood function is non-decreasing, as we have altered the vanilla MM procedure. We now prove that f ɛ in (5) is non-increasing as the iteration count increases. Suppose for the (l + 1) st iteration for H (l+1), truncation to zero only occurs for the (k, i) th eleent and and all other eleents stay unchanged, eaning h (l+1) = 0 and h (l+1) k,i = h (l) k,i for all (k, i ) (k, i). We would like to show that f ɛ (W, H (l+1) ) f ɛ (W, H (l) ). It suffices to show u 2 ( H (l+1), H (l) W) f ɛ (W, H (l) ), because if this is true, we have the following inequality f ɛ (W, H (l+1) ) u 2 ( H (l+1), H (l) W) f ɛ (W, H (l) ), (8) where the first inequality holds as u 2 is an auxiliary function for H. The truncation is invoked only when the update in (6b) becoes negative, i.e., when w (l+1) (h(l) +ɛ) [W (l+1) (H (l) +ɛ1)] i j i:(i,j) P ɛ. ( + ji )w (l+1) [W (l+1) (H (l) +ɛ1)] i+[w (l+1) (H (l) +ɛ1)] j j i:(i,j) P Using this inequality and perforing soe straightforward but tedious algebra as shown in Sec. S-1 in the suppleentary aterial [21], we can justify the second inequality in (8) as follows f ɛ(w, H (l) ) u 2( H (l+1), H (l) W) j i:(i,j) P ( + ji )w [W(H (l) +ɛ1)] i+[w(h (l) +ɛ1)] j The last inequality follows because [ ( (l) h h (l) ɛ log ɛ +ɛ ) ] 0., W and H (l) are nonnegative, and h (l) h(l) ɛ log( +ɛ ɛ ) 0 since x log(x + 1) for all x 0 with equality at x = 0. Hence, the likelihood is non-decreasing during the MM update even though we (l+1) included an additional operation that truncates h < 0 to zero in (7). 3.3 Noralization It is well-known that NMF is not unique in the general case, and it is characterized by a scale and perutation indeterinacies [5]. For the proble at hand, for the learned W and H atrices to be interpretable as sll levels with respect to different latent variables, it is iperative we consider noralizing the appropriately after every MM iteration in (6). However, there are different ways to noralize the entries in the atrices and one has to ensure 6

7 Algorith 1 MM Alg. for BTL-NMF odel with colun noralization of W Input: M tournaents; N players; nuber of ties player i beats player j in tournaent in dataset D = { : i, j {1,..., N}, {1,..., M} } Init: Fix K N, ɛ > 0, τ > 0 and initialize W (0) R M K ++, H (0) R K N ++. while diff τ > 0 do (1) Update {1,..., M}, k {1,..., K}, i {1,..., N} w (l+1) = h (l+1) i,j = ax i,j { w (l) (h(l) +ɛ) [W (l) (H (l) +ɛ1)] i h (l) +h(l) kj +2ɛ [W (l) (H (l) +ɛ1)] i +[W (l) (H (l) +ɛ1)] j w (l+1) (h(l) +ɛ) j i [W (l+1) (H (l) +ɛ1)] i ( + )w (l+1) ji j i [W (l+1) (H (l) +ɛ1)] i +[W (l+1) (H (l) +ɛ1)] j ɛ, 0 (2) Noralize {1,..., M}, k {1,..., K}, i {1,..., N} w (l+1) w(l+1) ; ĥ (l+1) (l+1) ( ) w (l+1) h w (l+1) + ɛ w (l+1) 1 Calculate β = { (3) diff ax k,i ĥ(l+1) 1+KNɛ ax,k w (l+1) +KNɛ w (l), h (l+1), ax k,i ĥ(l+1) +(1 β)ɛ β h (l+1) h (l) } end while return (W, H) that fors a local axiizer of the likelihood p(d W, H) } that after noralization, the likelihood of the odel stays unchanged. This [W(H+ɛ1)] is tantaount to keeping the ratio i [W(H+ɛ1)] i+[w(h+ɛ1)] j unchanged for all (, i, j). The key observations here are twofold: First, concerning H, since ters indexed by (, i) and (, j) appear in the denoinator but only (, i) appears in the nuerator, we can noralize over all eleents of H to keep this fraction unchanged. Second, concerning W, since only ters indexed by ter appear both in nuerator and denoinator, we can noralize either rows or coluns as we will show in the following Row Noralization of W and Global Noralization of H Define the row sus of W as r := k w k,i and let α := h +KNɛ 1+KNɛ. Now consider the following operations: w w, and h h + (1 α)ɛ. r α The above update to obtain h ay result in it being negative; however, the truncation operation in (7) ensures that h is eventually nonnegative. 2 See (l+1) also the update to obtain h in Algorith 1. The operations above keep the likelihood unchanged and achieve the desired row noralization of W since k w w ( h + ɛ) k w ( h + ɛ) + k w ( h kj + ɛ) = k r ( h + ɛ) k ( h + ɛ) + w k r ( h kj + ɛ) = k w ( h +ɛ) α k w ( h +ɛ) α + k w ( h +ɛ) α = w r k w (h + ɛ) k w (h + ɛ) + k w (h kj + ɛ) Colun Noralization of W and Global Noralization of H Define the colun sus of W as c k := w k,i and let β := ĥ+knɛ 1+KNɛ. Now consider the following operations: w w, ĥ c h c k + ɛ(c k 1), and h ĥ + (1 β)ɛ. k β 2 One ight be tepted to noralize H + ɛ1 R K N +. This, however, does not resolve nuerical issues (analogous to division by zero in (4)) as soe entries of H + ɛ1 ay be zero. 7

8 This would keep the likelihood unchanged and achieve the desired colun noralization of W since k w w ( h + ɛ) k w ( h + ɛ) + k w ( h kj + ɛ) = k c k ( h + ɛ)c k = k w (ĥ+ɛ) β k w (ĥ+ɛ) β + k w (ĥ+ɛ) β = k w c k ( h + ɛ)c k + k w c k ( h kj + ɛ)c k k w (h + ɛ) k w (h + ɛ) + k w (h kj + ɛ). Using this noralization strategy, it is easy to verify that all the entries of Λ = WH su to one. 3 This allows us to interpret the entries of Λ as conditional probabilities. 3.4 Suary of Algorith Algorith 1 presents pseudo-code for optiizing (5) with coluns of W noralized. The algorith when the rows of W are noralized is siilar; we replace the noralization step with the procedure outlined in Sec In suary, we have proved that the sequence of iterates {(W (l), H (l) )} l=1 results in the sequence of objective functions {f ɛ (W (l), H (l) )} l=1 being non-increasing. Furtherore, if ɛ > 0, nuerical probles do not arise and with the noralization as described in Sec , the entries in Λ can be interpreted as conditional probabilities as we will further illustrate in Sec Convergence of Matrices {(W (l), H (l) )} l=1 to Stationary Points While we have proved that the sequence of objectives {f ɛ (W (l), H (l) )} l=1 is non-increasing (and hence it converges because it is bounded), it is not clear as to whether the sequence of iterates generated by the algorith {(W (l), H (l) )} l=1 converges and if so to what. We define the arginal functions f 1,ɛ (W H) := f ɛ (W, H) and f 2,ɛ (H W) := f ɛ (W, H). For any function g : D R, we let g (x; d) := li inf λ 0 (g(x + λd) g(x))/λ be the directional derivative of g at point x in direction d. We say that (W, H) is a stationary point of the iniization proble if the following two conditions hold: in f ɛ (W, H) (9) W R M K +,H R K N + f 1,ɛ(W; W W H) 0, f 2,ɛ(H; H H W) 0, W R M K +, H R K N +. This definition generalizes the usual notion of a stationary point when the function is differentiable and the doain is unconstrained (i.e., x is a stationary point if f(x) = 0). However, in our NMF setting, the atrices are constrained to be nonnegative, hence the need for this generalized definition. If the atrices are initialized to soe W (0) and H (0) that are (strictly) positive and ɛ > 0, then we have the following desirable property. Theore 1. If W and H are initialized to have positive entries (i.e., W (0) R M K ++ = (0, ) M K and H (0) R++ K N ) and ɛ > 0, then every liit point of {(W (l), H (l) )} l=1 generated by Algorith 1 is a stationary point of (9). Thus, apart fro ensuring that there are no nuerical errors, another reason why we incorporate ɛ > 0 in the odified objective function in (5) is because a stronger convergence guarantee can be ensured. The proof of this theore, provided in Sec. S-2 of [21], follows along the lines of the ain result in Zhao and Tan [12], which itself hinges on the convergence analysis of block successive iniization ethods provided by Razaviyayn, Hong, and Luo [13]. In essence, we need to verify that f 1,ɛ and f 2,ɛ together with their auxiliary functions u 1 and u 2 satisfy the five regularity conditions in Definition 3 of [12]. However, there are soe iportant differences vis-à-vis [12] (e.g., analysis of the noralization step in Algorith 1) which we describe in detail in Reark 1 of [21]. 3 We have i [Λ] i = i k w h = i k h w = k,i h = 1. 8

9 Against R. Nadal N. Djokovic R. Federer A. Murray R. Nadal N. Djokovic R. Federer A. Murray Table 1: Partial en s dataset for the French Open Male Feale Total Entries = = 6400 Nuber Percentage Nuber Percentage Non-zero % % Zeros on the diagonal % % Missing data % % True zeros % % Table 2: Sparsity of datasets { } 4 Nuerical Experients and Discussion In this section, we describe how the datasets are collected and provide interesting and insightful interpretations of the nuerical results. 4.1 Details on the Datasets Collected The Association of Tennis Professionals (ATP) is the ain governing body for ale tennis players. The official ATP website contains records of all atches played on tour. The tournaents of the ATP tour belong to different categories; these include the four Grand Slas, the ATP Masters 1000, etc. The points obtained by the players that ultiately deterine their ATP ranngs and qualification for entry and seeding in following tournaents depend on the categories of tournaents that they participate or win in. We selected the ost iportant M = 14 tournaents for en s dataset, i.e., tournaents that yield the ost ranng points which include the four Grand Slas, ATP World Tour Finals and nine ATP Masters 1000, listed in the first colun of Table 3. After deterining the tournaents, we selected N = 20 players. We wish to have as any atches as possible between each pair of players, so that the atrices { }, {1,..., M} would not be too sparse and the algorith would thus have ore data to learn fro. We chose players who both have the highest aount of participation in the M = 14 tournaents fro 2008 to 2017 and also played the ost nuber of atches played in the sae period. These players are listed in the first colun of Table 4. For each tournaent, we collected an N N atrix { }, where denotes the nuber of ties player i beat player j in tournaent. A subatrix consisting of the statistics of atches played between Nadal, Djokovic, Federer, and Murray at the French Open is shown in Table 1. We see that over the 10 years, Nadal beat Djokovic three ties and Djokovic beat Nadal once at the French Open. The governing body for woen s tennis is the Woen s Tennis Association (WTA) instead of the ATP. As such, we collected data fro WTA website. The selection of tournaents and players is siilar to that for the en. The tournaents selected include the four Grand Slas, WTA Finals, four WTA Preier Mandatory tournaents, and five Preier 5 tournaents. However, for the first Preier 5 tournaent of the season, the event is either held in Dubai or Doha, and the last tournaent was held in Tokyo between 2009 and 2013; this has since been replaced by Wuhan. We decide to treat these two events as four distinct tournaents held in Dubai, Doha, Tokyo and Wuhan. Hence, the nuber of tournaents chosen for the woen is M = 16. After collecting the data, we checked the sparsity level of the dataset D = { }. The zeros in D can be categorized into three different classes. 9

10 (a) K = 2 (b) K = 3 Figure 2: Histogra of negative log-likelihood in the 150 trials Tournaents Row Noralization Colun Noralization Australian Open 5.77E E E E-02 Indian Wells Masters 6.52E E E E-02 Miai Open 5.27E E E E-02 Monte-Carlo Masters 1.68E E E E-01 Madrid Open 3.02E E E E-01 Italian Open 0.00E E E E-01 French Open 3.44E E E E-01 Wibledon 6.43E E E E-02 Canadian Open 1.00E E E E-152 Cincinnati Masters 5.23E E E E-02 US Open 5.07E E E E-02 Shanghai Masters 7.16E E E E-02 Paris Masters 1.68E E E E-02 ATP World Tour Finals 5.72E E E E-02 Table 3: Learned dictionary atrix W for the en s dataset 1. (Zeros on the diagonal) By convention, ii = 0 for all (i, ); 2. (Missing data) By convention, if player i and j have never played with each other in tournaent, then = = 0; 3. (True zeros) If player i has played with player j in tournaent but lost every such atch, then = 0 and ji > 0. The distributions over the three types of zeros and non-zero entries for ale and feale players are presented in Table 2. We see that there is ore issing data in the woen s dataset. This is because there has been a sall set of doinant ale players (e.g., Nadal, Djokovic, Federer) over the past 10 years but the sae is not true for woen players. For the woen, this eans that the atches in the past ten years are played by a ore diverse set of players, resulting in the nuber of atches between the top N = 20 players being saller copared to the top N = 20 en, even though we have selected the sae nuber of top players. 4.2 Running of the Algorith The nuber of latent variables is expected to be sall and we set K to be 2 or 3. We only present results for K = 2 in the ain paper; the results corresponding to Tables 3 to 6 for K = 3 are displayed in Tables S-1 to S-4 in the suppleentary aterial [21]. We also set ɛ = which is close to the sallest positive value in the Python environent. The algorith terinates when the difference of every eleent of W and H between in the successive iterations is less than τ = We checked that the ɛ-odified algorith in Sec. 3.2 results in non-decreasing likelihoods. See Fig. S-1 in the suppleentary aterial [21]. Since (5) is non-convex, the MM algorith can be trapped in local inia. Hence, we considered 150 different rando initializations for W (0) 10

11 Players atrix H T Total Matches Novak Djokovic 1.20E E Rafael Nadal 2.48E E Roger Federer 1.15E E Andy Murray 7.57E E Toas Berdych 0.00E E David Ferrer 6.26E E Stan Wawrinka 2.93E E Jo-Wilfried Tsonga 3.36E E Richard Gasquet 5.49E E Juan Martin del Potro 2.90E E Marin Cilic 2.12E E Fernando Verdasco 1.36E E Kei Nishikori 7.07E E Gilles Sion 1.32E E Milos Raonic 1.45E E Philipp Kohlschreiber 2.18E E John Isner 2.70E E Feliciano Lopez 1.43E E Gael Monfils 3.86E E Nicolas Alagro 6.48E E Table 4: Learned transpose H T of the coefficient atrix for the en s dataset and H (0) and analyzed the result that gave the axiu likelihood aong the 150 trials. Histogras of the negative log-likelihoods are shown in Figs. 2(a) and 2(b) for K = 2 and K = 3 respectively. We observe that the optial value of the log-likelihood for K = 3 is higher than that of K = 2 since the forer odel is richer. We also observe that the W s and H s produced over the 150 runs are roughly the sae up to perutation of rows and coluns, i.e., our solution is stable and robust (cf. Theore 1 and Sec. 4.5). 4.3 Results for Men Players The learned dictionary atrix W is shown in Table 3. In the Tournaents colun, those tournaents whose surface types are known to be clay are highlighted in gray. For ease of visualization, higher values are shaded darker. If the rows of W are noralized, we observe that for clay tournaents, the value in the second colun is always larger than that in the first, and vice versa. The only exception is the Paris Masters. 4 Since the row sus are equal to 1, we can interpret the values in the first and second coluns of a fixed row as the probabilities that a particular tournaent is being played on non-clay or clay surface respectively. If the coluns of W are noralized, it is observed that the tournaents with highest value of the second colun are exactly the four tournaents played on clay. Fro W, we learn that surface type in particular, whether or not a tournaent is played on clay is a gerane latent variable that influences the perforances of en players. Table 4 displays the transpose of H whose eleents su to one. Thus, if the colun k {1, 2} represents the surface type, we can treat h as the sll of player i conditioned on hi playing on surface type k. We ay regard the first and second coluns of H T as the sll levels of players on non-clay and clay respectively. We observe that Nadal, nicknaed the King of Clay, is the best player on clay aong the N = 20 players, and as an individual, he is also uch ore slful on clay copared to non-clay. Djokovic, the first an in the Open era to hold all four Grand Slas on three different surfaces 4 This ay be attributed to its position in the seasonal calendar. The Paris Masters is the last tournaent before ATP World Tour Finals. Top players often choose to sp this tournaent to prepare for ATP World Tour Finals which is ore prestigious. This has led to soe surprising results, e.g., David Ferrer, a strong clay player, won the Paris Masters in 2012 (even though the Paris Masters is a hard court indoor tournaent). 11

12 Tournaent Novak Djokovic Rafael Nadal Roger Federer Andy Murray Toas Berdych David Ferrer Stan Wawrinka Jo-Wilfried Tsonga Richard Gasquet Juan Martin del Potro Australian Open 2.16E E E E E E E E E E-03 Indian Wells Masters 2.29E E E E E E E E E E-03 Miai Open 2.95E E E E E E E E E E-03 Monte-Carlo Masters 1.19E E E E E E E E E E-03 Madrid Open 1.38E E E E E E E E E E-03 Italian Open 1.19E E E E E E E E E E-03 French Open 1.39E E E E E E E E E E-03 Wibledon 2.63E E E E E E E E E E-03 Canadian Open 1.16E E E E E E E E E E-03 Cincinnati Masters 1.82E E E E E E E E E E-03 US Open 1.17E E E E E E E E E E-03 Shanghai Masters 8.12E E E E E E E E E E-03 Paris Masters 7.29E E E E E E E E E E-03 ATP World Tour Finals 1.13E E E E E E E E E E-03 Table 5: Learned Λ = WH atrix for first 10 en players Tournaent Marin Cilic Fernando Verdasco Gilles Sion Milos Raonic John Isner Philipp Kohlschreiber John Isner Feliciano Lopez Gael Monfils Nicolas Alagro Australian Open 2.36E E E E E E E E E E-04 Indian Wells Masters 2.79E E E E E E E E E E-04 Miai Open 2.98E E E E E E E E E E-04 Monte-Carlo Masters 4.10E E E E E E E E E E-04 Madrid Open 8.34E E E E E E E E E E-04 Italian Open 0.00E E E E E E E E E E-07 French Open 9.48E E E E E E E E E E-04 Wibledon 3.17E E E E E E E E E E-04 Canadian Open 2.05E E E E E E E E E E-04 Cincinnati Masters 1.82E E E E E E E E E E-04 US Open 1.14E E E E E E E E E E-04 Shanghai Masters 1.08E E E E E E E E E E-04 Paris Masters 2.51E E E E E E E E E E-05 ATP World Tour Finals 1.22E E E E E E E E E E-04 Table 6: Learned Λ = WH atrix for last 10 en players 12

13 Tournaents Row Noralization Colun Noralization Australian Open 1.00E E E E-23 Qatar Open 6.05E E E E-02 Dubai Tennis Chapionships 1.00E E E E-39 Indian Wells Open 5.64E E E E-02 Miai Open 5.86E E E E-02 Madrid Open 5.02E E E E-02 Italian Open 3.61E E E E-02 French Open 1.84E E E E-02 Wibledon 1.86E E E E-01 Canadian Open 4.59E E E E-02 Cincinnati Open 9.70E E E E-01 US Open 6.12E E E E-02 Pan Pacific Open 1.72E E E E-01 Wuhan Open 1.00E E E E-61 China Open 2.26E E E E-01 WTA Finals 1.17E E E E-02 Table 7: Learned dictionary atrix W for the woen s dataset (hard court, clay and grass) at the sae tie (between Wibledon 2015 to the French Open 2016, also known as the Nole Sla), is ore of a balanced top player as his sll levels are high in both coluns of H T. Federer won the ost titles on tournaents played on grass and, as expected, his sll level in the first colun is indeed uch higher than the second. As for Murray, the H T atrix also reflects his weakness on clay. Wawrinka, a player who is known to favor clay has sll level in the second colun being uch higher than that in the first. The last colun of Table 4 lists the total nuber of atches that each player participated in (within our dataset). We verified that the sll levels in H T for each player are not strongly correlated to how any atches are being considered in the dataset. Although Berdych has data of ore atches copared to Ferrer, his scores are not higher than that of Ferrer. Thus our algorith and conclusions are not skewed towards the availability of data. The learned sll atrix Λ = WH with colun noralization of W is presented in Tables 5 and 6. As entioned in Sec. 2.1, [Λ] i denotes the sll level of player i in tournaent. We observe that Nadal s sll levels are higher than Djokovic s only for the French Open, Madrid Open, Monte-Carlo Masters, Paris Masters and Italian Open, which are tournaents played on clay except for the Paris Masters. As for Federer, his sll level is highest for Wibledon, which happens to be the only tournaent on grass; here, it is known that he is the player with the best record in the Open era. Furtherore, if we consider Wawrinka, the five tournaents in which his sll levels are the highest include the four clay tournaents. These observations again show that our odel has learned interesting latent variables fro W. It has also learned players slls on different types of surfaces and tournaents fro H and Λ respectively. 4.4 Results for Woen Players We perfored the sae experient for the woen players except that we now consider M = 16 tournaents. The factor atrices W and H (in its transpose for) are presented in Tables 7 and 8 respectively. It can be seen fro W that, unlike for the en players, the surface type is not a pertinent latent variable since there is no correlation between the values in the coluns and the surface type. We suspect that the sll levels of top woen players are not as heavily influenced by the surface type copared to the en. However, the tournaents in Table 7 are ordered in chronological order and we notice that there is a slight correlation between the values in the colun and the tie of the tournaent (first half or second half of the year). Any latent variable would naturally be less pronounced, due to the sparser dataset for woen players (cf. Table 2). A soewhat interesting observation 13

14 Players atrix H T Total Matches Serena Willias 5.93E E Agnieszka Radwanska 2.39E E Victoria Azarenka 7.04E E Caroline Wozniac 3.03E E Maria Sharapova 8.38E E Siona Halep 1.50E E Petra Kvitova 2.39E E Angelique Kerber 6.81E E Saantha Stosur 4.15E E Ana Ivanovic 9.55E E Jelena Jankovic 1.17E E Anastasia Pavlyuchenkova 6.91E E Carla Suarez Navarro 3.51E E Doinika Cibulkova 2.97E E Lucie Safarova 0.00E E Elina Svitolina 5.03E E Sara Errani 7.99E E Karolina Pliskova 9.92E E Roberta Vinci 4.14E E Marion Bartoli 1.45E E Table 8: Learned transpose H T of coefficient atrix for the woen s dataset is that the values in W obtained using the row noralization and the colun noralization ethods are siilar. This indicates that the latent variables, if any, learned by the two ethods are the sae, which is a reassuring conclusion. By coputing the sus of the sll levels for each feale player (i.e., row sus of H T ), we see that S. Willias is the ost slful aong the 20 players over the past 10 years. She is followed by Sharapova and Azarenka. As a atter of fact, S. Willias and Azarenka have been year-end nuber one 4 ties and once, respectively, over the period 2008 to Even though Sharapova was never at the top at the end of any season (she was, however, ranked nuber one several ties, ost recently in 2012), she had been consistent over this period such that the odel and the longitudinal dataset allow us to conclude that she is ranked second. In fact, she is known for her unusual longevity being at the top of the woen s gae. She started her tennis career very young and won her first Grand Sla at the age of 17. Finally, the odel groups S. Willias, Sharapova, Stosur together, while Azarenka, Navarro, and Vinci are in another group. We believe that there ay be soe siilarities between players who are clustered in the sae group. The Λ atrix for woen players can be found in Tables S-5 and S-6 in the suppleentary aterial [21]. 4.5 Coparison to BTL and ixture-btl Finally, we copared our approach to the BTL and ixture-btl [14, 15] approaches for the ale players. To learn these odels, we aggregated our dataset { } into a single atrix {b = b() }. For the BTL odel, we axiized the likelihood to find the optial paraeters. For the ixture-btl odel with K = 2 coponents, we ran an Expectation-Maxiization (EM) algorith [22] to find approxiately-optial values of the paraeters and the ixture weights. Note that the BTL odel corresponds to a ixture-btl odel with K = 1. The learned sll vectors are shown in Table 9. Since EM is susceptible to being trapped in local optia and is sensitive to initialization, we ran it 100 ties and reported the solution with likelihood that is close to the highest one. 5 The solution for ixture-btl is not stable; other solutions with likelihoods that 5 The solution with the highest likelihood is shown in Trial 2 of Table S-7 but it appears that the solution there is degenerate. 14

15 Players K = 1 K = 2 Novak Djokovic 2.14E E E-01 Rafael Nadal 1.79E E E-02 Roger Federer 1.31E E E-02 Andy Murray 7.79E E E-03 Toas Berdych 3.09E E E-04 David Ferrer 3.72E E E-03 Stan Wawrinka 4.32E E E-03 Jo-Wilfried Tsonga 2.98E E E-01 Richard Gasquet 2.34E E E-03 Juan Martin del Potro 4.75E E E-02 Marin Cilic 1.86E E E-03 Fernando Verdasco 2.24E E E-09 Kei Nishikori 3.43E E E-02 Gilles Sion 1.90E E E-03 Milos Raonic 2.33E E E-03 Philipp Kohlschreiber 7.12E E E-03 John Isner 1.84E E E-08 Feliciano Lopez 1.89E E E-04 Gael Monfils 1.66E E E-03 Nicolas Alagro 7.24E E E-03 Mixture weights 1.00E E E-01 Log-likelihoods Table 9: Learned λ s for the BTL (K = 1) and ixture-btl (K = 2) odels are very close to the axiu one result in significantly different paraeter values. Two other solutions with siilar likelihoods are shown in Table S-7 in the suppleentary aterial [21]. As can be seen, soe of the solutions are far fro representative of the true sll levels of the players (e.g., in Trial 2 of Table S-7, Tsonga has a very high score in the first colun and the slls of the other players are all very sall in coparison) and they are vastly different fro one another. This is in stark contrast to our BTL-NMF odel and algorith in which Theore 1 states that the liit of {(W (l), H (l) )} l=1 is a stationary point of (9). We nuerically verified that the BTL-NMF solution is stable, i.e., different runs yield (W, H) pairs that are approxiately equal up to perutation of rows and coluns. 6 As seen fro Table 9, for ixture- BTL, neither tournaent-specific inforation nor seantic eanings of latent variables can be gleaned fro the paraeter vectors. The results of BTL are reasonable and expected but also lack tournaent-specific inforation. 5 Conclusion and Future Work We proposed a ranng odel cobining the BTL odel with the NMF fraework as in Fig. 1. We derived MM-based algoriths to axiize the likelihood of the data. To ensure nuerical stability, we regularized the MM algorith and proved that desirable properties, such as onotonicity of the objective and convergence of the iterates to stationary points, hold. We drew interesting conclusions based on longitudinal datasets for top ale and feale players. A latent variable in the for of the court surface was also uncovered in a principled anner. We copared our approach to the ixture-btl approach [14,15] and concluded that the forer is advantageous in various aspects (e.g., stability of the solution, interpretability of latent variables). In the future, we plan to run our algorith on a larger longitudinal dataset consisting of pairwise coparison data fro ore years (e.g., the past 50 years) to learn, for exaple, who is the best-of-all-tie ale or feale player. In addition, it would be desirable to understand if there is a natural Bayesian 6 Note, however, that stationary points are not necessarily equivalent up to perutation or rescaling. 15

16 interpretation [19, 23] of the ɛ-odified objective function in (5). References [1] A. E. Elo. The Rating of Chess Players, Past and Present. Ishi Press International, [2] R. Bradley and M. Terry. Rank analysis of incoplete block designs I: The ethod of paired coparisons. Bioetrika, 35: , [3] R. Luce. Individual choice behavior: A theoretical analysis. Wiley, [4] D. D. Lee and H. S. Seung. Learning the parts of objects with nonnegative atrix factorization. Nature, 401: , [5] A. Cichoc, R. Zdunek, A. H. Phan, and S.-I. Aari. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation. John Wiley & Sons, Ltd, [6] J. I. Marden. Analyzing and odeling rank data. CRC Press, [7] D. D. Lee and H. S. Seung. Algoriths for nonnegative atrix factorization. In Neural Inforation Processing Systes, pages , [8] C. Févotte, N. Bertin, and J.-L. Durrieu. Nonnegative atrix factorization with the Itakura-Saito divergence. with application to usic analysis. Neural Coputation, 21(3): , Mar [9] M. W. Berry and M. Browne. Eail surveillance using non-negative atrix factorization. Coputational and Matheatical Organization Theory, 11(3): , [10] A. Geerts, T. Decroos, and J. Davis. Characterizing soccer players playing style fro atch event streas. In Machine Learning and Data Mining for Sports Analytics ECML/PKDD 2018 workshop, pages , [11] D. R. Hunter and K. Lange. A tutorial on MM algoriths. Aerican Statistician, 58:30 37, [12] R. Zhao and V. Y. F. Tan. A unified convergence analysis of the ultiplicative update algorith for regularized nonnegative atrix factorization. IEEE Transactions on Signal Processing, 66(1): , [13] M. Razaviyayn, M. Hong, and Z.-Q. Luo. A unified convergence analysis of block successive iniization ethods for nonsooth optiization. SIAM Journal on Optiization, 23(2): , [14] S. Oh and D. Shah. Learning ixed ultinoial logit odel fro ordinal data. In Neural Inforation Processing Systes, pages , [15] N. B. Shah and M. J. Wainwright. Siple, robust and optial ranng fro pairwise coparisons. Journal of Machine Learning Research, 18(199):1 38, [16] C. Suh, V. Y. F. Tan, and R. Zhao. Adversarial top-k ranng. IEEE Transactions on Inforation Theory, 63(4): , [17] W. Ding, P. Ishwar, and V. Saligraa. A topic odeling approach to ranng. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS), pages , [18] C. Févotte and J. Idier. Algoriths for nonnegative atrix factorization with the β-divergence. Neural Coputation, 23(9): , [19] V. Y. F. Tan and C. Févotte. Autoatic relevance deterination in nonnegative atrix factorization with the β-divergence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7): , [20] D. R. Hunter. MM algoriths for generalized Bradley-Terry odels. The Annals of Statistics, 32(1): , [21] R. Xia, V. Y. F. Tan, L. Filstroff, and C. Févotte. Suppleentary aterial for a ranng odel otivated by nonnegative atrix factorization with applications to tennis tournaents. btl-nf, [22] A. P. Depster, N. M. Laird, and D. B. Rubin. Maxiu likelihood fro incoplete data via the EM algorith. Journal of the Royal Statistical Society B, 39(1 38), [23] F. Caron and A. Doucet. Efficient Bayesian inference for generalized Bradley-Terry odels. Journal of Coputational and Graphical Statistics, 21(1): ,