Seman&c Learning. Hanna Hajishirzi. (Some slides taken from semantic parsing tutorial)

Transcription

1 Seman&c Learning Hanna Hajishirzi (Some slides taken from semantic parsing tutorial)

2 Outline Semantics Semantic Parsing Grounded Language Acquisition

3 Seman&c Parsing Texas borders Kansas. next-to(tex,kan) Texas borders Kansas. What states border Texas? next-to(tex,kan) λx.state(x) next-to(x,tex)

4 Seman&c Parsing Seman&c Parsing: Transforming natural language (NL) sentences into computer executable complete meaning representa&ons (MRs) for domain- specific applica&ons Realis&c seman&c parsing currently entails domain dependence Example applica&on domains ATIS: Air Travel Informa&on Service CLang: Robocup Coach Language Geoquery: A Database Query Applica&on

5 ATIS: Air Travel Informa&on Service Interface to an air travel database [Price, 1990] Widely- used benchmark for spoken language understanding May I see all the Nlights from Cleveland to Dallas? Semantic Parsing Air- Transportation Show: (Flight- Number) Origin: (City "Cleveland") Destination: (City "Dallas") Query NA 1439, TQ 23,

6 CLang: RoboCup Coach Language In RoboCup Coach compe&&on teams compete to coach simulated players [h4p:// The coaching instruc&ons are given in a computer language called CLang [Chen et al. 2003] If the ball is in our goal area then player 1 should intercept it. Simulated soccer field Semantic Parsing (bpos (goal-area our) (do our {1} intercept)) CLang

7 Geoquery: A Database Query Applica&on Query applica&on for U.S. geography database containing about 800 facts [Zelle & Mooney, 1996] Which rivers run through the states bordering Texas? Semantic Parsing answer(traverse(next_to(stateid( texas )))) Arkansas, Canadian, Cimarron, Gila, Mississippi, Rio Grande Query Answer

8 Meaning Representa&on Languages Meaning representa&on language (MRL) for an applica&on is assumed to be present MRL is designed by the creators of the applica&on to suit the applica&ons needs independent of natural language MRL is unambiguous by design

9 Engineering Mo&va&on for Seman&c Parsing Applica&ons of domain- dependent seman&c parsing Natural language interfaces to compu&ng systems Communica&on with robots in natural language Personalized so_ware assistants Ques&on- answering systems Machine learning makes developing seman&c parsers for specific applica&ons more tractable

10 Dis&nc&ons from Other NLP Tasks Shallow seman&c processing Informa&on extrac&on Seman&c role labeling Intermediate linguis&c representa&ons Part- of- speech tagging Syntac&c parsing Seman&c role labeling Output meant for humans Ques&on answering Summariza&on Machine transla&on

11 Dis&nc&ons from Other NLP Tasks: Deeper Seman&c Analysis Show the long Alice sent me yesterday Semantic Parsing

12 Relations to Other NLP Tasks Tasks being performed within semantic parsing Word sense disambiguation Syntactic parsing as dictated by semantics Tasks closely related to semantic parsing Machine translation Natural language generation

13 Relations to Other NLP Tasks: Word Sense Disambiguation Semantic parsing includes performing word sense disambiguation Which rivers run through the states bordering Mississippi? Semantic Parsing State? River? answer(traverse(next_to(stateid( mississippi ))))

14 Rela&ons to Other NLP Tasks: Machine Transla&on The MR could be looked upon as another NL [Papineni et al., 1997; Wong & Mooney, 2006] Which rivers run through the states bordering Mississippi? answer(traverse(next_to(stateid( mississippi ))))

15 Rela&ons to Other NLP Tasks: Natural Language Genera&on Reversing a seman&c parsing system becomes a natural language genera&on system [Jacobs, 1985; Wong & Mooney, 2007a] Which rivers run through the states bordering Mississippi? Semantic Parsing NL Generation answer(traverse(next_to(stateid( mississippi ))))

16 Outline Semantics Semantic Parsing Grounded Language Acquisition

17 Seman&c Learning Texas borders Kansas. What states border Texas? next-to(tex,kan) λx.state(x) next-to(x,tex) Machine Learning Problem: Given: many input, output pairs Learn: a func&on that maps sentences to meaning representa&on language

18 Learning Semantic Parsers Training Sentences & Meaning Representa&ons Seman&c Parser Learner Novel sentence Seman&c Parser Meaning Representa&on

19 Semantic Parsing using CCG Zettlemoyer & Collins (2005)

20 Combinatory Categorial Grammar (CCG) Highly structured lexical entries A few general parsing rules (Steedman, 2000; Steedman & Baldridge, 2005) Each lexical entry is a word paired with a category Texas := NP borders := (S \ NP) / NP Mexico := NP New Mexico := NP

21 Parsing Rules (Combinators) Describe how adjacent categories are combined Functional application: A / B B A (>) B A \ B A (<) Forward composition Backward composition Texas borders New Mexico NP (S \ NP) / NP NP S S \ NP > <

22 CCG for Semantic Parsing Extend categories with semantic types (Lambda calculus expressions) Texas := NP : texas borders := (S \ NP) / NP : λx.λy.borders(y, x) Lexicon Lexical'Entry Category Text Syntax λ

23 Sample CCG Derivation Texas borders New Mexico NP texas (S \ NP) / NP λx.λy.borders(y, x) NP new_mexico S \ NP λy.borders(y, new_mexico) S borders(texas, new_mexico) > <

24 Another Sample CCG Derivation Texas touches New Mexico NP texas (S \ NP) / NP λx.λy.borders(y, x) NP new_mexico S \ NP λy.borders(y, new_mexico) S borders(texas, new_mexico) > <

25 'Probabilis=c'CCGs Probabilis&c CCGs Lexicon: Parameters: Λ= Texas NP tex, θ Probability'distribu=on:''sentence x, parse y,!logical'form z Log?linear'model: Parsing: ed as: P (y, z x;, )= e (x,y,z) (y,z ) e (x,y,z ) on 7 defines the features used in the ex f(x) = arg max p(z x;, ) z where p(z x;, )= y p(y, z x;, )

26 Learning Probabilistic CCG Training Sentences & Logical Forms Lexical Generation Lexicon L Parameter Estimation Sentences CCG Parser Feature weights w Logical Forms

27 Lexical Generation Input sentence: Texas borders New Mexico Output substrings: Texas borders New Mexico Texas borders borders New New Mexico Texas borders New Input logical form: borders(texas, new_mexico) Output categories:

28 Input Trigger constant c arity one predicate p arity one predicate p Category Rules Output Category NP : c N : λx.p(x) S \ NP : λx.p(x) arity two predicate p (S \ NP) / NP : λx.λy.p(y, x) arity two predicate p (S \ NP) / NP : λx.λy.p(x, y) arity one predicate p arity two predicate p and constant c arity two predicate p arity one function f arity one function f N / N : λg.λx.p(x) g(x) N / N : λg.λx.p(x, c) g(x) (N \ N) / NP : λx.λg.λy.p(y, x) g(x) NP / N : λg.argmax/min(g(x), λx.f(x)) S / NP : λx.f(x)

29 Input sentence: Texas borders New Mexico Output substrings: Texas borders New Mexico Texas borders borders New New Mexico Texas borders New Lexical Generation Input logical form: borders(texas, new_mexico) Output categories: NP : texas NP : new _mexico (S \ NP) / NP : λx.λy.borders(y, x) (S \ NP) / NP : λx.λy.borders(x, y) Take cross product to form an initial lexicon, include some domain independent entries What S/(S\NP)/N : λf.λg.λx.f(x) g(x)

30 Parameter Estimation A lexicon can lead to mul&ple parses, parameters decide the best parse Maximum condi&onal likelihood: Itera&vely find parameters that maximizes the probability of the training data Deriva&ons d are not annotated, treated as hidden variables Keep only those lexical items that occur in the highest scoring deriva&ons of training set

31 A Machine Transla&on Approach to Seman&c Parsing Wong & Mooney (2006) Based on a seman&c grammar of the natural language Uses machine transla&on techniques Synchronous context- free grammars (SCFG) Word alignments (Brown et al., 1993)

32 Seman&c Parsing Grammar rules are extracted from word alignments between training sentences and their meaning representa&ons Which rivers run through the states bordering Mississippi? answer(traverse(next_to(stateid( mississippi ))))

33 Semantic Parsing Semantic parsing maps NL sentences to completely formal MRs. Semantic parsers can be effectively learned from supervised corpora consisting of only sentences paired with their formal MRs (and possibly also SAPTs). The state-of-the-art in semantic parsing has been significantly advanced in recent years using a variety of statistical machine learning techniques and grammar formalisms

34 Domain- Specific Seman&c Parsing Generating training corpora is expensive Use domain knowledge to semantic parsing RoboCup Soccer domain (Hajishirzi et al., 2011)

35 Domain Knowledge Variables Team Player. Constants pink: team purple: team Pink1: player. Predicates Holding(Pink1, ball) AtCorner(ball) AtOffside(ball) Ac&ons Pass(Player1, Player2) BadPass(Player1, Player2) Kick(Player1) Dribble(Player1) Pass(Player1, Player2) Precondi&ons: Effects: Holding(Player1, ball) Holding(Player2, ball) 35

36 Seman&c Parsing with Domain Knowledge Map to events specinied in a logical format Pink7 makes a bad pass that was picked off by Purple7 Purple7 passes forward to Purple9 Purple9 kicks to Purple10 badpass kick kick Deeper understanding of narratives Uses domain knowledge in narrative understanding 36

37 Map Narra&ves to Events Challenge: Long narratives Inconsistent consecutive sentences No labeled data Solution: Use prior knowledge about event descriptions Use iterative learning Use consistency checking 37

38 Sentences sent1: P7 passes forward to P9. sent2: P9 kicks the ball to P10. Prior Knowledge Pass(x,y): Pre: holding(x) Eff:holding(y) Steal(x): Pre: ~holding(x) Eff: holding(x) <sent1, steal(p7,p9), true> Examples(sentence,event, state) <sent2, kick(p9), holding(p7)> Vote Generato r Check whether the event can happen in the current state sent1,steal(p7,p9),true + sent2,kick(p9),holding(p7) - Reasoning New weights ClassiNier P(event sentence,state) 38

39 Map Narra&ves to Event Sequences Find the most likely path badpass kick kick 39

40 Outline Semantics Semantic Parsing Grounded Language Acquisition

41 Grounded Language Acquisi&on Collecting Domain SpeciNic Knowledge is not always feasible Children acquire language through exposure to linguistic input in the context of a rich, relevant, perceptual environment

42 Grounded Seman&cs Learn to ground the semantics of language Block Learn language through correlated linguistic and world state

43 Grounded Seman&cs Game Events World Cup Live 94' 43 Essien is penalised for a challenge on Fabregas in midnield. Time! Type! Quali;ier! Team! Player! 2592! Pass! Throw In! Chelsea! Kalou! 2594! Tackle! Long Ball! Arsenal! Clichy! 2594! Disposses s! Alex Song celebrates giving Arsenal a first-ha Through Ball! Chelsea! Essien! 2596! foul! Through Ball! Chelsea! Essien! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! Grounded language acquisition: Learn to correspond text to a part of world state

44 Challenges 1. Not all events are mentioned in the sentence Commentary Game Events World Cup Live 94' 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! Alex Song celebrates giving Arsenal a first-ha 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

45 Challenges 2. Some sentences correspond to no event Statistics, game analysis 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

46 Challenges Hand- annotated alignment is too expensive Commentary Game Events World Cup Live 94' 44 Song began the move [4], exchanged with Wilshere on the edge of the box [6,7] before stealing the ball off Fabregas' foot [8] and sliding it home with his left foot from ten yards out [9]. What a fantastic goal Time! Type! Quali;ier! Team! Player! Alex Song celebrates giving Arsenal a first-ha 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

47 Weak Supervision Commentary Game Events World Cup Live 94' 43 Essien penalised for a challenge on Fabregas in midnield. 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! 2592! Pass! Throw In! Chelsea! Kalou! 2594! Tackle! Long Ball! Arsenal! Clichy! 2594! Disposses s! Alex Song celebrates giving Arsenal a first-ha Through Ball! Chelsea! Essien! 2596! foul! Through Ball! Chelsea! Essien! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

48 Robocup Sportscaster Trace Natural Language Commentary Purple goalie turns the ball over to Pink8 Purple team is very sloppy today Pink8 passes the ball to Pink11 (Chen et.al 07,08) Meaning Representation badpass ( Purple1, Pink8 ) turnover ( Purple1, Pink8 ) kick ( Pink8) pass ( Pink8, Pink11 ) kick ( Pink11 ) Pink11 looks around for a teammate Pink11 makes a long pass to Pink8 Pink8 passes back to Pink11 kick ( Pink11 ) ballstopped kick ( Pink11 ) pass ( Pink11, Pink8 ) kick ( Pink8 ) pass ( Pink8, Pink11 )

49 Robocup Sportscaster Trace Natural Language Commentary Purple goalie turns the ball over to Pink8 Purple team is very sloppy today Pink8 passes the ball to Pink11 Meaning Representation badpass ( Purple1, Pink8 ) turnover ( Purple1, Pink8 ) kick ( Pink8) pass ( Pink8, Pink11 ) kick ( Pink11 ) Pink11 looks around for a teammate Pink11 makes a long pass to Pink8 Pink8 passes back to Pink11 kick ( Pink11 ) ballstopped kick ( Pink11 ) pass ( Pink11, Pink8 ) kick ( Pink8 ) pass ( Pink8, Pink11 )

50 System Overview Sportscaster Robocup Simulator Purple7 loses the ball to Pink2 Pink2 kicks the ball to Pink5 Pink5 makes a long pass to Pink8 Pink8 shoots the ball Pass ( Purple5, Purple7 ) Turnover ( purple7, pink2 ) Kick ( pink2 ) Pass ( pink2, pink5 ) Kick ( pink5 ) Pass ( pink5, pink8) Ballstopped Kick ( pink8 ) Ambiguous Training Data

51 System Overview Sportscaster Robocup Simulator Purple7 loses the ball to Pink2 Pink2 kicks the ball to Pink5 Pink5 makes a long pass to Pink8 Pink8 shoots the ball Ambiguous Training Data Pass ( Purple5, Purple7 ) Turnover ( purple7, pink2 ) Kick ( pink2 ) Pass ( pink2, pink5 ) Kick ( pink5 ) Pass ( pink5, pink8) Ballstopped Kick ( pink8 ) Initial Semantic Parser Semantic Parser Learner

52 System Overview Sportscaster Robocup Simulator Purple7 loses the ball to Kick ( pink2 ) Pink2 Pink2 kicks the ball to Pink5 Pass ( pink2, pink5 ) Pink5 makes a long pass to Pink8 Kick ( pink5 ) Pink8 shoots the ball Kick ( pink8 ) Unambiguous Training Data Purple7 loses the ball to Pink2 Pink2 kicks the ball to Pink5 Pink5 makes a long pass to Pink8 Pink8 shoots the ball Pass ( purple5, purple7 ) Turnover ( purple7, pink2 ) Kick ( pink2 ) Pass ( pink2, pink5 ) Kick ( pink5 ) Pass ( pink5, pink8) Ballstopped Kick ( pink8 ) Initial Semantic Parser Ambiguous Training Data

53 System Overview Sportscaster Robocup Simulator Purple7 loses the ball to Kick ( pink2 ) Pink2 Pink2 kicks the ball to Pink5 Pass ( pink2, pink5 ) Pink5 makes a long pass to Pink8 Kick ( pink5 ) Pink8 shoots the ball Kick ( pink8 ) Unambiguous Training Data Purple7 loses the ball to Pink2 Pink2 kicks the ball to Pink5 Pink5 makes a long pass to Pink8 Pink8 shoots the ball Ambiguous Training Data Pass ( purple5, purple7 ) Turnover ( purple7, pink2 ) Kick ( pink2 ) Pass ( pink2, pink5 ) Kick ( pink5 ) Pass ( pink5, pink8) Ballstopped Kick ( pink8 ) Semantic Parser Semantic Parser Learner

54 System Overview Sportscaster Robocup Simulator Purple7 loses the ball to Turnover ( purple7, Pink2 pink2 ) Pink2 kicks the ball to Pink5 Pass ( pink2, pink5 ) Pink5 makes a long pass to Pink8 Kick ( pink5 ) Pink8 shoots the ball Kick ( pink8 ) Unambiguous Training Data Purple7 loses the ball to Pink2 Pink2 kicks the ball to Pink5 Pink5 makes a long pass to Pink8 Pink8 shoots the ball Ambiguous Training Data Pass ( purple5, purple7 ) Turnover ( purple7, pink2 ) Kick ( pink2 ) Pass ( pink2, pink5 ) Kick ( pink5 ) Pass ( pink5, pink8) Ballstopped Kick ( pink8 ) Semantic Parser Semantic Parser Learner

55 System Overview Sportscaster Robocup Simulator Purple7 loses the ball to Turnover ( purple7, Pink2 pink2 ) Pink2 kicks the ball to Pink5 Pass ( pink2, pink5 ) Pink5 makes a long pass to Pink8 Kick ( pink5 ) Pink8 shoots the ball Kick ( pink8 ) Unambiguous Training Data Purple7 loses the ball to Pink2 Pink2 kicks the ball to Pink5 Pink5 makes a long pass to Pink8 Pink8 shoots the ball Ambiguous Training Data Pass ( purple5, purple7 ) Turnover ( purple7, pink2 ) Kick ( pink2 ) Pass ( pink2, pink5 ) Kick ( pink5 ) Pass ( pink5, pink8) Ballstopped Kick ( pink8 ) Semantic Parser Semantic Parser Learner

56 System Overview Sportscaster Robocup Simulator Purple7 loses the ball to Turnover ( purple7, Pink2 pink2 ) Pink2 kicks the ball to Pink5 Pass ( pink2, pink5 ) Pink5 makes a long pass to Pink8 Pass ( pink5, pink8) Pink8 shoots the ball Kick ( pink8 ) Unambiguous Training Data Purple7 loses the ball to Pink2 Pink2 kicks the ball to Pink5 Pink5 makes a long pass to Pink8 Pink8 shoots the ball Ambiguous Training Data Pass ( purple5, purple7 ) Turnover ( purple7, pink2 ) Kick ( pink2 ) Pass ( pink2, pink5 ) Kick ( pink5 ) Pass ( pink5, pink8) Ballstopped Kick ( pink8 ) Semantic Parser Semantic Parser Learner

57 (Hajishirzi et al., 2012) Professional Soccer Dataset Commentary Game Events World Cup Live 94' 43 Essien penalised for a challenge on Fabregas in midnield. 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! 2592! Pass! Throw In! Chelsea! Kalou! 2594! Tackle! Long Ball! Arsenal! Clichy! 2594! Disposses s! Alex Song celebrates giving Arsenal a first-ha Through Ball! Chelsea! Essien! 2596! foul! Through Ball! Chelsea! Essien! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

58 Challenges 1. Complex structure and paraphrases Commentary 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Game Events Time! Type! Quali;ier! Team! Player! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

59 Challenges 1. Complex structure and paraphrases Commentary 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Game Events Time! Type! Quali;ier! Team! Player! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

60 Challenges 1: Complex structure and paraphrases Commentary 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Game Events Time! Type! Quali;ier! Team! Player! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

61 Challenges 1. Complex structure and paraphrases Commentary 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Game Events Time! Type! Quali;ier! Team! Player! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

62 Challenges 2: Comments correspond to combination of events (Macro events) 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song! Arsenal is coming forward

63 Problem Commentary Game Events World Cup Live 94' 43 Essien penalised for a challenge on Fabregas in midnield. 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! 2592! Pass! Throw In! Chelsea! Kalou! 2594! Tackle! Long Ball! Arsenal! Clichy! 2594! Disposses s! Alex Song celebrates giving Arsenal a first-ha Through Ball! Chelsea! Essien! 2596! foul! Through Ball! Chelsea! Essien! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

64 Problem Commentary Game Events World Cup Live 94' 43 Essien penalised for a challenge on Fabregas in midnield. 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! 2592! Pass! Throw In! Chelsea! Kalou! 2594! Tackle! Long Ball! Arsenal! Clichy! 2594! Disposses s! Alex Song celebrates giving Arsenal a first-ha Through Ball! Chelsea! Essien! 2596! foul! Through Ball! Chelsea! Essien! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song!

65 How to Measure Correspondence? Model correspondence of a sentence and an event Pair every sentence with every event in the bucket Learn particular pattern to each pair 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Pass Head Chelsea Malouda 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Foul Head Arsenal Koschielny

66 How to Measure Correspondence? Model correspondence of a sentence and an event Pair every sentence with every event in the bucket Learn particular pattern to each pair 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Pass Head Chelsea Malouda 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Foul Head Arsenal Koschielny Intuition: A pair is good if the pattern of correspondence occurs frequently among the data

67 43 Essien penalised for a challenge on Fabregas in midnield. 44 Song began the move, exchanged with Wilshere on the edge of the box before stealing the ball off Fabregas' foot and sliding it home with his left foot from ten yards out. What a fantastic goal. Time! Type! Quali;ier! Team! Player! 2592! Pass! Throw In! Chelsea! Kalou! 2594! Tackle! Long Ball! Arsenal! Clichy! 2594! Disposs ess! Through Ball! Chelsea! Essien! 2596! foul! Through Ball! Chelsea! Essien! 2603! pass! Free Kick! Arsenal! Fabregas! 2605! Pass! Chipped! Arsenal! Clichy! 2609! Pass! Through Ball! Arsenal! Koscielny! 2615! Pass! Long Ball! Arsenal! Song! 2623! Pass! Head! Arsenal! Nasri! 2625! Pass! Through Ball! Arsenal! Song! 2627! Pass! Through Ball! Arsenal! Wilshere! 2629! Pass! Through Ball! Arsenal! Fabregas! 2630! Goal! Through Ball! Arsenal! Song! Pair- 1: Pair- 2: Pair- 3: Pair- 3: Pair- 4: Pair- 5: Pair- 6: Pair- 7: Pair- 8: Pair- 9: Pair- 10: S- 43 S- 43 S- 43 S- 43 S- 43 S- 43 S- 43 S- 44 S- 44 S- 44 S- 44 E- 1 E- 2 E- 3 E- 4 E- 5 E- 6 E- 7 E- 5 E- 6 E- 7 E- 8 For every sentence: Generate pairs of sentence and events in the bucket Pair- 11: Pair- 12: Pair- 13: Pair- 14: S- 44 S- 44 S- 44 S- 44 E- 9 E- 10 E- 11 E- 12

68 Measuring Correspondence Intuition: The pattern of correspondence in a pair appears more consistently across all games Consistency of a pair is measured by its popularity among other pairs Popularity of a pair requires a notion of similarity between pairs

69 Similarity between Pairs 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Pass Head Chelsea Malouda 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Foul Head Arsenal Koschielny

70 Similarity between Pairs Co- occurrence of words and arguments matter 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Pass Head Chelsea Malouda 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Foul Head Arsenal Koschielny

71 Similarity between Pairs Co- occurrence of words and arguments matter 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Pass Head Chelsea Malouda 1: Chelsea looking for penalty as Malouda s header hits Koscielny. Foul Head Arsenal Koschielny Discriminative Similarity Learn the correspondence pattern in each pair Discriminate the pair from the rest

72 S- 43 E- 1 Pair- 1: S- 43 E- 2 Pair- 2: S- 43 E- 3 Pair- 3: S- 43 E- 4 Pair- 3: S- 43 E- 5 Pair- 4: S- 43 E- 6 Pair- 5: S- 43 E- 7 Pair- 6: S- 44 E- 5 Pair- 7: S- 44 E- 6 Pair- 8: S- 44 E- 7 Pair- 9: S- 44 E- 8 Pair- 10: S- 44 E- 9 Pair- 11: S- 44 E- 10 Pair- 12: S- 44 E- 11 Pair- 13: S- 44 E- 12 Pair- 14: S- 44 E- 13 Pair- 15: Popularity Ranking S- 43 E- 5 Pair- 5: S- 43 E- 2 Pair- 2: S- 43 E- 1 Pair- 1: S- 43 E- 3 Pair- 3: S- 43 E- 4 Pair- 4: S- 43 E- 7 Pair- 7: S- 43 E- 6 Pair- 6: S- 44 E- 10 Pair- 10: S- 44 E- 8 Pair- 8: S- 44 E- 15 Pair- 15: S- 44 E- 12 Pair- 12: S- 44 E- 11 Pair- 11: S- 44 E- 3 Pair- 3: S- 44 E- 9 Pair- 9: S- 44 E- 14 Pair- 14: S- 44 E- 7 Pair- 7: Pair Models Macro- event? Macro- event?

73 Qualita&ve Results 4: Cole is sent off for a lunge on Koscielny, it was poor, it was late but I'm not entirely sure that should have been red. Foul Long Ball Liverpool J. Cole Foul Through Ball Arsenal L. Koscielny Card None Liverpool J. Cole

74 Qualita&ve Results 5: First attack for Drogba, outmuscling Sagna and sending an effort in from the edge of the box which is blocked, and Song then brings down Drogba for a free kick. Take On Head Pass Chelsea D. Drogba Challenge Through Ball Arsenal B. Sagna Save Through Ball Arsenal A. Song Foul Through Ball Arsenal A. Song

75 Thank You! More questions: