Sharing the How (and not the What )

Size: px

Start display at page:

Download "Sharing the How (and not the What )"

Willis Burke
5 years ago
Views:

1 Sharing the How (and not the What ) Percy Liang NIPS Workshop on Transfer Learning December 12, 2015

2 What is shared between tasks? Task A Task B 1

3 What is shared between tasks? Task A Task B 1

4 detecting trucks detecting cars 2

5 detecting trucks detecting cars sharing low-level representations ( what ) 2

6 tallest mountain in Quebec most popular restaurant in Palo Alto 3

7 tallest mountain in Quebec most popular restaurant in Palo Alto sharing abstractions ( how ) 3

8 Natural language interfaces 4

9 Natural language interfaces Tasks parametrized by natural language: User: Find me the best restaurants in Montreal. Computer: Toqué, Gibby s restaurant, Le Local,... 4

10 Natural language interfaces Tasks parametrized by natural language: User: Find me the best restaurants in Montreal. Computer: Toqué, Gibby s restaurant, Le Local,... User: Find me speakers at the transfer learning workshop. Computer: Bengio, Ben-David, Darrell, Liang, Mohri, Pontil, Salakhutdinov, Yang 4

11 Natural language interfaces Tasks parametrized by natural language: User: Find me the best restaurants in Montreal. Computer: Toqué, Gibby s restaurant, Le Local,... User: Find me speakers at the transfer learning workshop. Computer: Bengio, Ben-David, Darrell, Liang, Mohri, Pontil, Salakhutdinov, Yang Background: semantic parsing in NLP [Mooney, Artzi, Kwiatkowski, Zettlemoyer, Collins, Roth, Berant, Liang, etc.] 4

12 Outline Program induction Zero-shot entity extraction Question answering on new tables 5

13 Program induction [Liang et al., 2010, ICML] I like programs, but I wish programs would just program themselves since I don t like programming. I like programs, but I wish programs would just program themselves since I don t like programming. 6

14 Program induction [Liang et al., 2010, ICML] I like programs, but I wish programs would just program themselves since I don t like programming. I like programs, but I wish programs would just program themselves since I don t like programming. Goal: Programming by Demonstration [Lau et al. 2003; Liang et al. 2010; Gulwani et al. 2011] If the user demonstrates italicizing the first occurrence, can we generalize to the remaining? 6

15 Program induction [Liang et al., 2010, ICML] I like programs, but I wish programs would just program themselves since I don t like programming. I like programs, but I wish programs would just program themselves since I don t like programming. Goal: Programming by Demonstration [Lau et al. 2003; Liang et al. 2010; Gulwani et al. 2011] If the user demonstrates italicizing the first occurrence, can we generalize to the remaining? Solution: represent task by a program to be learned 1. Move to next occurrence of word with prefix program 2. Insert 3. Move to end of word 4. Insert 6

16 Program induction [Liang et al., 2010, ICML] I like programs, but I wish programs would just program themselves since I don t like programming. I like programs, but I wish programs would just program themselves since I don t like programming. Goal: Programming by Demonstration [Lau et al. 2003; Liang et al. 2010; Gulwani et al. 2011] If the user demonstrates italicizing the first occurrence, can we generalize to the remaining? Solution: represent task by a program to be learned 1. Move to next occurrence of word with prefix program 2. Insert 3. Move to end of word 4. Insert Challenge: learn from very few examples 6

17 Sharing programs [Liang et al., 2010, ICML] min(, ) max(, ) 7

18 Sharing programs [Liang et al., 2010, ICML] min(, ) max(, ) Find programs that share common subprograms. Programs do tend to share common components. Bayesian nonparametric prior over all programs 7

19 Refactoring [Liang et al., 2010, ICML] 8

20 Outline Program induction Zero-shot entity extraction Question answering on new tables 9

21 Zero-shot entity extraction [ACL 2014] Panupong (Ice) Pasupat 10

22 Semantic parsing on the web Input: query x hiking trails near Baltimore web page w 11

23 Semantic parsing on the web Input: query x hiking trails near Baltimore web page w 11

24 Semantic parsing on the web Input: query x hiking trails near Baltimore web page w 11

25 Semantic parsing on the web Input: query x hiking trails near Baltimore web page w Output: list of entities y [Avalon Super Loop, Patapsco Valley State Park,...] 11

26 [Sahuguet and Azavant, 1999; Liu et al., 2000; Crescenzi et al., 2001] Logical forms: XPath expressions html head body table h1 table tr tr tr... tr td td td td th th td td td td z = /html[1]/body[1]/table[2]/tr/td[1] 12

27 Framework hiking trails near Baltimore x w html head body

28 Framework hiking trails near Baltimore x w html head body Generation ( Z 8500) Z 13

29 Framework hiking trails near Baltimore x w html head body Generation ( Z 8500) Z Model /html[1]/body[1]/table[2]/tr/td[1] z 13

30 Framework hiking trails near Baltimore x w html head body Generation ( Z 8500) Z Model /html[1]/body[1]/table[2]/tr/td[1] z Execution [Avalon Super Loop, Patapsco Valley State Park,...] y 13

31 Features p θ (z x, w) exp{θ φ(x, w, z)} 14

32 Features p θ (z x, w) exp{θ φ(x, w, z)} Structural Features: captures context > 14

33 Features p θ (z x, w) exp{θ φ(x, w, z)} Denotation Features: captures content hiking trails near Baltimore Avalon Super Loop Patapsco Valley State Park Gunpowder Falls State Park Rachel Carson Conservation Park Union Mills Hike... > hiking trails near Baltimore Home About Baltimore Tour Pricing Contact Online Support... 14

34 Dataset airlines of italy natural causes of global warming lsu football coaches bf3 submachine guns badminton tournaments foods high in dha technical colleges in south carolina songs on glee season 5 singers who use auto tune san francisco radio stations 15

35 Dataset airlines of italy natural causes of global warming lsu football coaches 15

36 Dataset statistics 2773 examples 2269 unique queries 894 unique headwords long tail! 1483 unique web domains long tail! ( wrapper induction) 16

37 Results Baseline (Most frequent extraction predicates) Accuracy 5 17

38 Correct prediction Query: disney channel movies /html[1]/body/div[2]/div/div/div[3]/div[1]/div/div/div/div/b 18

39 Ranking error Query: doctors at emory /html/body/div[3]/div[4]/table/tbody/tr/td[2] Need better understanding of entities and categories! 19

40 Coverage error Query: hedge funds in new york /html/body/div[3]/div[3]/div[4]/.../table/tbody/tr/td[2]/a Need compositionality! 20

41 Outline Program induction Zero-shot entity extraction Question answering on new tables 21

42 Semantic parsing on tables [ACL 2015] Panupong (Ice) Pasupat 22

43 In what city did Piotr s last 1st place finish occur? 23

44 How long did it take this competitor to finish the 4x400 meter relay at Universiade in 2005? 23

45 Where was the competition held immediately before the one in Turkey? 23

46 How many times has this competitor placed 5th or better in competition? 23

47 Dataset WikiTableQuestions (2108 tables, questions) 3929 unique column headers = relations Breadth: Freebase can answer only 20% of the questions 24

48 Dataset WikiTableQuestions (2108 tables, questions) 3929 unique column headers = relations Breadth: Freebase can answer only 20% of the questions Tables in test data are not seen during training 24

49 Dataset WikiTableQuestions (2108 tables, questions) 3929 unique column headers = relations Breadth: Freebase can answer only 20% of the questions Tables in test data are not seen during training Depth: crowdsourced complex questions 13.5% 20.5% 24.5% 21.5% 20% table lookup + first, last,... + top, most,... other phenomena + next, after,... + count, total,... + arithmetic, comparison, alternative, multiple filters 24

50 Graph representation Add normalization / auxiliary edges (custom functions), push resolution to semantic parsing 25

51 Lambda DCS logical forms Entity Chicago 26

52 Lambda DCS logical forms Entity Chicago Join PlaceOfBirth.Chicago 26

53 Lambda DCS logical forms Entity Chicago Join PlaceOfBirth.Chicago Intersect Type.Person PlaceOfBirth.Chicago 26

54 Lambda DCS logical forms Entity Chicago Join PlaceOfBirth.Chicago Intersect Type.Person PlaceOfBirth.Chicago Aggregation count(type.person PlaceOfBirth.Chicago) 26

55 Lambda DCS logical forms Entity Chicago Join PlaceOfBirth.Chicago Intersect Type.Person PlaceOfBirth.Chicago Aggregation count(type.person PlaceOfBirth.Chicago) Superlative argmin(type.person PlaceOfBirth.Chicago, DateOfBirth) 26

56 Lambda DCS logical forms Entity Chicago Join PlaceOfBirth.Chicago Intersect Type.Person PlaceOfBirth.Chicago Aggregation count(type.person PlaceOfBirth.Chicago) Superlative argmin(type.person PlaceOfBirth.Chicago, DateOfBirth) Anaphora µx.type.person Children.Influence.x 26

57 Lambda DCS logical forms Entity Chicago Join PlaceOfBirth.Chicago Intersect Type.Person PlaceOfBirth.Chicago Aggregation count(type.person PlaceOfBirth.Chicago) Superlative argmin(type.person PlaceOfBirth.Chicago, DateOfBirth) Anaphora µx.type.person Children.Influence.x Variable argmax(type.person, R[λx.count(Parent.Parent.x)]) 26

58 Parsing/generation Greece held its last Summer Olympics in which year?? 2004 Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

59 Parsing/generation Greece held its last Summer Olympics in which year? R[Index].Country.Greece 2004 Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

60 Parsing/generation Greece held its last Summer Olympics in which year? R[Nations].Country.Greece 2004 Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

61 Parsing/generation Greece held its last Summer Olympics in which year? argmax(country.greece, Nations) 2004 Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

62 Parsing/generation Greece held its last Summer Olympics in which year? argmax(country.greece, Index) 2004 Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

63 Parsing/generation Greece held its last Summer Olympics in which year?... (hundreds of logical forms later) Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

64 Parsing/generation Greece held its last Summer Olympics in which year? R[Date].R[Year].argmax(Country.Greece, Index) 2004 Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

65 Parsing/generation Greece held its last Summer Olympics in which year? R[Date].R[Year].R[Prev].R[Prev].argmax(Type.Row, Index) 2004 Year City Country Nations 1896 Athens Greece Paris France St. Louis USA Athens Greece Beijing China London UK

66 x: Greece held its last Summer Olympics in which year? z: R[Date].R[Year].argmax(Country.Greece, Index) y: 2004 Feature vector φ(x, z) R F : 28

67 x: Greece held its last Summer Olympics in which year? z: R[Date].R[Year].argmax(Country.Greece, Index) y: 2004 Feature vector φ(x, z) R F : Feature template Feature Value (word, predicate) (Greece,Greece) 1 (held,greece) 1 (its,greece) 1 (Greece,argmax) 1 phrase=predicate (missing predicates) missing (denotation size) size=1 1 (phrase,denotation type) (which,date) 1 (which year,date) 1 (in which,date)

68 Results IR: Train classifer to pick answer directly from table. WQ: Use logical complexity of Freebase work IR WQ Full 29

69 Examples of correct predictions What train was developed after the erlangener erprobungstrager? According to the table, what is the last title that spicy horse produced? Who finished directly after the driver who finished in 1:28.745? Which album has the highest number of sales but doesn t have a designated artist? How many districts have a population density of at lest ? 30

70 Examples of errors Unhandled operations (19%): Was there more gold medals won than silver? Which movies were number 1 for at least two consecutive weeks? How many titles had the same author listed as the illustrator? 31

71 Examples of errors Unhandled operations (19%): Was there more gold medals won than silver? Which movies were number 1 for at least two consecutive weeks? How many titles had the same author listed as the illustrator? Table normalization: In what city did Piotr s last 1st place finish occur?...[bangkok, Thailand]... How long does the show defcon 3 last?...[2pm-3pm]... 31

72 Examples of errors Unhandled operations (19%): Was there more gold medals won than silver? Which movies were number 1 for at least two consecutive weeks? How many titles had the same author listed as the illustrator? Table normalization: In what city did Piotr s last 1st place finish occur?...[bangkok, Thailand]... How long does the show defcon 3 last?...[2pm-3pm]... Lexical mismatch: Mexican Mexico, airplane Model 31

73 Starting from zero examples... 32

74 Zero inputs! [Wang/Berant/Liang, ACL 2015] Traditional approach: Ad-hoc (incomplete) papers published in 2015 who wrote the origin of species? 33

75 Zero inputs! [Wang/Berant/Liang, ACL 2015] Traditional approach: Ad-hoc (incomplete), Annotation: understand + answer (expensive) papers published in 2015 Type.Article PublicationDate.2015 who wrote the origin of species? R[Author].OriginOfSpecies 33

76 Zero inputs! [Wang/Berant/Liang, ACL 2015] Traditional approach: Ad-hoc (incomplete), Annotation: understand + answer (expensive) papers published in 2015 Type.Article PublicationDate.2015 who wrote the origin of species? R[Author].OriginOfSpecies New approach: Grammar (complete) arg max(type.article, PublicationDate) article with the largest publication date arg max(type.person, R[λx.Count(Type.Article Author.x)]) person that is author of the most number of article 33

77 Zero inputs! [Wang/Berant/Liang, ACL 2015] Traditional approach: Ad-hoc (incomplete), Annotation: understand + answer (expensive) papers published in 2015 Type.Article PublicationDate.2015 who wrote the origin of species? R[Author].OriginOfSpecies New approach: Grammar (complete), Annotation: understand (cheap) arg max(type.article, PublicationDate) article with the largest publication date what is the newest published article? arg max(type.person, R[λx.Count(Type.Article Author.x)]) person that is author of the most number of article who has published the most articles? 33

78 On-the-job learning [NIPS 2015] 34

79 On-the-job learning [NIPS 2015] Maintain high accuracy Diminishing costs 35

80 Final remarks... 36

81 What is shared between tasks? Task A Task B 37

82 What is shared between tasks? Task A programs Task B 37

83 Task 1: input representation abstraction output Task 2: input representation abstraction output 38

84 Parting words Programs allow transferring abstractions (think Chess/Go, analogies) Task parametrized by natural language (and examples) More general problem: starting from zero examples 39

85 Code and data Funding Google Microsoft DARPA Sloan Foundation Thank you! 40

Learning Latent Programs for Question Answering

Learning Latent Programs for Question Answering Percy Liang Stanford University ICLR May 8, 2015 Goals Introduce a new challenge task for representation learning. 1 Goals Introduce a new challenge task