Commonsense Knowledge Acquisition and Applications

Transcription

1 Commonsense Knowledge Acquisition and Applications Towards Commonsense Enriched Machines Niket Tandon Ph.D. Supervisor: Gerhard Weikum Max Planck Institute for Informatics

2 2 Hard Rock property brown Hand, leg part of Person Climber is a Person Climbing a rock scene Adventurous Activity

3 3 Human- Machine Knowledge Gap Humans Machines Hard Rock property brown 1 Rock Hand, leg part of Person 2 Hands Climber is a Person 2 Legs Climbing a rock scene Adventurous Activity 1 Person

4 4 Human- Machine Knowledge Gap Humans Machines Hard Rock property brown 1 Rock Commonsense of objects Hand, leg Climber part of is a Person Person 2 Hands 2 Legs Commonsense of relationships Climbing a rock scene Adventurous Activity 1 Person Commonsense of interactions

5 How will the machines be smarter if we fill this knowledge gap Smarter Robots Get me a coffee (where?) Smarter Vision Better classifiers Monitor or TV? given mouse, keyboard Smarter IR Adventurous activities 5

6 Can we fill the human machine knowledge gap using existing Encyclopedic KBs like FreeBase? Encyclopedic Knowledge Facts about instances/events Facts about Instances: A. Honnold, married, Lisa Honnold Facts about classes/activities Common sense Knowledge Their events: A. Honnold, married on,

7 Facts about instances Facts about classes Encyclopedic Knowledge 1. EKB acquisition Unimodal 1. CKB acquisition Multimodal Commonsense Knowledge 2. EKB Curation Textual verification 2. CKB Curation Textual + Visual 3. EKB Completion Negative training assumptions hold 3. CKB Completion Negative training assumptions fail If (e i, r k, e j ) holds, then (e i, r k, e j!= e j ) is -ve A. Honnold, bornin, US A. Honnold, bornin, UK climber, at location, {mountain, university} 7

8 Facts about instances Facts about classes Encyclopedic Knowledge 1. EKB acquisition Unimodal 1. CKB acquisition Multimodal Commonsense Knowledge 2. EKB Curation Textual verification 2. CKB Curation Textual + Visual 3. EKB Completion 3. CKB Completion Negative training Negative training assumptions EKBs have holdseveral functional assumptions relations fail hence the assumption holds. If (e1 i, r k, e j ) holds, then 0.8 (e0.6 i, r k, e j!= e j ) is -ve 0.4 A. 0.2 Honnold, bornin, US 0 A. Honnold, bornin, UK EKB CKB Functional Non-functional 8

9 Commonsense knowledge acquisition is different and harder Humans hardly express the obvious: Scarce & Implicit Spread across multiple modalities: Multimodal Unusual reported more than usual: Reporting Bias Culture specific, Location specific: Contextual 9

10 KBs possessing commonsense knowledge KB Supervision Pros Cons Cyc ConceptNet Tandon et. al AAAI 11 Desiderata manually curated semiautomated bootstrapped using ConceptNet minimal supervision accuracy coverage accuracy coverage organized, high accuracy > 80%, high coverage >10M cost coverage less organized noise, less organized --- Need: automatically constructed, semantically organized Commonsense KB 10

11 Need: robust techniques to automatically construct semantically organized Commonsense KB

12 Three research questions: Investigate robust techniques to acquire: RQ 1. Commonsense of objects in the environment - fine-grained, semantically refined properties.

13 Three research questions: Investigate robust techniques to acquire: RQ 2. Commonsense of relationships between objects. - part whole relation, comparative relation

14 Three research questions: Investigate robust techniques to acquire: RQ 3. Commonsense of interactions between objects. - activities and their semantic attributes.

15 Three research questions: Investigate robust techniques to acquire:

16 Three research questions: Investigate robust techniques to acquire: RQ.1 RQ.2 RQ.3

17 Research question 1 RQ 1. Commonsense of objects in the environment - fine-grained, semantically refined properties. RQ.2 Previous work: lump together these properties do not distinguish the meanings of the words have low coverage RQ.3

18 Input Large text corpus containing e. g. summit is crisp 18 Output triples < w1 n s, r, w2 a s > summit n 2 hastemperature crisp a 3

19 Input Large text corpus containing e. g. summit is crisp Output triples < w1 n s, r, w2 a s > summit n 2 hastemperature crisp a 3 disambiguated n fine-grained relations: r R disambiguated a 1.) 2.) 3.) hasappearance hassound hastaste hastemperature hassound evokesemotion 1.) 2.) 3.) 19

20 Extract generic hasproperty triples over input <noun> verb [adv] <adj> <adj> <noun> e.g. summit is crisp.. Our approach summit, crisp mountain, cold chili, hot Disambiguate args and classify triple 20

21 Extract generic hasproperty triples over input Typically requires training data Disambiguate args and classify triple

22 Extract generic hasproperty triples over input < w1 n, w2 a > summit, crisp Suppose r = hastemperature Disambiguate args and classify triple <, r, w2 a s > < w1 n s, r, > < w1 n s, r, w2 a s > range r inference crisp a 3, hot a 1, cold a 1, icy a 2 domain r inference beach n 3, summit n 2, metal n 1, metal n 2 assertion r inference < summit 2 n, crisp 3 a > < beach 1 n, hot 1 a > 22

23 domain(r), range(r), assertion(r) inference Noisy, Surface form candidates for r Graph construction Graph inference 23

24 An instance of the problem: range(r) summit mountain dancer cold hot crisp

25 An instance of the problem: range(r) crisp a 1 crisp a 3 cold a 1 clearly defined cold and invigorating temperature low or inadequate temperature 25

26 An instance of the problem: range(r) sense #1 sense #2 sense #3 1/2 1/3 1/4 26

27 Label propagation for graph inference, given few seeds. - Label per node = in/not in range of hastemperature Similar nodes Similar labels But, limited training data summit, crisp mountain, cold salsa, hot 27

28 Label propagation for graph inference, given few seeds. - Label per node = in/not in range of hastemperature Similar nodes Similar labels But, limited training data 28

29 Label Propagation: Loss function (Talukdar et. al 2009) V U Seed label loss Similar node diff label loss Label prior loss (high degree nodes are noise) 29

30 Label propagation for graph inference, given few seeds. - Label per node = in/not in range of hastemperature Seed label loss Similar node diff label loss Label prior loss 30

31 Noisy, surface form candidates for r WebChild : Model recap Graph construction Graph inference Clean, disambiguat ed triples in r 31

32 Resulting KB WebChild: Large (~5Million), Semantically organized Accurate (0.82 sampled precision) Domain (hasshape) mountain-n 1 leaf-n 1... Range (hasshape) triangular-a 1 tapered-a 1... Assertions (hassshape) lens-n 1, spherical-a 2 palace-n 2, domed-a 1...

33 Summary of property commonsense WebChild: First commonsense KB with fine-grained relations and disambiguated arguments ; 4.6 million assertions including domain and range for 19 relations. Take away message: Transductive methods help overcome sparsity of commonsense in text.

34 Research question 3 RQ 3. Commonsense of interactions between objects. - activities and their semantic attributes. Previous work: largely discuss events, but activities only at small-scale do not organize the attributes of the activities do not distinguish the meanings of the attribute values

35 An Activity frame {Climb up a mountain Participants Location Time Visuals, Hike up a hill} climber, boy, rope camp, forest, sea shore day, holiday 35

36 Semantic organization of Activity frames Go up an elevation.... Previous activity Get to village.... {Climb up a mountain Participants Location Time Visuals Parent activity, Hike up a hill} climber, boy, rope camp, forest, sea shore day, holiday Next activity Reach at the top

37 Contain events but not activity knowledge May contain activities but no visuals and varying granularity of scene boundaries, transitions. 37

38 Contain events but not activity knowledge May contain activities but no visuals and varying granularity of scene boundaries, transitions. Hollywood narratives are good 38

39 Semantic parsing of scripts Graph construction 39

40 Semantic parsing of scripts Graph construction Input: Text in a scene taken from a semi-structured movie script e.g. : He began to shoot a video on the summit Output: Disambiguated, semantic roles e.g. the man : agent began to shoot : action a video : patient summit : location SRL systems are computationally expensive, domain specific 40

41 State of the art WSD customized for phrases the man began to shoot man.1 man.2 shoot.1 shoot.4 a video video.1 41

42 State of the art WSD customized for phrases VerbNet contains curated semantic roles for verbs the man began to shoot a video man.1 man.2 shoot.1 shoot.4 video.1 NP VP NP agent. patient. shoot.vn.1 animate animate agent. patient. shoot.vn.3 animate inanimate NP VP NP Selectional Selectional restriction restriction Can we use two different information sources to perform SRL given no training data? 42

43 State of the art WSD customized for phrases Jointly leverage Syntactic and semantic role semantics from VerbNet the man began to shoot a video WordNet class hierarchy man.1 man.2 shoot.1 shoot.4 video.1 NP VP NP agent. patient. shoot.vn.1 animate animate agent. patient. shoot.vn.3 animate inanimate NP VP NP WordNet VerbNet linkage Thing/ inanimate 43

44 State of the art WSD customized for phrases Jointly leverage Syntactic and semantic role semantics from VerbNet the man began to shoot a video man.1 man.2 shoot.1 shoot.4 video.1 WordNet class hierarchy Thing/ inanimate NP VP NP agent. patient. shoot.vn.1 animate animate agent. patient. shoot.vn.3 animate inanimate NP VP WordNet NP VerbNet linkage Binary decision variable 44

45 State of the art WSD customized for phrases Jointly leverage Syntactic and semantic role semantics from VerbNet the man began to shoot a video man.1 man.2 shoot.1 shoot.4 video.1 WordNet class hierarchy Thing/ inanimate NP VP NP agent. patient. shoot.vn.1 animate animate agent. patient. shoot.vn.3 animate inanimate NP VP WordNet NP VerbNet linkage WSD prior WN prior 45

46 State of the art WSD customized for phrases Jointly leverage Syntactic and semantic role semantics from VerbNet the man began to shoot a video man.1 man.2 shoot.1 shoot.4 video.1 WordNet class hierarchy Thing/ inanimate NP VP NP agent. patient. shoot.vn.1 animate animate agent. patient. shoot.vn.3 animate inanimate NP VP NP WN VN linkage Sense, VN syntactic match score 46

47 State of the art WSD customized for phrases Jointly leverage Syntactic and semantic role semantics from VerbNet the man began to shoot a video man.1 man.2 shoot.1 shoot.4 video.1 WordNet class hierarchy Thing/ inanimate NP VP NP agent. patient. shoot.vn.1 animate animate agent. patient. shoot.vn.3 animate inanimate NP VP NP WN VN linkage Sense, VN semantic match score 47

48 Joint WSD and SRL WSD prior WN prior Word, VN match score Selectional restriction score x ij = binary decision var. for word i, mapped to WN sense j One VN sense per verb WN, VN sense consistency Selectional restr. constraints binary decision 48

49 Semantic parsing of scripts Graph construction Joint WSD and SRL O/P the man began to shoot man.1 man.2 shoot.1 shoot.4 NP VP NP agent. patient. shoot.vn.1 animate animate Agent: man.1 Action: shoot.4 a video video.1 agent. patient. shoot.vn.3 animate inanimate NP VP NP Patient: video.1

50 Semantic parsing of scripts Graph construction Climb up a Participants mountain climber, rope Location summit, forest Time day

51 Semantic parsing of scripts Graph construction Go up an.... eleva tion Climb up a Participants Location mountain climber, rope summit, forest Hike up a Participants Location hill climber sea shore Reach.... top Time day Time holiday Construct a graph of activity frames with three edge types: TypeOf : T(a,b) Similar : S(a,b) Previous: P(a,b) 51

52 Similarity: S (climb up a mountain, hike up a hill) Activity Similarity + Attribute similarity Climb up a mountain Hike up a Hill Participants climber, rope Participants climber Location forest Location woods Time day Time holiday 52

53 TypeOf: T (climb up a mountain, go up an elevation) Activity hypernymy + Attribute hypernymy Climb up a mountain Go up an elevation Participants climber, rope Participants Person Location forest Location Exterior Time day Time day 53

54 Previous: P (reach the top, climb up a mountain) Climb up a mountain Reach the top Scene: Carrie and Big start out early to head to the village. They climb up the beautiful mountain which felt as if they were in a different world. After several hours they eventually reach the top. Allow gaps between activities within one scene. PMI style counting to suppress generic activities. 54

55 Semantic parsing of scripts Graph construction Goup an elevation.... Climb up a mountain Hike up a hill Participants Location climber, rope summit, forest similar Participants Location climber sea shore Time day Time holiday Reach top

56 Semantic parsing of scripts Graph construction 56

57 Resulting KB: Knowlywood Knowlywood Statistics Scenes 1,708,782 Activity synsets 505,788 Accuracy 0.85 ± 0.01 #Images from scenes 30,000 57

58 Summary of activity commonsense Knowlywood: First organized commonsense activity KB with activity attributes and disambiguated values containing nearly 1 million activities with visuals. Take away message: Jointly leveraging different annotated resources helps overcome sparsity of training data.

59 The overall KB: WebChild KB > 3M concepts, > 18M triples, >1000 relations

60 Conclusions and take home messages: Knowledge to make machines smarter can be acquired with robust techniques that jointly leverage global information Properties (WSDM 14) Comparatives, part-whole (AAAI 14, AAAI 16) Activities (WWW 15, CIKM 15) Research Question 1 Research Question 2 Research Question 3 WEBCHILD KB Applications (CVPR 15, ACL 15, ISWC 16..) 60

61 Conclusions and take home messages: Knowledge to make machines smarter can be acquired with robust techniques that jointly leverage global information Properties (WSDM 14) Comparatives, part-whole (AAAI 14, AAAI 16) Activities (WWW 15, CIKM 15) WEBCHILD KB Applications (CVPR 15, ACL 15, ISWC 16..) ML + NLP community RQ1 Range, domain, assertions of fine-grained relations limited training data can be overcome by jointly leveraging multiple cues RQ2 Computer Fine-grained Vision comparative, community part-whole relations commonsense helps computer vision vision helps commonsense acquisition RQ3 Activity frames with semantic attributes AI community semantically organized knowledge is a step towards filling human machine gap 61