Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum. Max Planck Institute for Informatics, Saarbrücken, Germany

Transcription

1 Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum Max Planck Institute for Informatics, Saarbrücken, Germany

2 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic 1

3 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! 1

4 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Text-only 1

5 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Visual objects: bicycle, bus, car Text-only Text + visual 1

6 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Visual objects: bicycle, bus, car Biking by the river Visual objects: train, piano Text-only Text + visual 1

7 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Visual objects: bicycle, bus, car Biking by the river Visual objects: train, piano Text-only Text + visual Text-only Text + visual 1

8 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Visual objects: bicycle, bus, car Text-only Text + visual Biking by the river Visual objects: train, piano Text-only Text + visual Riding for a cause. Visual objects: person, bicycle CSK: (riding bicycle, be, environment friendly) 1

9 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Visual objects: bicycle, bus, car Biking by the river Visual objects: train, piano Riding for a cause. Visual objects: person, bicycle Text-only Text + visual Text-only Text + visual Text/visual 1

10 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Visual objects: bicycle, bus, car Text-only Text + visual Biking by the river Visual objects: train, piano Text-only Text + visual Riding for a cause. Visual objects: person, bicycle CSK: (riding bicycle, be, environment friendly) Text/visual Text + visual + CSK 1

11 User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Wow! Double-decker buses still run! Visual objects: bicycle, bus, car Text-only Text + visual Biking by the river Visual objects: train, piano Text-only Text + visual Riding for a cause. Visual objects: person, bicycle CSK: (riding bicycle, be, environment friendly) Text/visual Text + visual + CSK Our contribution 1

12 CSK: Where do we get it from? CSK: How do we use it? CSK: How to combine noisy signals? CSK: Does it help? 2

13 CSK: WHERE DO WE GET IT FROM? Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood 3

14 CSK: WHERE DO WE GET IT FROM? Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood Our corpus: Wiki articles from domain tourism 3

15 CSK: WHERE DO WE GET IT FROM? Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood Our corpus: Wiki articles from domain tourism Pruned by Jaccard Similarity 3

16 CSK: WHERE DO WE GET IT FROM? Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood Our corpus: Wiki articles from domain tourism Pruned by Jaccard Similarity tourism be travel for recreation, leisure, family, business purposes people fall in love the bloody hell be you Domain-specific ReVerb triples ~22,000 CSK triples 3

17 CSK: HOW DO WE USE IT? Query string: travel with backpack CSK to expand query t1: (tourists, use, travel maps) t2: (tourists, carry, backpack) t3: (backpack, is a type of, bag) 4

18 CSK: HOW DO WE USE IT? Query string: travel with backpack CSK to expand query t1: (tourists, use, travel maps) t2: (tourists, carry, backpack) t3: (backpack, is a type of, bag) Document x with features Textual: A tourist reading a map by the road Visual: person, bag, bottle, bus Text-only systems Text + visual + CSK systems 4

19 CSK: HOW DO WE USE IT? Query string: travel with backpack CSK to expand query t1: (tourists, use, travel maps) t2: (tourists, carry, backpack) t3: (backpack, is a type of, bag) Document x with features Textual: A tourist reading a map by the road Visual: person, bag, bottle, bus CSK bridge vocabulary gap between query and document CSK establish relations between concepts CSK diminish noise from modalities ensemble effect Text-only systems Text + visual + CSK systems 4

20 CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Document x 5

21 CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x 5

22 CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x Visual features x v : backpack, person 5

23 CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x Visual features x v : backpack, bag, container person, casual agent, organism 5

24 CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x Visual features x v : backpack, bag, container person, casual agent, organism Query: group excursion Query expansion: (an excursion, be trip by, a group of people) (organized excursions, book through, a tour company) CSK features 5

25 CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Visual features x v : backpack, bag, container person, casual agent, organism Query: group excursion Query expansion: (an excursion, be trip by, a group of people) (organized excursions, book through, a tour company) 5

26 CSK: HOW TO COMBINE NOISY SIGNALS? Mixture LM: Commonsense-aware LM: Smoothed LM:, where Basic LM:, where 6

27 CSK: HOW TO COMBINE NOISY SIGNALS? Mixture LM: Commonsense-aware LM: Smoothed LM:, where Basic LM:, where 6

28 CSK: HOW TO COMBINE NOISY SIGNALS? Mixture LM: Commonsense-aware LM: CSK triple Smoothed LM:, where Basic LM:, where 6

29 CSK: HOW TO COMBINE NOISY SIGNALS? Mixture LM: Commonsense-aware LM: Smoothed LM: Probabilities based on word-wise overlaps, where Basic LM:, where 6

30 CSK: HOW TO COMBINE NOISY SIGNALS? Mixture LM: Commonsense-aware LM: Smoothed LM:, where Basic LM:, where Background corpus Co-occurring Flickr tags 6

31 CSK: HOW TO COMBINE NOISY SIGNALS? Mixture LM: Commonsense-aware LM: Smoothed LM:, where Basic LM:, where Textual and visual features 6

32 CSK: DOES IT HELP? Image Dataset o Flickr30k o MS COCO captioned dataset o Pascal Sentence Dataset o SBU captioned dataset 7

33 CSK: DOES IT HELP? Image Dataset o Flickr30k o MS COCO captioned dataset o Pascal Sentence Dataset o SBU captioned dataset Boat trip to see the mythical pink dolphins... this is John checking in with the office for that day. 7

34 CSK: DOES IT HELP? Image Dataset o Flickr30k o MS COCO captioned dataset o Pascal Sentence Dataset o SBU captioned dataset Boat trip to see the mythical pink dolphins... this is John checking in with the office for that day. A group of tourists is crossing a bridge that connects a walking path to a trail of nature. Many people cross a very tall footbridge with a tree-covered hill in the background. This shows a group of people walking over an arched red bridge. People cross a large bridge to get over the body of water. People walking over a white and red bridge over a pond. 7

35 CSK: DOES IT HELP? Image Dataset o Flickr30k o MS COCO captioned dataset o Pascal Sentence Dataset o SBU captioned dataset Boat trip to see the mythical pink dolphins... this is John checking in with the office for that day. social media post A group of tourists is crossing a bridge that connects a walking path to a trail of nature. Many people cross a very tall footbridge with a tree-covered hill in the background. This shows a group of people walking over an arched red bridge. People cross a large bridge to get over the body of water. People walking over a white and red bridge over a pond. blog post 7

36 CSK: DOES IT HELP? Image Dataset o Flickr30k o MS COCO captioned dataset o Pascal Sentence Dataset o SBU captioned dataset Boat trip to see the mythical pink dolphins... this is John checking in with the office for that day. social media post A group of tourists is crossing a bridge that connects a walking path to a trail of nature. Many people cross a very tall footbridge with a tree-covered hill in the background. This shows a group of people walking over an arched red bridge. People cross a large bridge to get over the body of water. People walking over a white and red bridge over a pond. blog post ~ 50,000 images with captions 7

37 CSK: DOES IT HELP? Baselines: Text-only and Text + Visual search approaches Evaluation metric: Average 10 8

38 CSK: DOES IT HELP? Baselines: Text-only and Text + Visual search approaches Evaluation metric: Average 10 Text-only Text + Visual Text + Visual + CSK (Know2Look) 47% 64% 85% 8

39 CSK: DOES IT HELP? Baselines: Text-only and Text + Visual search approaches Evaluation metric: Average 10 Text-only Text + Visual Text + Visual + CSK (Know2Look) 47% 64% 85% Examples queries: o Concrete ball park, bridge road, table home, bicycle road o Abstract diesel transport, housing town o Mixed old clock, backpack travel, boat tour 8

40 CSK: DOES IT HELP? Baselines: Text-only and Text + Visual search approaches Evaluation metric: Average 10 Text-only Text + Visual Text + Visual + CSK (Know2Look) 47% 64% 85% Examples queries: o Concrete ball park, bridge road, table home, bicycle road o Abstract diesel transport, housing town o Mixed old clock, backpack travel, boat tour Co-occurring Flickr tags 8

41 CSK: DOES IT HELP? Query: group excursion Text-only Text + Visual Text + Visual + CSK (Know2Look) xx j : A small excursion boat anchored on the beach at the resort in Mexico. " xv j : lunar excursion module, conveyance xx j : A group of people riding camels. y k : (an excursion, be trip by, a group of people) 9

42 CSK: DOES IT HELP? Query: group excursion Text-only Text + Visual Text + Visual + CSK (Know2Look) xx j : A small excursion boat anchored on the beach at the resort in Mexico. " xv j : lunar excursion module, conveyance xx j : A group of people riding camels. y k : (an excursion, be trip by, a group of people) 9

43 Noisy OpenIE triples capture commonsense knowledge Noisy textual cues + noisy visual object detection + noisy commonsense knowledge ensemble effect better results for multimodal document retrieval CSK act as bridge between text and vision 10

44 Noisy OpenIE triples capture commonsense knowledge Noisy textual cues + noisy visual object detection + noisy commonsense knowledge ensemble effect better results for multimodal document retrieval CSK act as bridge between text and vision Do word co-occurrences or word embeddings provide similar results? Does structured commonsense knowledge improve retrieval? 10

45 Noisy OpenIE triples capture commonsense knowledge Noisy textual cues + noisy visual object detection + noisy commonsense knowledge ensemble effect better results for multimodal document retrieval CSK act as bridge between text and vision Do word co-occurrences or word embeddings provide similar results? Does structured commonsense knowledge improve retrieval? Thank you 10