A Minimal Span-Based Neural Constituency Parser. Mitchell Stern, Jacob Andreas, Dan Klein UC Berkeley

Transcription

1 A Minimal pan-based Neural Constituency Parser Mitchell tern, Jacob Andreas, Dan Klein UC Berkeley

2 Parsing as pan Classification. he enjoys playing tennis

3 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

4 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

5 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

6 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

7 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

8 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

9 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

10 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

11 Parsing as pan Classification. he enjoys playing tennis he enjoys playing tennis

12 Parsing as pan Classification. X he enjoys Y playing tennis he enjoys playing tennis

13 Parsing as pan Classification. X he enjoys Y playing tennis he enjoys playing tennis

14 Grammar: Minimality in Parsing

15 Minimality in Parsing Grammar: (decided) (workers) (decided) [Collins (1999)]

16 Grammar: (decided) Minimality in Parsing (workers) (decided) ^ ^ [Collins (1999)] [Klein and Manning (2003)]

17 Grammar: (decided) Minimality in Parsing (workers) (decided) ^ ^ [Collins (1999)] [Klein and Manning (2003)] [Hall et al. (2014)]

18 Grammar: (decided) Minimality in Parsing (workers) (decided) ^ ^ [Collins (1999)] [Klein and Manning (2003)] [Hall et al. (2014)] [Vinyals et al. (2015)]

19 coring: Minimality in Parsing

20 Minimality in Parsing coring: score( ^ ^) [Klein and Manning (2003)]

21 Minimality in Parsing coring: score( ^ ^) score(i, k, j, ) [Klein and Manning (2003)] [Hall et al. (2014)]

22 Minimality in Parsing coring: score( ^ ^) score(i, k, j, ) score(i, j, ) and score action (i, k, j) [Klein and Manning (2003)] [Hall et al. (2014)] [Cross and Huang (2016)]

23 Minimality in Parsing coring: score( ^ ^) score(i, k, j, ) score(i, j, ) and score action (i, k, j) score(i, j, ) [Klein and Manning (2003)] [Hall et al. (2014)] [Cross and Huang (2016)] [This work]

24 Decoding: Minimality in Parsing

25 Minimality in Parsing Decoding: Chart-based Globally optimal, O(n 3 ) time complexity

26 Minimality in Parsing Decoding: Chart-based Globally optimal, O(n 3 ) time complexity Transition-based Greedy, O(n) or O(n 2 ) time complexity

27 Tree coring Function

28 Tree coring Function he enjoys playing tennis

29 Tree coring Function he enjoys playing tennis

30 Tree coring Function he enjoys playing tennis

31 Tree coring Function he enjoys playing tennis

32 Tree coring Function he enjoys playing tennis

33 Tree coring Function he enjoys playing tennis

34 Dynamic Program: Base Case

35 Dynamic Program: Base Case Pick best label

36 Dynamic Program: General Case

37 Dynamic Program: General Case Pick best label

38 Dynamic Program: General Case Pick best label Pick best split point

39 coring Function Implementation [Inspired by Cross and Huang (2016)]

40 coring Function Implementation he enjoys playing tennis. [Inspired by Cross and Huang (2016)]

41 coring Function Implementation Bidirectional LTM he enjoys playing tennis. [Inspired by Cross and Huang (2016)]

42 coring Function Implementation (f j - f i, b i - b j ) (f i, b i ) (f j, b j ) pan Difference Bidirectional LTM he enjoys playing tennis. [Inspired by Cross and Huang (2016)]

43 coring Function Implementation s s(i, j, X) (f j - f i, b i - b j ) (f i, b i ) (f j, b j ) Feedforward Network pan Difference Bidirectional LTM he enjoys playing tennis. [Inspired by Cross and Huang (2016)]

44 Training

45 Training Want for all

46 Training Want for all Require larger margin for higher loss:

47 Training Want for all Require larger margin for higher loss: Use hinge penalty function:

48 Training Use loss-augmented decoding during training:

49 Training Use loss-augmented decoding during training: Loss-augmented decoding for Hamming loss: Replace with

50 Initial Results Parser F1 core Hall et al. (2014) 89.2 Vinyals et al. (2015) 88.3 Cross and Huang (2016) 91.3 Dyer et al. (2016) 91.7 Liu and Zhang (2017) 91.7 Our Chart Parser 91.7

51 Top-Down Parsing. he enjoys playing tennis

52 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

53 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

54 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

55 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

56 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

57 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

58 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

59 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

60 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

61 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

62 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

63 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

64 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

65 Top-Down Parsing. he enjoys playing tennis he enjoys playing tennis.

66 Greedily Label and plit

67 Greedily Label and plit

68 Greedily Label and plit

69 Top-Down Training Margin constraint for each decision: score(gold) 1 + score(other)

70 Top-Down Training Margin constraint for each decision: score(gold) 1 + score(other) Train with exploration using a dynamic oracle [Goldberg and Nivre (2012), Cross and Huang (2016)]

71 Initial Results Parser F1 core Hall et al. (2014) 89.2 Vinyals et al. (2015) 88.3 Cross and Huang (2016) 91.3 Dyer et al. (2016) 91.7 Liu and Zhang (2016) 91.7 Our Chart Parser 91.7 Our Top-Down Parser 91.6

72 Extensions

73 Extensions Label scoring for unary chains: plit unary chains into top-middle-bottom

74 Extensions Label scoring for unary chains: plit unary chains into top-middle-bottom tructured label loss for unary chains: Hamming distance on labels (vs. 0-1 loss)

75 Extensions Label scoring for unary chains: plit unary chains into top-middle-bottom tructured label loss for unary chains: Hamming distance on labels (vs. 0-1 loss) plit-based (vs. span-based) scoring: Left-right, concatenate, deep biaffine [Cross and Huang (2016)] [Dozat and Manning (2016)]

76 Final Results Parser F1 core Hall et al. (2014) 89.2 Vinyals et al. (2015) 88.3 Cross and Huang (2016) 91.3 Dyer et al. (2016) 91.7 Liu and Zhang (2016) 91.7 Our Best Chart Parser 91.8 Our Best Top-Down Parser 91.8

77 Conclusion

78 Conclusion A minimal span-based parser can achieve state-ofthe-art results.

79 Conclusion A minimal span-based parser can achieve state-ofthe-art results. Little is lost going from global to greedy decoding.

80 Conclusion A minimal span-based parser can achieve state-ofthe-art results. Little is lost going from global to greedy decoding. Various extensions yield only minimal gains beyond the core system.

81 Thanks!