Parsers. Introduction to Computational Linguistics: Parsing Algorithms. Ambiguity refresher. CFG refresher. Example: search space for top-down parser

Transcription

1 Itroductio to Computatioal Liguistics: Parsig Algorithms haro Goldwater (based o slides by Mark teedma ad Philipp Koeh) July 5 Parsers A parser is a algorithm that computes a structure for a iut strig ge a grammar. Uderstadig differet parsig algorithms is importat for: Computer scietists: parsers used to compile programs, check html, etc. NLP researchers: efficiet parsers eeded for large-scale laguage tasks (e.g., used to create Google s ifoboxes ). Psycholiguists: what algorithm might be used by the huma setece processig mechaism? haro Goldwater Parsig July 5 Ambiguity refresher Parsers eed to hadle (rampat!) sytactic ambiguity i atural laguage. global ambiguity: multiple full parses are possible, e.g., PP attachmet: I saw the ma with the telescope local ambiguity: ambiguous partial structures, eed ot be cosistet with full parse. classic garde path seteces: the old ma the boats but also lots of ormal seteces: the dog bit the child Ambiguity ca be structural (differet possible phrasal costituets) or lexical (word with multiple PO tags), ofte both. haro Goldwater Parsig CFG refresher Parsig algorithms exist for may types of grammars, but we ll cosider just cotext-free grammars for ow. CFG refresher (or see J&M.-.5): Two types of grammar symbols: termials (t): words. No-termials (NT): phrasal categories like,, VP, PP. ometimes we distiguish pre-termials (PO tags), a type of NT. Rules must have the form NT β, where β is ay strig of NT s ad t s. A CFG i Chomsky Normal Form oly has rules of the form NT i NT j NT k or NT i t j haro Goldwater Parsig Parser properties haro Goldwater Parsig Example: search space for top-dow parser All parsers have two fudametal properties: tart with ode. Directioality: the sequece i which the structures are costructed. top-dow: start with root category (), choose expasios, build dow to words. bottom-up: build subtrees over words, build up to. Mixed strategies also possible (e.g., left corer parsers) Choose oe of may possible expasios. Each of which has childre with may possible expasios... VP aux VP earch strategy: the order i which the search space of possible aalyses is explored. etc haro Goldwater Parsig 4 earch strategies depth-first search: explore oe brach of the search space at a time, as far as possible. If this brach is a dead-ed, parser eeds to backtrack. breadth-first search: expad all possible braches i parallel (or simulated parallel). Requires storig may icomplete parses i memory at oce. best-first search: score each partial parse ad pursue the highest-scorig optios first. (Will get back to this whe discussig statistical parsig.) haro Goldwater Parsig 5 Recurse Descet Parsig A recurse descet parser treats a grammar as a specificatio of how to break dow a top-level goal (fid ) ito subgoals (fid VP). It is a top-dow, depth-first parser: blidly expad otermials util reachig a termial (word). If termial matches ext iut word, cotiue; else, backtrack. haro Goldwater Parsig 6 haro Goldwater Parsig 7

2 Backtrack poits Need to keep track of backtrack poits, to retur to if we backtrack. Each backtrack poit stores: A partial parse tree (what was completed whe we made a choice) The rules we have t tried yet The iut words we have t matched yet To esure depth-first search, backtrack poits are stored i a stack: last i, first out. We start with RD Parsig: iitializatio The rules of our cotext-free grammar, e.g., VP VP V NN bit V bit DT NN DT the NN dog V dog Curret partial parse (also curret subgoal): the ode. A ordered list of subgoals, iitially cotaiig just. A empty stack of backtrack poits. The iut sequece (e.g., the dog bit) haro Goldwater Parsig 8 RD Parsig: iterate steps If first subgoal i list is a o-termial A: Pick a rule from the grammar to expad it (e.g., A B C) Replace A i subgoal list with B C If first subgoal i list is a termial w: If iut is empty, backtrack. If ext iut word is differet from w, backtrack. If ext iut word is w, match! i.e., cosume iut word w ad subgoal w ad move to ext subgoal. haro Goldwater Parsig 9 RD Parsig: iterate steps If first subgoal i list is a o-termial A: Pick a rule from the grammar to expad it (e.g., A B C) Replace A i subgoal list with B C If first subgoal i list is a termial w: If iut is empty, backtrack. If ext iut word is differet from w, backtrack. If ext iut word is w, match! i.e., cosume iut word w ad subgoal w ad move to ext subgoal. If stack is empty, we lose! No parse is possible. If o more subgoals, we wi! We foud a parse. haro Goldwater Parsig Recurse descet example haro Goldwater Parsig Parsers vs Recogizers Grammar ad setece from slide 9. Operatios: Expad (E) Match (M) Backtrack to step (B) tep Op. ubgoals Iut the dog bit E VP the dog bit E DT NN VP the dog bit E the NN VP the dog bit 4 M NN VP dog bit 5 E bit VP dog bit 6 B4 NN VP dog bit 7 E dog VP dog bit 8 M VP bit 9 E V bit E bit bit M The above sketch is actually a recogizer: it tells us whether the setece has a valid parse, but ot what the parse is. Would eed to add more ails to keep track of parse structure as it is built. haro Goldwater Parsig hift-reduce Parsig earch strategy ad directioality are orthogoal properties. hift-reduce parsig is depth-first (like RD) but bottom-up (ulike RD). Basic shift-reduce recogizer repeatedly: hifts iut symbols oto a stack. Wheever possible, reduces oe or more items from top of stack that match RH of rule, replacig with LH of rule. Like RD parser, eeds to maitai backtrack poits. haro Goldwater Parsig ame example grammar ad setece. Operatios: hift () Reduce (R) Backtrack to step (B) hift-reduce example tep Op. tack Iut the dog bit the dog bit R DT dog bit DT dog bit 4 R DT V bit 5 DT V bit 6 R DT V V 7 B5 DT V bit 8 R DT V NN 9 B DT dog bit R DT NN bit R bit R bit bit... haro Goldwater Parsig 4 haro Goldwater Parsig 5

3 Depth-first parsig i practice Depth-first parsers are very efficiet for uambiguous structures. Widely used to parse/compile programmig laguages Laguage/grammar is specially costructed to be uambiguous (sometimes with fiite lookahead). Depth-first parsig i practice Depth-first parsers are very efficiet for uambiguous structures. Widely used to parse/compile programmig laguages Laguage/grammar is specially costructed to be uambiguous (sometimes with fiite lookahead). But ca be massely iefficiet (expoetial i setece legth) if faced with local ambiguity. Blid backtrackig may require re-buildig the same structure over ad over. o, much less commo for atural laguage parsig (though some work o best-first probabilistic shift-reduce parsig: uses lookahead to help predict which expasios to make). haro Goldwater Parsig 6 Breadth-first search usig dyamic programmig With a CFG, a parser should be able to avoid re-aalyzig sub-strigs because the aalysis of ay sub-strig is idepe of the rest of the parse. The dog saw a ma i the park PP haro Goldwater Parsig 7 Breadth-first search usig dyamic programmig With a CFG, a parser should be able to avoid re-aalyzig sub-strigs because the aalysis of ay sub-strig is idepe of the rest of the parse. The dog saw a ma i the park PP To exploit this fact, chart parsig algorithms use dyamic programmig to store ad reuse sub-parses. This permits explorig multiple potetial parses at oce: strategy. a breadth-first haro Goldwater Parsig 8 Parsig as dyamic programmig As i HMM algorithms, dyamic programmig fills a table of solutios to subproblems (memoizatio), the composes these to fid the full solutio. For parsig, subproblems are aalyses of substrigs, memoized i chart (aka well-formed substrig table, WFT). Chart etries are idexed by start ad ed positios i the setece, ad correspod to: either a complete costituet (sub-tree) spaig those positios (if workig bottom-up), or a hypothesis about what complete costituet might be foud (if workig top-dow). haro Goldwater Parsig 9 Depictig a WFT/Chart Chart ca be depicted as either a matrix or a graph. I either case, we assume idices betwee each word i the setece: ee with a telescope 4 i 5 had 6 If usig a matrix, cell [i, j] holds iformatio about the word spa from positio i to positio j: The root ode of ay costituet(s) spaig those words Poiters to its sub-costituets (Depedig o parsig method,) predictios about what costituets might follow the substrig. haro Goldwater Parsig xample, partway through parsig. Here, oly showig root odes of each costituet. Lower left of chart ever used; ofte oly upper right is show. Depictig a WFT as a Matrix 4 5 V Prep Det ee with a telescope 4 i 5 had 6 PP N haro Goldwater Parsig Depictig a WFT as a Graph Here, each setece positio idex is a ode or vertex. edges (arcs) represet spas, labelled with the same iformatio that goes i a cell i the matrix represetatio. Prep PP Det with a telescope 4 N haro Goldwater Parsig haro Goldwater Parsig

4 Algorithms for Chart Parsig May differet chart parsig algorithms, icludig the CKY algorithm, which memoizes oly complete costituets various algorithms that also memoize predictios/partial costituets ofte usig mixed bottom-up ad top-dow approaches, e.g., the Earley algorithm described i J&M, ad left-corer parsig. CKY Algorithm CKY (Cocke, Kasami, Youger) is a bottom-up, breadth-first parsig algorithm. Origial (simplest) versio assumes grammar i Chomsky Normal Form. Add costituet A i cell (i, j) if: there is a rule A B, ad B is i cell (i, j), or there is a rule A B C, ad B is i cell (i,k) ad C is i cell (k, j). haro Goldwater Parsig 4 CKY Algorithm CKY (Cocke, Kasami, Youger) is a bottom-up, breadth-first parsig algorithm. Origial (simplest) versio assumes grammar i Chomsky Normal Form. Add costituet A i cell (i, j) if: there is a rule A B, ad B is i cell (i, j), or there is a rule A B C, ad B is i cell (i,k) ad C is i cell (k, j). Fills chart i order: oly looks for rules that use a costituet from i to j after fidig all costituets edig at i. o, guarateed to fid all possible parses. haro Goldwater Parsig 5 CKY Pseudocode Assume iut setece with idices to, ad chart c. for le = to : #umber of words i costituet for i = to -le: #start positio j = i+le #ed positio #process uary rules if A->B ad c[i,j] has B, add A to c[i,j] for k = i+ to j- #mid positio #process biary rules if A->B C ad c[i,k] has B ad c[k,j] has C, add A to c[i,j] This algorithm performs recogitio i time O( ). haro Goldwater Parsig 6 Grammatical rules VP Det Nom Nom Nom N Rel Nom N VP TV VP IV PP VP IV PP Prep Rel Relpro VP CKY example Lexical rules Det a the (ermier) N fish frogs soup (ou) Prep i for (prepositio) TV saw ate (trasite verb) IV fish swim (itrasite verb) Relpro that (relate proou) Nom: ial (the part of the after the ermier, if ay). Rel: subject relate clause, as i the frogs that ate fish. haro Goldwater Parsig 7 Visualizig the Chart 4 haro Goldwater Parsig 8 Visualizig the Chart (,) 4 haro Goldwater Parsig 9 Visualizig the Chart (,) 4 Uary brachig rules: the Uary brachig rules: N frogs, Nom N, Nom haro Goldwater Parsig haro Goldwater Parsig

5 Visualizig the Chart (,) 4 Visualizig the Chart (,4) 4 Uary brachig rules: ate Uary brachig rules: N fish, Nom N, Nom, fish, haro Goldwater Parsig Visualizig the Chart (,) 4 haro Goldwater Parsig Visualizig the Chart (,) 4 Biary brachig rule: Det Nom (,) & (,) (,) (,) & (,) haro Goldwater Parsig 4 Visualizig the Chart (,4) 4 haro Goldwater Parsig 5 Visualizig the Chart (,) 4 Biary brachig rule: VP TV (,) & (,4) (,4) (,) & (,) (,) & (,) haro Goldwater Parsig 6 Visualizig the Chart (,4) 4 haro Goldwater Parsig 7 Visualizig the Chart (,4) 4 s s s Biary rule: VP (,) & (,4) (,4) (,) & (,4) (,) & (,4) Biary rule: VP (,) & (,4) (,4) (,) & (,4) haro Goldwater Parsig 8 haro Goldwater Parsig 9

6 A ote about CKY orderig Notice that to fill cell (i, j), we use a cell from row i ad a cell from colum j. o, we must fill i all cells dow ad left of (i, j) before fillig (i, j). Here, we filled i all short etries, the loger oes, but other orders ca work (e.g., J&M fill i all spas edig at j, the icremet j.) From CKY Recogizer to CKY Parser As just specified, CKY oly recogizes, but ca t retur the parse. e.g., we do t kow from cell how it was costructed. Easy to fix: wheever a costituet is foud for cell (i, j), store the idices of the sub-costituets that formed it. ca mea storig multiple copies of A with differet idices. ometimes called a packed parse forest: represets a possibly expoetial umber of trees i a compact way. haro Goldwater Parsig 4 CKY i practice Avoids re-computig substructures, so much more efficiet tha depth-first parsers (i worst case). till may compute a lot of uecessary partial parses. imple versio requires covertig the grammar to CNF (may cause blowup). haro Goldwater Parsig 4 Itroductio to Computatioal Liguistics: Parsig ad Huma etece Processig haro Goldwater July 5 Various other chart parsig methods avoid these issues by combiig top-dow ad bottom-up approaches (see J&M). We also have t said aythig about how to choose betwee differet parses whe there s global ambiguity. haro Goldwater Parsig 4 Recap: Parsig We have see several differet parsig algorithms: Recurse descet parsig: top-dow, depth-first. hift-reduce parsig: bottom-up, depth-first. CKY parsig: bottom-up, breadth-first. Do ay of these seem plausbile as a model of huma setece processig? haro Goldwater Huma Parsig July 5 Properties of huma parsig mechaism Fast: real-time. Recogizes global ambiguity: at least to some extet. Icremetal: words (ad meaig) are itegrated ito structure immediately. Ca be led astray: by local ambiguity (garde path seteces). But mostly is t: may local ambiguities do t cause problems. haro Goldwater Huma Parsig Global ambiguity ome examples are clearly (eve humorously) ambiguous: I saw the ma with a telescope. he sat o the chair covered i dust. Milk drikers are turig to powder. But most ambiguity is t eve oticed! I saw the ma with a hat. he stood o the stoop covered i tears. Breast feeders are turig to a ew eriched formula. haro Goldwater Huma Parsig ame goes for local ambiguity: The old ma the boats We paited the wall with cracks Fat people eat accumulates versus Local ambiguity The dog bit the cat The gree is used for playig soccer We stopped short of goig haro Goldwater Huma Parsig haro Goldwater Huma Parsig 4

7 Properties of our parsig algorithms Fast? We argued that RD ad R are iefficiet, but could perhaps be improved by good heuristics for choosig ext rules. Ayway, hard to evaluate what couts as fast. Recogize global ambiguity? CKY builds all parses, so defiitely yes. RD/R ca retur multiple parses if ru past the first oe. But otice a distictio... erial versus parallel parsig Depth-first parsers are iheretly serial: oe structure processed at a time. o, recogizig ambiguity implies backtrackig/re-parsig, ad oe structure will always be recogized first. Breadth-first parsers are idealized as parallel: multiple structures processed simultaeously. If truly parallel, ambiguous structures iified simultaeously. haro Goldwater Huma Parsig 5 Huma parsig: serial or parallel? O the face of it, full parallel parsig seems implausible: Fids/otices all the global ambiguities. Does t get stuck i garde paths. erial parsig provides a possible explaatio for garde paths (backtrackig). But there are parallel optios too: limited parallelism: pursue a fixed (small) umber of structures at oce, may still require occasioal backtrackig. raked parallelism: (possibly i combiatio with above). Possible structures raked by preferece; garde path if low-raked structure turs out to be correct. haro Goldwater Huma Parsig 6 What about icremetality? First, what does it mea to be icremetal? Each word is itegrated ito the parse as soo as it is see/heard. Problems with curret parse are ected immediately; also possible to make predictios about upcomig words. Evidece: eye-trackig of sematic iterpretatio, garde paths. haro Goldwater Huma Parsig 7 Is CKY parsig icremetal? The chart-fillig order we used (short spas first) clearly is t. haro Goldwater Huma Parsig 8 There are still 4 discoected structures before the rule VP DV applies, reducig the umber to (after which, ). The girl gave the dog a boe What about J&M s orderig (fill all cells edig at j, the j + )? c dv c c Cosider processig The girl gave the dog a boe: VP Det CN VP TV VP DV Det the a CN girl dog boe TV bit DV gave The girl gave the dog a boe c dv c c haro Goldwater Huma Parsig 9 Aother problematic example Cosider the garde path setece the old ma the boats. haro Goldwater Huma Parsig Aother problematic example Grammar for processig the old ma the boats: Assume a serial bottom-up parser (or limited parallel key is that the correct structure is ot cosidered iitially). At what poit (ituitely) does a huma realize the iitial aalysis is icorrect? At what poit does the parser realize this? VP Det CN Det Adj CN VP TV VP DV Det the a Adj old CN ma boats old TV ma like DV gave haro Goldwater Huma Parsig haro Goldwater Huma Parsig

8 Garde path is too late! The bottom-up parser does t realize its mistake util it reaches the ed of the setece, ad caot create a full parse: The old ma the boats ummary so far RD parsig caot model humas because of problems with (eg.) left recursio. R parsig caot model humas because it does t recogize garde paths immediately. adj c c CKY parsig caot model humas because it is too parallel, or (if limited), also does t recogize garde paths immediately. o, where does that leave us? But humas recogize a problem at the secod the: they have a expectatio about what should come ext, ad it is violated. haro Goldwater Huma Parsig Left Corer Parsig Left corer parsig is more cogitely plausible: each word is immediately itegrated ito a sigle evolvig structure which makes predictios about what will come ext. Mixed directioality: costraied by iut (like bottom-up) but also makig predictios (like top-dow). Chart cotais acte edges: icomplete costituets represetig predictios. Ex: /CN is a icomplete costituet that will become a complete if a CN is see ext (cf. categorial grammar). haro Goldwater Huma Parsig 4 Rules of Left Corer Parsig. Projectio: For a completed edge Y ad a grammar rule X Y Z, add a acte edge X/Z, where Y ad X/Z spa the same part of the strig.. Completio: For a acte edge X/Y ad a completed edge Y that are adjacet, add a completed edge X that spas the width of both.. Compositio: For two adjacet acte edges X/Y ad Y/Z, add a acte edge X/Z that spas the width of both. Rule is ot ecessary for LC parsig, but is ecessary for a fully icremetal versio (i.e., to esure a sigle coected structure). haro Goldwater Huma Parsig 5 The kittes bite the dog haro Goldwater Huma Parsig 6 Rules of Left Corer Parsig Example of a left corer chart: /c s/ c / /c c If dealig with o-biary grammar:. Projectio: For a completed edge Y ad a grammar rule X Y α, add a acte edge X/α, where Y ad X/α spa the same part of the strig.. Partial Completio: For a acte edge X/Y α ad a completed edge Y that are adjacet, add a acte edge X/α that spas the width of both. s/. Completio: For a acte edge X/Y ad a completed edge Y that are adjacet, add a completed edge X that spas the width of both. s/c 4. Compositio: For two adjacet acte edges X/Y ad Y/Z, add a acte edge X/Z that spas the width of both. s haro Goldwater Huma Parsig 7 Example LC parse for garde path setece To try o your ow with the grammar provided earlier: the old ma the boats Cofirm that the parser realizes a problem where it should! haro Goldwater Huma Parsig 8 ummary Left-corer parsig achieves full icremetality by usig chart etries to represet partial/predicte sytactic structure. Looks promisig for modellig ambiguity resolutio ad garde paths. But still have t explaied why some parses are preferred or some locally ambiguous seteces (but ot others) cause garde paths. Other ope research issues: Developig fully icremetal parsers for wide rage of grammar formalisms (some easier tha others). How/whe does sematics fit i? haro Goldwater Huma Parsig 9 haro Goldwater Huma Parsig