Summarization in Sequential Data with Applications to Procedure Learning

Size: px

Start display at page:

Download "Summarization in Sequential Data with Applications to Procedure Learning"

Gregory Gilmore
5 years ago
Views:

1 Summarization in Sequential Data with Applications to Procedure Learning Ehsan Elhamifar College of Computer and Information Science Northeastern University

2 Data deluge > 400hrs videos / minute > 300M photos / day 24/7/365

3 Big data opportunities Capturing modes and patterns unseen in small data - Increase learning and inference performance From Nearest Neighbors It takes a large amount of data for our method to succeed. We saw dramatic improvement when moving from ten thousand to two million images. [Hayes, Efros, SIGGRAPH 07] As we increase the size of the dataset from 10^5 to the 10^8 images, the quality of the retrieved set increases dramatically. [Torralba, Fergus, Freeman, PAMI 08]

Big data opportunities Capturing modes and patterns unseen in small data - Increase learning and inference performance To Deep Neural Networks Classification Error (%) 30 25 20 15 10

4 Big data opportunities Capturing modes and patterns unseen in small data - Increase learning and inference performance To Deep Neural Networks Classification Error (%) Dataset Size From 100 to 100,000 training samples, classification error drops from 25% to less than 5%. [Srivastava-Hinton-Krizhevsky-Sutskever-Salakhutdinov 14]

memory resources - We have more data than

5 Big data challenges Capturing modes and patterns unseen in small data - Increase learning and inference performance: from NNs to DNNs Limiting computational and memory resources - We have more data than capacities of our resources Economist, Feb 2010

6 Subset selection Find small data capturing statistical properties of large data Subset Selection

7 Subset selection components How to characterize informativeness of items? How to recover the most informative items? max S f(s) 1. What objective functions? 2. How to optimize?

8 This Talk max S f(s) Subset selection is NP-hard, hence, approximate algorithms - Better summarization: closer to global optimum Focus on one class of objective functions: Facility Location Develop efficient sparse optimization Show it can achieve global optimum (under certain condition)

This Talk Sequential data (video, text, signals, ): large part of Big Data (CNN) -- A Japanese rocket roared into orbit early Friday (Thursday afternoon ET)

The Global Precipitation Measurement (GPM) satellite is the first of five earth science launches NASA has planned for 2014.

said Gail Skofronick Jackson, GPM's deputy project scientist.

m. Thursday ET). In a little over a half hour, it had reached orbit, deployed its solar panels and began beaming signals back to its controllers, NASA said.

9 This Talk Sequential data (video, text, signals, ): large part of Big Data (CNN) -- A Japanese rocket roared into orbit early Friday (Thursday afternoon ET) carrying what NASA calls its most precise instrument yet for measuring rain and snowfall. The Global Precipitation Measurement (GPM) satellite is the first of five earth science launches NASA has planned for The 4-ton spacecraft is the most sophisticated platform yet for measuring rainfall, capable of recording amounts as small as a hundredth of an inch an hour, said Gail Skofronick Jackson, GPM's deputy project scientist. The $900 million satellite is a joint project with the Japanese space agency JAXA, and it lifted off from Tanegashima Space Center at 3:37 a.m. Friday (1:37 p.m. Thursday ET). In a little over a half hour, it had reached orbit, deployed its solar panels and began beaming signals back to its controllers, NASA said. Also, once fully activated, GPM will use both radar and microwave instruments to detect falling snow for the first time. It will also combine data from other satellites with its own readings, beaming back a snapshot of worldwide precipitation every three hours, Sequential data have structured dependencies - Instructional videos: ordering of key steps Key steps

10 This Talk Sequential data (video, text, signals, ): large part of Big Data Sequential data have structured dependencies - Existing SS: data is a bag of randomly permutable items Generalize facility location to handle seq summarization x 1 x 7 x 4 x 8 x 3 x 5 x 2 max S f(s) x 9 x 6 Develop new objective func Develop efficient opt algs x 9 x 6 x 4 x 2

11 Subset selection from dissimilarities Given dissimilarities d : X Y! R 0, source target select a small subset of X that well represent Y w.r.t. d(, ) x 1 x 2 d 11 y 1 y 2 x 1 x 2 y 1 y 2 x M d MN y N x M y N d(x i, y j )=d ij X Y d(, ) : how well x i represents - = data, = data = Euclidean/geodesic distance X Y d(, ) - = models, = data = representation/coding error y j

12 Subset selection from dissimilarities Given dissimilarities d : X Y! R 0, source target select a small subset of X that well represent Y w.r.t. d(, ) x 1 x 2 d 11 y 1 y 2 x 1 x 2 y 1 y 2 x M d MN y N x M y N d(x i, y j )=d ij : how well x i represents y j Not necessarily metric: asymmetric, violate triangle ineq

13 Dissimilarity-based Sparse Subset Selection Let r j 2{1,...,M} be index of representative of Approach: define & maximize potential function (r 1,...,r N ), enc (r 1,...,r N ) card (r 1,...,r N ) y j x i x rj y j enc(r 1,...,r N ), NY j=1 e d r j,j Prefer lower encoding card(r 1,...,r N ), e {r 1,...,r N } Prefer small # representatives Max over {r 1,...,r N } 2 {1,...,M} N : combinatorial E. Elhamifar, G. Sapiro, S. Sastry, TPAMI, 2016

14 DS3 formulation Develop a sparse binary optimization d ij z ij Define variables z i,j, I(r j = i) 2 {0, 1} x i MY NY = exp( d i,j z i,j ) exp I(k[z i,1...z i,n ]k p ) i=1 j=1 i=1 y j Minimizing log : Nonconvex min {z i,j } i=1 NX j=1 s. t. z i,j 2 {0, 1}, d i,j z i,j + z i,j =1, i=1 i=1 I(k[z i,1... z i,n ]k p ) 8 i, j ~ Facility location E. Elhamifar, G. Sapiro, S. Sastry, TPAMI, 2016

15 DS3 formulation Develop a sparse binary optimization d ij z ij Define variables z i,j, I(r j = i) 2 {0, 1} x i MY NY = exp( d i,j z i,j ) exp I(k[z i,1...z i,n ]k p ) i=1 j=1 i=1 y j Minimizing log : min {z i,j } NX d i,j z i,j + zi,1 z i,n i=1 j=1 i=1 Convex p 2{2, 1} s. t. z i,j 0, z i,j =1, i=1 DS3 8 i, j p E. Elhamifar, G. Sapiro, S. Sastry, TPAMI, 2016

16 Toy example Source = target = {data points} 0 data points data points 1 representatives 8 2 > 1 representatives 3 > 2 data points representatives Z matrix for λ = 0.01 λ max, 1 4 Z matrix for λ = 0.1 λ max, 1 Z matrix for λ = λ max, min {z i,j } i=1 NX d i,j z i,j + j=1 i=1 zi,1 z i,n p s. t. z i,j 0, z i,j =1, i=1 8 i, j

17 Theoretical analysis When is convex relaxation exact? Theorem: Let r j be representative of j, via non-convex optimization. The convex relaxation is exact if 1) + d rj,j <d j0,j N rj Clusters sufficiently separated j r j j 0 2) d j0,j apple Nrj + d rj,j j 0 r j j Sufficiently small radius E. Elhamifar, M. De Paolis Kaluza `17

18 Video summarization Comparison with greedy - Solve FL via Greedy and sparse coding (DS3) TVSum SumMe Cost function value Greedy DS3 Greedy DS3

19 Recognition via representatives Select fraction of training samples via DS Suburb Suburb Coast Coast Forest Forest Highway Highway Insidecity Insidecity Mountain Mountain Opencountry Opencountry Street Street Tallbuilding Tallbuilding Office Office Bedroom Industrial Kitchen Livingroom Store Store Livingroom Kitchen Industrial Bedroom Office Tallbuilding Street Opencountry Mountain Insidecity Highway Forest Coast Suburb Bedroom Industrial Kitchen Livingroom Store Store Livingroom Kitchen Industrial Bedroom Office Tallbuilding Street Opencountry Mountain Insidecity Highway Forest Coast Suburb DS3 selecting 35% samples Trained via all samples E. Elhamifar, G. Sapiro, S. Sastry, TPAMI, 2016

20 Recognition via representatives Select fraction of training samples via DS Suburb Suburb Coast Coast Forest Forest Highway Highway Insidecity Insidecity Mountain Mountain Opencountry Opencountry Street Street Tallbuilding Tallbuilding Office Office Bedroom Industrial Kitchen Livingroom Store Store Livingroom Kitchen Industrial Bedroom Office Tallbuilding Street Opencountry Mountain Insidecity Highway Forest Coast Suburb Bedroom Industrial Kitchen Livingroom Store Store Livingroom Kitchen Industrial Bedroom Office Tallbuilding Street Opencountry Mountain Insidecity Highway Forest Coast Suburb DS3 selecting 35% samples Trained via all samples E. Elhamifar, G. Sapiro, S. Sastry, TPAMI, 2016

21 Recognition via representatives Select fraction of training samples via DS3 Suburb Suburb Coast Forest Highway Coast Forest Highway Insidecity Insidecity Mountain Mountain Opencountry Opencountry Representatives improve computational time and Street Street Tallbuilding can improve Tallbuilding performance Office Office Bedroom Industrial Kitchen Livingroom Store Store Livingroom Kitchen Industrial Bedroom Office Tallbuilding Street Opencountry Mountain Insidecity Highway Forest Coast Suburb Bedroom Industrial Kitchen Livingroom Store Suburb Store Livingroom Kitchen Industrial Bedroom Office Tallbuilding Street Opencountry Mountain Insidecity Highway Forest Coast E. Elhamifar, G. Sapiro, S. Sastry, TPAMI, 2016

22 Sequential subset selection Incorporate dynamics of sequential data into SS - Source set: with transition dynamics - Target set: sequential data y 1 x 1 y T x M Goal: find (r 1,...,r T ) with high transition probability that well encodes the data and has small cardinality y 1 y 2 y 3 y 1 y 2 y 3 x 1 x 2 x 3 x 1 x 2 x 3 (r 1,r 2,r 3 )=(1, 1, 2) (r 1,r 2,r 3 )=(1, 1, 3) E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

23 Sequential facility location Let r t be index of representative of y t y 1 y T x 1 x M Maximize global potential function Prefer a sequence of compatible reps max (r 1,...,r T ) enc(r 1,...,r N ) card (r 1,...,r N ) dyn (r 1,...,r N ) - Dynamic potential: n-th order Markov model dyn(r 1,...,r N ), Y t p t (x rt x rt 1,...,x rt n )! E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

24 Sequential facility location SeqFL: Solve optimization on assignment variables {z i,t } max {z i,t } TX t=1 + ( i=1 z i,t d i,t M X i=1 z i,1 log p 1 (x i )+ i=1 s. t. z i,t 2 {0, 1}, k z i,1 z i,t k1 TX t=2 i,i 0 =1 z i,t =1, 8 i, t. i=1 z i,t 1 z i 0,t log p(x i 0 x i )) x 1 y 1 z i,t 1 z i 0,t =1 y t 1 y t x M y T p(x i 0 x i ) should be large! x i x i 0 E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

25 Sequential facility location SeqFL: Solve optimization on assignment variables max {z i,t } TX t=1 + ( i=1 Non-convex: does not make sense (here) to relax! - Solve via z i,t d i,t M X i=1 z i,1 log p 1 (x i )+ i=1 s. t. z i,t 2 {0, 1}, k z i,1 z i,t k1 TX t=2 i,i 0 =1 z i,t =1, 8 i, t. i=1 z i,t 1 z i 0,t log p(x i 0 x i )) Max-Sum Message Passing {z i,t } x 1 x M y 1 y T E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

26 SeqFL via message passing C 1 11 C 2 12 C 3 13 R 1 C t it R i z 11 D 11;12 z 12 D 12;13 z 13 it it it 21 z 21. D 21;12 22 z 22. D 22;13 23 z 23 R 2 D 1(t 1);it. 0 it,1(t+1) 0 it,i 0 (t+1) z it it,1(t+1) it,i 0 (t+1). D it;1(t+1) R 3 D i 0 (t 1);it. 0 it,m(t+1) it,m(t+1). D it;i 0 (t+1) z 31. D 31;12. z 32. D 32,13. z 33. D M(t 1);it D it;m(t+1) Encoding factor: it (z i,t ) = d i,t z i,t Cardinality factor: R i (z i,:) =, kz i,: k 1 > 0 0, otherwise Dynamic factor: D it 1;i 0,t (z i,t 1,z i 0,t) = log p(x i 0 x i )z i,t 1 z i 0,t Constraint factor: C t (z :,t ) = 0, P M i=1 z i,t =1 1, otherwise. E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

27 Synthetic experiments Data from HMM with M = 50 states and T = 100 Dynamic cost kdpp M-kDPP Seq-kDPP Greedy DS3 SeqFL Encoding cost kdpp M-kDPP Seq-kDPP Greedy DS3 SeqFL Number of representatives Number of representatives Sequence of reps in SeqFL: compatible with dynamic model Encoding + *Dynamic cost kdpp M-kDPP Seq-kDPP Greedy DS3 SeqFL Number of representatives

28 Procedure learning We learn how to do tasks using instructional videos How to perform CPR How to change a flat tire How to make a carrot cake How to assemble a bike How to install chrome cast Large # videos

29 Procedure learning We learn how to do tasks using instructional videos How to perform CPR How to change a flat tire How to make a carrot cake How to assemble a bike How to install chrome cast Goal: Learn grammar of complex tasks using video data procedure:

30 Procedure learning via SeqFL 1) Extract superframes (Gygli et al 14) from each video 2) Extract deep features for each superframe 3) Build a dynamic model: 1st order HMM x 7 x 2 CNN- LSTM h (1) h (1) 1 2 h (1) h (1) 3 T 1 x 1 x 4 x 9 x 6 x 8 x 5 CNN- LSTM h (K) 1 h (K) 2 h (K) 3 h (K) T K x 3 E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

31 Procedure learning via SeqFL 1) Extract superframes (Gygli et al 14) from each video 2) Extract deep features for each superframe 3) Build a dynamic model: 1st order HMM 4) SeqFL (X = HMM, Y = test videos) Video 1 x 7 x 2 x 1 x 4 r x x 5 6 x SeqFL 8 x 9 x 3 h (1) h (1) h (1) 1 2 h (1) 3 T 1 (1) 1, r (1) 2, r (1) 3,, r (1) T 1 Video K SeqFL h (K) 1 h (K) 2 h (K) 3 h (K),,,, T K r (K) 1 r (K) 2 r (K) 3 r (K) T K E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

32 Procedure learning via SeqFL 1) Extract superframes (Gygli et al 14) from each video 2) Extract deep features for each superframe 3) Build a dynamic model: 1st order HMM 4) SeqFL (X = HMM, Y = test videos) 5) Align sequences of representatives 6) Map representatives to actions x 7 x 2,,,, r (1) 1 r (1) 2 r (1) 3 r (1) T 1 Video 1 summary x 1 x 9 x 6 x 4 x 8 x 3 x 5,,,, r (K) 1 r (K) 2 r (K) 3 r (K) T K Video K summary x 5! x 2! x 9! x 3! x 8! x 3! x 8 E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

33 Experimental results Instructional video dataset (Alayrac et al CVPR 16) - 5 tasks, 30 videos/task, labels of key steps given No dynamics with dynamics E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

How to perform CPR Experimental results SeqFL Give

Compression Give Breath Give Compression Ground Truth

Give Compression Give Breath Give Compression How to

Plant Add Top Ground Truth Put Soil Tap Pot Take Plant

34 How to perform CPR Experimental results SeqFL Give Compression Check Breathing Give Breath Give Compression Give Breath Give Compression Ground Truth Check Response Open Airway Check Breathing Give Breath Give Compression Give Breath Give Compression How to repot a plant SeqFL Put Soil Add Top Loosen Root Place Plant Add Top Ground Truth Put Soil Tap Pot Take Plant Loosen Root Place Plant Add Top E. Elhamifar, C. De Paolis Kaluza, NIPS, 2017

35 Online summarization Sequential data: grow incrementally, collecting all and running SS infeasible Approach: incremental subset selection D (0) n D (1) n D (2) n SS E (1) o SS E (2) o t =0 t =1 t =2 SS [ - Instead of running SS on n, run SS on E o (t) [ D n (t) at - Proved exactness of the online SS t D (t) t E. Elhamifar, C. De Paolis Kaluza, CVPR 17

36 Summary and ongoing work Important to have summarization close to global optimum - Developed efficient sparse optimization for facility location - Showed exactness under appropriate condition Important to handle sequential data: compatible seq of reps - Developed SeqFL for sequential summarization - Addressed procedure learning from instructional videos

37 Thanks! Codes:

HIGH RESOLUTION DEPTH IMAGE RECOVERY ALGORITHM USING GRAYSCALE IMAGE.

HIGH RESOLUTION DEPTH IMAGE RECOVERY ALGORITHM USING GRAYSCALE IMAGE Kazunori Uruma 1, Katsumi Konishi 2, Tomohiro Takahashi 1 and Toshihiro Furukawa 1 1 Graduate School of Engineering, Tokyo University