Petacat: Applying ideas from Copycat to image understanding
How Streetscenes Works (Bileschi, 2006) 1. Densely tile the image with windows of different sizes. 2. HMAX C2 features are computed in each window. 3. The features in each window are given as input to each of five trained support vector machines ( pedestrian, car, bicycle, building, tree ) 4. If any return a classification with score above a learned threshold, that object is said to be detected.
Object detection (here, car ) with HMAX model (Bileschi, 2006)
Limitations of Streetscenes approach for image understanding
Limitations of Streetscenes approach for image understanding Exhaustive search not scalable Does not recognize spatial and abstract relationships among objects for whole scene understanding Has no prior knowledge about object categories and their place in conceptual space HMAX model is completely feed-forward; no feedback to allow context to aid in scene understanding. Where should feedback come in?
Representation of High-Level Knowledge: A Simple Semantic Network (or Ontology ) walking holds leash attached to Person action action walking
But...
Modified Ontology walking holds leash attached to Group Person action action walking running
Modified Ontology walking holds leash attached to Group Person action walking action running Allowing conceptual slippage
But...
Modified Ontology walking holds leash attached to Group Person action action Cat walking running Iguana
But...
But...
But...
But...
Modified Ontology Helicopter walking Bicycle holds leash attached to Group Person action action Cat Car walking running Iguana
But...
Sidewalk Beach Outside Inside Grass Lawn mower Sky Above Holding Attached to Gasoline Ground Car walking Tree Runway Left of Person Backpack Leash Airplane Standing Far from Walking grooming Close to Fanny pack Running Stick Track Helicopter Army
Need dynamical process of constructing representation.
Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes.
Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. Ongoing perception of context brings in appropriate concepts and conceptual slippages, and avoids exhaustive search
Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. Ongoing perception of context brings in appropriate concepts and conceptual slippages, and avoids exhaustive search Prior, higher-level knowledge interacts with lower-level vision in both directions (bottom-up and top-down).
Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. Ongoing perception of context brings in appropriate concepts and conceptual slippages, and avoids exhaustive search Prior, higher-level knowledge interacts with lower-level vision in both directions (bottom-up and top-down). Concepts are fluid, allowed to slip in certain contexts.
Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. Ongoing perception of context brings in appropriate concepts and conceptual slippages, and avoids exhaustive search Prior, higher-level knowledge interacts with lower-level vision in both directions (bottom-up and top-down). Concepts are fluid, allowed to slip in certain contexts. This allows perception of essential similarity in the face of superficial differences i.e., analogy-making.
Active Symbol Architecture (Hofstadter et al., 1995)
Active Symbol Architecture (Hofstadter et al., 1995) Basis for Copycat (analogy-making), Hofstadter & Mitchell Tabletop (anlaogy-making), Hofstadter & French Metacat (analogy-making and self-awareness), Hofstadter & Marshall and many others
Active Symbol Architecture (Hofstadter et al., 1995) Semantic network Workspace Temperature Perceptual agents (codelets)
Petacat: (Descendant of Copycat) Integration of Active Symbol Architecture and HMAX Initial task: Decide if image is an instance of taking a dog for a walk, and if so, how good an instance it is.
indoors has location Semantic Network taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
indoors has location Semantic Network taking a dog for a walk Property links Slip links road a beach a trail has action grass a sidewalk outdoors is on runs has action stands person walks drives sits is in front of is touching belt flies swims leash is touching rope string horse dog cat is on is on is behind Object Action Spatial Relation is touching is next to has location is in front of
indoors has location Semantic Network taking a dog for a walk Property links Slip links road a beach a trail has action grass a sidewalk outdoors is on runs has action stands person walks drives sits is in front of is touching belt flies swims leash is touching rope string horse dog cat is on is on is behind Object Action Spatial Relation is touching is next to Properties of nodes has location is in front of
Workspace
Workspace Semantic network
Perceptual Agents (Codelets) Semantic network Codelets as active symbols
indoors has location taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
indoors has location taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
indoors has location taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
indoors has location taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
Illustration of what we plan to have happen not a real run of Petacat?
Illustration of what we plan to have happen not a real run of Petacat Person???
Illustration of what we plan to have happen not a real run of Petacat Person??? Sidewalk?
Illustration of what we plan to have happen not a real run of Petacat Person? Outdoors??? Sidewalk??
Illustration of what we plan to have happen not a real run of Petacat Person? Outdoors??? Sidewalk?? Scout codelets: Send C1 features in window to corresponding SVM. If positive result, post builder codelet with urgency equal to SVM s confidence.
Illustration of what we plan to have happen not a real run of Petacat Person? negative? negative? negative Outdoors? positive: 0.7 Sidewalk? positive: 0.4? positive: 0.8 Scout codelets: Send C1 features in window to corresponding SVM. If positive result, post builder codelet with urgency equal to SVM s confidence.
Illustration of what we plan to have happen not a real run of Petacat Person? negative? negative? negative Outdoors? positive: 0.7 Sidewalk? positive: 0.4? positive: 0.8 Builder codelets: Ask HMAX to compute C2 features using prototypes specific to the object (or scene), and send them to corresponding SVM. If positive, decide to build structure with probability equal to SVM confidence. Break competing structures if necessary.
Illustration of what we plan to have happen not a real run of Petacat Outdoors Builder codelets: Ask HMAX to compute object-/scene-specific C2 features, and send them to corresponding SVM. If positive, decide to build structure with probability equal to SVM confidence. Break competing structures if necessary.
indoors has location taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
Illustration of what we plan to have happen not a real run of Petacat Person? Outdoors Leash? Leash? Person?? Sidewalk?
Illustration of what we plan to have happen not a real run of Petacat Outdoors Person Strength: 0.6 Sidewalk
Illustration of what we plan to have happen not a real run of Petacat Outdoors Sidewalk
indoors has location taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
Illustration of what we plan to have happen not a real run of Petacat Leash? Outdoors Leash?? Rope?? Sidewalk? Sidewalk
Illustration of what we plan to have happen not a real run of Petacat Leash Outdoors (weak) Sidewalk
Illustration of what we plan to have happen not a real run of Petacat Leash Outdoors (strong) (weak) Sidewalk
Illustration of what we plan to have happen not a real run of Petacat Leash Outdoors Sidewalk
indoors has location taking a dog for a walk outdoors Object road a beach a grass a sidewalk is on person is in front of is touching leash dog is touching is on Action Spatial Relation trail has action runs has action stands walks drives sits belt flies swims rope string horse cat is on is behind is touching is next to has location is in front of
Illustration of what we plan to have happen not a real run of Petacat is next to is in front of Leash Outdoors is next to is in front of group Sidewalk Once objects begin to be built, relation and grouping codelets can run on them.
Illustration of what we plan to have happen not a real run of Petacat is next to Leash Outdoors is next to group Sidewalk Once objects begin to be built, relation and grouping codelets can run on them.
Illustration of what we plan to have happen not a real run of Petacat is next to Leash Outdoors is next to is next to group Sidewalk
How codelets decide where to look System starts out with weak segmentation (e.g., normalized cuts algorithm)
How codelets decide where to look System starts out with weak segmentation (e.g., normalized cuts algorithm) System creates heat maps for location and scale of objects in general (at each pixel, probability of finding an object at this location and at a particular height/width of bounding box. + + + +
How codelets decide where to look System starts out with weak segmentation (e.g., normalized cuts algorithm) System creates heat maps for location and scale of objects in general (at each pixel, probability of finding an object at this location and at a particular height/width of bounding box. Object scout codelets choose location and + + + + scale probabilisitically from these heat maps.
How codelets decide where to look When codelets look for individual object categories (e.g., dog), objectspecific heat maps are created + + Person heat map
How codelets decide where to look When codelets look for individual object categories (e.g., dog), objectspecific heat maps are created Person? Person? As codelets build structure, heat maps are continually updated to reflect prior (learned) expectations about location and scale as a function of location and scale of built objects (as well as original weak segmentation). + + Person heat map
Illustration of what we plan to have happen not a real run of Petacat How Petacat makes a final decision taking a dog for a walk is next to Leash Outdoors is next to is next to group Sidewalk Temperature
Illustration of what we plan to have happen not a real run of Petacat How Petacat makes a final decision taking a dog for a walk is next to Leash Outdoors is next to is next to group Sidewalk Temperature Situation codelet is more likely to run when temperature is low.
Illustration of what we plan to have happen not a real run of Petacat is next to Leash Outdoors is next to Situation codelet tries to match prototypical situation with existing workspace structures, possibly allowing slippages. is next to group Sidewalk
is next to taking a dog for a walk Leash person is in front of leash has location Outdoors outdoors is next to dog Situation codelet tries to match prototypical situation with existing workspace structures, possibly allowing slippages. is next to group Sidewalk
is next to taking a dog for a walk person Leash is in front of leash has location Outdoors outdoors is next to dog is next to is next to group group Sidewalk
taking a dog for a walk If resulting temperature is low enough, is next to classify scene as positive person Leash is in front of leash has location Outdoors outdoors is next to dog is next to is next to group group Sidewalk
taking a dog for a walk If resulting temperature is low enough, is next to classify scene as positive person Leash is in front of leash is next to has location Outdoors is next to If situation codelet fails enough times is next or to does not run for a long time, group Sidewalk program has increasing chance of ending with negative classification. group outdoors dog
If Petacat classifies the picture as positive, the temperature at the end of the run gives a measure of how good an instance the picture is (e.g., of the dog walking situation).
Summary:
Summary: How does Petacat avoid exhaustive search?
Summary: How does Petacat avoid exhaustive search? Recall Streetscenes system, which, given an image, does exhaustive search over: Window size and location in the image C1, C2 features in windows Object categories (e.g., car, pedestrian, tree, etc.)
Summary: How does Petacat avoid exhaustive search? Recall Streetscenes system, which, given an image, does exhaustive search over: Window size and location in the image In Petacat, codelets choose window size and location based on learned expectations and perceived context, with probabilities continually changing as more information is obtained C1, C2 features in windows Object categories (e.g., car, pedestrian, tree, etc.)
Summary: How does Petacat avoid exhaustive search? Recall Streetscenes system, which, given an image, does exhaustive search over: Window size and location in the image In Petacat, codelets choose window size and location based on learned expectations and perceived context, with probabilities continually changing as more information is obtained C1, C2 features in windows Codelets request C2 features only in relevant windows, and request only C2 features that are relevant to what the codelet is looking for. Object categories (e.g., car, pedestrian, tree, etc.)
Summary: How does Petacat avoid exhaustive search? Recall Streetscenes system, which, given an image, does exhaustive search over: Window size and location in the image In Petacat, codelets choose window size and location based on learned expectations and perceived context, with probabilities continually changing as more information is obtained C1, C2 features in windows Codelets request C2 features only in relevant windows, and request only C2 features that are relevant to what the codelet is looking for. Object categories (e.g., car, pedestrian, tree, etc.) Codelets look for object categories that are activated by context, based on prior expectations and currently perceived information.
Summary: How does Petacat avoid exhaustive search? Petacat effects a parallel terraced scan (Hofstadter, 1995): Codelets build structures at a rate (urgency) based on their perceived promise, which is continually updated as new information is perceived. Temperature allows this (continually changing) rate to depend on the global state of the system.
Relation to neuroscience/psychophysics Gilbert & Sigman (2007): Emphasis of role to top-down processing in vision. V1 and V2 may work as active blackboards that integrate and sustain the result of computations performed in higher areas. Kahneman, Triesman, and Gibbs (1992): Notion of object files : temporary and modifiable perceptual structures, created on the fly in working memory, which interact with a permanent network of concepts. Churchland, Ramachandran, and Sejnowski: Theory of interactive vision Treisman and colleagues: Shift between parallel, random, preattentive bottom-up processing and more deterministic, focused, serial, attentive top-down processing.
Does Petacat understand pictures?
Does Petacat understand pictures? Understanding (MM s defintion): - Ability to appropriately use one s knowledge and make appropriate conceptual slippages in a wide variety of environments/contexts. - Ability to use one s existing concepts to learn new concepts