Deformable Convolutional Networks

Transcription

1 Deformable Convolutional Networks -- MSRA COCO Detection & Segmentation Challenge 2017 Entry Jifeng Dai With Haozhi Qi*, Zheng Zhang, Bin Xiao, Han Hu, Bowen Cheng*, Yichen Wei Visual Computing Group Microsoft Research Asia (*interns at MSRA)

2 Outline Deformable ConvNets idea Deformable ConvNets for COCO challenge

3 Highlights Enabling effective modeling of spatial transformation in ConvNets No additional supervision for learning spatial transformation Significant accuracy improvements on sophisticated vision tasks Code is available at

4 Modeling Spatial Transformations A long standing problem in computer vision Deformation: Scale: Viewpoint variation: Intra-class variation: (Some examples are taken from Li Fei-fei s course CS223B, )

5 Traditional Approaches 1) To build training datasets with sufficient desired variations 2) To use transformation-invariant features and algorithms Scale Invariant Feature Transform (SIFT) Deformable Part-based Model (DPM) Drawbacks: geometric transformations are assumed fixed and known, hand-crafted design of invariant features and algorithms

6 Spatial Transformations in CNNs Regular CNNs are inherently limited to model large unknown transformations The limitation originates from the fixed geometric structures of CNN modules regular convolution 2 layers of regular convolution regular RoI Pooling

7 Spatial Transformer Networks Learning a global, parametric transformation on feature maps Prefixed transformation family, infeasible for complex vision tasks

8 Deformable Convolution Local, dense, non-parametric transformation Learning to deform the sampling locations in the convolution/roi Pooling modules regular deformed scale & aspect ratio rotation

9 Deformable Convolution Regular convolution Deformable convolution where is generated by a sibling branch of regular convolution

10 Deformable RoI Pooling Regular RoI pooling Deformable RoI pooling where is generated by a sibling fc branch deformable RoI Pooling

11 Deformable ConvNets Same input & output as the plain versions Regular convolution -> deformable convolution Regular RoI pooling -> deformable RoI pooling End-to-end trainable without additional supervision

12 Sampling Locations of Deformable Convolution (a) standard convolution (b) deformable convolution

13 Part Offsets in Deformable RoI Pooling

14 Deformable ConvNets for Object Detection Regular object detectors Fast(er) RCNN R-FCN FPN Car Person RoI Pooling RoI Pooling Position Sensitive RoI Pooling Car Person Conv5 P5 Person Conv5 Position Sensitive Score Map Conv4 P4 Car Conv3 P3

15 Deformable ConvNets for Object Detection Deformable object detectors Deformable RoI Pooling Fast(er) RCNN R-FCN FPN Car Person Deformable PS RoI Pooling Car Person Conv5 P5 Deformable RoI Pooling Person Conv5 Position Sensitive Score Map Conv4 P4 Car Conv3 P3 : Deformable Convolution / RoI Pooling

16 conv 128 1x1 stride2, pad 0 conv 256 1x1 entry flow 224x224x3 images conv 32, 3x3, stride2, pad 1 conv 64, 3x3, pad 1 separable conv 128, 3x3, pad 1 separable conv 128, 3x3, pad 1 max pool 3x3, stride 2, pad 1 separable conv 256, 3x3, pad 1 separable conv 256, 3x3, pad 1 XCeption -> Aligned XCeption Proper feature alignment in XCeption Efficient: 9.5 GFLOPS on 224*224 img (ResNet-101, 7.6 GFLOPS) Accurate: map 2.8% better than ResNet-101 using FPN on COCO (det, test-dev) exit flow 14x14x728 feature maps conv 256 1x1, stride2, pad 0 separable conv 256, 3x3, pad 1 separable conv 256, 3x3, pad 1 separable conv 256, 3x3, pad 1 max pool 3x3, stride2, pad 1 middle flow 14x14x728 feature maps conv x1 stride2, pad 0 separable conv 728, 3x3, pad 1 separable conv 1024, 3x3, pad 1 separable conv 728, 3x3, pad 1 separable conv 728, 3x3, pad 1 max pool 3x3, stride2, pad 1 conv 728 1x1 separable conv 728, 3x3, pad 1 separable conv 728, 3x3, pad 1 conv 728 1x1, stride2, pad 0 separable conv 728, 3x3, pad 1 separable conv 728, 3x3, pad 1 separable conv 728, 3x3, pad 1 max pool 3x3, stride2, pad 1 separable conv 728, 3x3, pad 1 14x14x728 feature maps separable conv 1536, 3x3, pad 1 separable conv 1536, 3x3, pad 1 separable conv 2048, 3x3, pad 1 14x14x728 feature maps Repeat 16 times 7x7x2048 feature maps

17 Object Detection on COCO (Test-dev) MSRA 2017 Entry ~3% map improvements by Deformable ConvNets Best single model performance: 48.5% + ENSEMBLE (6 MODELS) HORIZONTAL FLIP + ITERATIVE TESTING MULTI-SCALE TESTING SOFT NMS MASK FPN+OHEM (ALIGNED XCEPTION) FPN+OHEM (RESNET-101) map (%) Deformable Regular

18 Object Detection on COCO (Test-dev) Deformable ConvNets v.s. regular ConvNets Noticeable improvements for varies baselines Marginal parameter & computation overhead FPN++ (ALIGNED-XCEPTION) FPN+OHEM (ALIGNED-XCEPTION) FPN+OHEM (RESNET-101) R-FCN (ALIGNED-INCEPTION-RESNET) R-FCN (RESNET-101) FASTER R-CNN, 2FC (RESNET-101) CLASS-AWARE RPN (RESNET-101) map (%) Deformable Regular

19 Conclusion Deformable ConvNets for dense spatial modeling Simple, efficient, deep, and end-to-end No additional supervision Feasible and effective on sophisticated vision tasks for the first time Our team Haozhi Qi* Zheng Zhang Bin Xiao Han Hu Bowen Cheng* Yichen Wei Jifeng Dai * interns at MSRA