Universal Style Transfer via Feature Transforms Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang UC Merced, Adobe Research, NVIDIA Research Presented: Dong Wang (Refer to slides by Ibrahim Ahmed and Trevor Chan) August 31, 2018 ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 1 / 20 NVI
Problem Transfer arbitrary visual styles to content images Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 2 / 20 NVI
Related Work Existing works often trade off between generalization, quality, and efficiency. Not efficient during inference: Image Style Transfer using Convolutional Neural Network. (CVPR 2016) Style Specific Networks: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. (ECCV 2016) Poor generalizing abilities in terms of visual quality: Arbitrary Style Transfer in Real-Time with Instance Normalization.(ICCV 2017) ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 3 / 20 NVI
Proposed Method Image Reconstruction + Feature Transforms Train autoencoder for image reconstruction, then fix it Whiten/ Coloring Transform on feature space ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 4 / 20 NVI
Image Reconstruction Encoder: Train VGG-19 on ImageNet Classification task Decoder: Trained to reconstruct the image More than one decoder trained for reconstruction 5 trained decoders Image source: Li et.al ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 5 / 20 NVI
Loss Function for Reconstruction Decoder Combination of pixel reconstruction loss and feature loss. L = I o I i 2 2 + λ Φ(I o ) Φ(I i ) 2 2 (1) I i, I o are the input image and reconstruction output. Φ( ) is the VGG encoder. λ is the weight to balance the two losses. Note: no style image is used in process of training. ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 6 / 20 NVI
Feature Transforms ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 7 / 20 NVI
Feature Transforms by Whitening/Coloring Content features are transformed at intermediate levels by statistics of the style features. In each layer, need content features to exhibit same characteristics of the style features of the same layer. WCT (Whitening/Coloring Transform) achieves this. ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 8 / 20 NVI
Whitening Transform Transform a random vector (a) to be uncorrelated and have unit variance decorrelate the components of original vector (b) scale the different components so they have unit variance (c) ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UC Merced, August 31, Adobe 2018Research, 9 / 20 NVI
Coloring Transform Coloring is the inverse of the whitening transform. Transform white noise into random vector with desired covariance matrix. ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 10 / 20 NVI
Apply WCT to Style Transfer Disassociate input image style and associate the input image with the style of the style image. From content image I c and style image I s, extract their vectorized feature maps f c and f s. WCT will directly transform the f c to match the covariance matrix of f s. ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 11 / 20 NVI
Whitening and Coloring Transform Whitening: Σ c = f c f T c = E c D c E T c ˆf c = E c Dc 1/2 Ec T f c T ˆf c ˆf c = I Coloring: Σ s = f s f T s = E s D s E T s ˆf cs = E s Ds 1/2 Es T ˆf c T ˆf cs ˆf cs = f s fs T ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 12 / 20 NVI
Whitened Image Feature Inverting whitened features. We invert the whitened VGG Relu 4 1 feature as an example. Left: original images, Right: inverted results (pixel intensities are rescaled for better visualization). The whitened features still maintain global content structures. ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 13 / 20 NVI
Multi-level Stylization ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 14 / 20 NVI
Multi-level Stylization Single-level stylization using different VGG features. (a)-(c) Intermediate results of our coarse-to-fine multi-level stylization framework. I1 is the final output of the multi-level pipeline. (d) Reversed fine-to-coarse multi-level pipeline. Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced,31, Adobe 2018 Research, 15 / 20 NVI
Experiment Results Compared with other models Other methods were inferior in terms of Handling arbitrary styles Efficiency Learning-free Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced,31, Adobe 2018 Research, 16 / 20 NVI
Experiment Results ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 17 / 20 NVI
Parameters Image size Style weight control between stylization and content preservation. ˆf cs = α ˆf cs + (1 α)f c ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 18 / 20 NVI
Spatial Control ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 19 / 20 NVI
Takeaways Work with arbitrary styles Do not have to train on style images Scale and weight of style transfer can be changed on the fly ijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Universal XinStyle Lu, Ming-Hsuan Transfer Yang (UCAugust Merced, 31, Adobe 2018 Research, 20 / 20 NVI