Impact of new imaging modalities on video surveillance and invasion of privacy Pavel Korshunov 25/11/14 VideoSense Winter School 2014 1
The reality today 25/11/14 - VideoSense Winter School 2014 2
Motivation 25/11/14 - VideoSense Winter School 2014 3
How do the new imaging technologies impact privacy? 25/11/14 - VideoSense Winter School 2014 4
Outline Privacy evaluation tools Subjective evaluation Crowdsourcing Objective metrics Datasets and evaluation results UHD HDR Mini-drones 25/11/14 - VideoSense Winter School 2014 5
Privacy-intelligibility tradeoff Protect privacy obfuscate or remove personal information from the video Perform surveillance determine suspicious event/person, apprehend and prosecute criminals Where is the balance? 25/11/14 - VideoSense Winter School 2014 6
Privacy filters Blurring, pixelization, masking, etc. Replacing an object with background or with another object Encryption and scrambling Geometrical filters a) b) c) d) e) f) g) b) c) Figure 3 Examples of privacy protection approaches: a) original image, b) pix with b=16, d) Gaussian blur with σ=8, e) Gaussian blur with σ=12, f) scrambli scrambling by random permutation. 25/11/14 - VideoSense Winter School 2014 7
Evaluation tools Subjective evaluation Crowdsourcing Objective metrics 25/11/14 - VideoSense Winter School 2014 8
Subjective evaluation Three naïve filters: blurring, pixelization, and masking Dataset of 10 annotated videos People acting normally or abnormally Wearing glasses, scarf, hats, etc. Blinking into the camera 25/11/14 - VideoSense Winter School 2014 9
Example 25/11/14 - VideoSense Winter School 2014 10
Questions asked race white asian I don t know Privacy gender female male I don t know glasses yes no I don t know sunglasses yes no I don t know Intelligibility scarf yes no I don t know blinking yes no I don t know 25/11/14 - VideoSense Winter School 2014 11
Privacy Subjective results 1 blurring filter pixelization filter masking filter 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 Intelligibility 25/11/14 - VideoSense Winter School 2014 12
Evaluation tools Subjective evaluation Crowdsourcing Objective metrics 25/11/14 - VideoSense Winter School 2014 13
What is crowdsourcing? Short and simple online tasks Large number of different workers Realistic settings A low cost alternative to subjective evaluations 25/11/14 - VideoSense Winter School 2014 14
Facebook-based evaluations The same experiment as for subjective evaluations Facebook-based system Shows videos Collects answers Trusted workers 25/11/14 - VideoSense Winter School 2014 15
Crowdsourcing: privacy 1 0.9 blurring masking pixelization 0.8 0.7 Online results 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Offline results 25/11/14 - VideoSense Winter School 2014 16
Crowdsourcing: intelligibility 1 0.9 blurring masking pixelization 0.8 0.7 Online results 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Offline results 25/11/14 - VideoSense Winter School 2014 17
Outcome Crowdsourcing is cheap and effective alternative to lab-based evaluations 25/11/14 - VideoSense Winter School 2014 18
Evaluation tools Subjective evaluation Crowdsourcing Objective metrics 25/11/14 - VideoSense Winter School 2014 19
Objective evaluations The cheapest and the most scalable option People counting in public transport Face recognition is the metric of privacy Face detection is the metric of intelligibility An ideal privacy protection filter Degrades face recognition Does not affect face detection 25/11/14 - VideoSense Winter School 2014 20
Evaluation examples Increase strength of privacy filter Note relative decrease in accuracy of face detection and recognition 25/11/14 - VideoSense Winter School 2014 21
Accuracy Blurring, FERET dataset Detection Recognition Gaussian kernel size 25/11/14 - VideoSense Winter School 2014 22
Morphing, FERET dataset Recognition morphed Recognition recovered 1 1 0.9 0.9 0.8 0.8 0.7 0.7 Accuracy 0.6 0.5 0.4 interpolation_0 interpolation_0.2 interpolation_0.4 interpolation_0.6 interpolation_0.8 interpolation_1 Accuracy 0.6 0.5 0.4 interpolation_0 interpolation_0.2 interpolation_0.4 interpolation_0.6 interpolation_0.8 interpolation_1 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Intensity strength 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Intensity strength 25/11/14 - VideoSense Winter School 2014 23
Outcome Finding objective metrics for privacy and intelligibility is hard Context and content are important 25/11/14 - VideoSense Winter School 2014 24
Outline Visual privacy filters Privacy-intelligibility tradeoff Evaluation tools Datasets Benchmarking Future challenges 25/11/14 - VideoSense Winter School 2014 25
Outline Privacy evaluation tools Subjective evaluation Crowdsourcing Objective metrics Datasets and evaluation results UHD HDR Mini-drones 25/11/14 - VideoSense Winter School 2014 26
Dataset design principles Wide range of surveillance scenarios Emphasis on personal visual information A mechanism to select privacy regions for evaluation New video capturing technologies 25/11/14 - VideoSense Winter School 2014 27
Types of video data Ultra high definition (4K UHD) video High dynamic range (HDR) images and video Video from mini-drones 25/11/14 - VideoSense Winter School 2014 28
Ultra High Definition (UHD) dataset 25/11/14 - VideoSense Winter School 2014 29
UHD dataset Samsung Galaxy Note 3 3840 2160 resolution video 26 surveillance sequences, 13 seconds Manually annotated body silhouettes and personal information UHD shows more details more privacy intrusive? Subjective evaluations UHD vs. HD vs. SD 25/11/14 - VideoSense Winter School 2014 30
Walking example 25/11/14 - VideoSense Winter School 2014 31
Exchanging bag example 25/11/14 - VideoSense Winter School 2014 32
Fighting example 25/11/14 - VideoSense Winter School 2014 33
Subjective evaluations of privacy 4K UHD Sony reference monitor Evaluation questions about Gender Race People s accessories Main action Visible items Accompanied questions on certainty 25/11/14 - VideoSense Winter School 2014 34
% of correct answers Measuring privacy 25/11/14 - VideoSense Winter School 2014 35
Measuring certainty 25/11/14 - VideoSense Winter School 2014 36
High Dynamic Range (HDR) dataset 25/11/14 - VideoSense Winter School 2014 37
Implications of HDR HDR surveillance cameras More details in the scene More privacy intrusive? Woman?? Man?? Middle Age?? Woman Young?? HDR monitor is needed Tone-mapped images Woman Elderly Wears sunglasses Man Middle Aged Woman Young 25/11/14 - VideoSense Winter School 2014 38
HDR dataset 20 images and 68 video sequences Variety of lighting conditions, scenarios, gender, and race Images shot with Nikon D7100 camera, 5 exposures Video shot with BlackMagic Pocket Cinema Camera JAI AD-132 GE Bosch DINION 7000 HD 25/11/14 - VideoSense Winter School 2014 39
25/11/14 - VideoSense Winter School 2014 40
25/11/14 - VideoSense Winter School 2014 41
25/11/14 - VideoSense Winter School 2014 42
25/11/14 - VideoSense Winter School 2014 43
25/11/14 - VideoSense Winter School 2014 44
Crowdsourcing evaluations of images About 400 people participated Evaluation questions about Gender Race Age Color of clothes How many people 25/11/14 - VideoSense Winter School 2014 45
Crowdsourcing results True HDR images LDR-like images 25/11/14 - VideoSense Winter School 2014 46
Conclusions Privacy is a subjective notion Privacy is context and content dependent Privacy protection disrupts main utility Privacy protection and evaluation are challenging 25/11/14 - VideoSense Winter School 2014 47
References subjective evaluations P. Korshunov, C. Araimo, F. De Simone, C. Velardo, and J.-L. Dugelay. Subjective Study of Privacy Filters in Video Surveillance. IEEE International Workshop on Multimedia Signal Processing (MMSP), Banff, Canada, 2012. P. Korshunov, S. Cai, and T. Ebrahimi. Crowdsourcing Approach for Evaluation of Privacy Filters in Video Surveillance. International ACM Workshop on Crowdsourcing for Multimedia (CrowdMM'12), Nara, Japan, 2012. P. Korshunov, A. Melle, J.-L. Dugelay, and T. Ebrahimi. A framework for objective evaluation of privacy filters in video surveillance. Applications of Digital Image Processing XXXVI, 2013., San Diego, California, USA, Proceedings of SPIE, 2013. P. Korshunov, H. Nemoto, A. Skodras, and T. Ebrahimi. The effect of HDR images on privacy: crowdsourcing evaluation. SPIE Photonics Europe 2014, Optics, Photonics and Digital Technologies for Multimedia Applications, April 2014, Brussels, Belgium. P. Korshunov and T. Ebrahimi. Towards optimal distortion-based visual privacy filters. IEEE International Conference on Image Processing (ICIP), Paris, France, 2014. 25/11/14 - VideoSense Winter School 2014 48
References datasets P. Korshunov and T. Ebrahimi. PEViD: privacy evaluation video dataset. Applications of Digital Image Processing XXXVI, 2013., San Diego, California, USA, Proceedings of SPIE, 2013. M. Rerabek, L. Yuan, L. Krasula, P. Korshunov, K. Fliegel, and Touradj Ebrahimi. Evaluation of privacy in high dynamic range video sequences. SPIE Optical Engineering + Applications, San Diego, California, USA, 2014. P. Korshunov and T. Ebrahimi. UHD Video Dataset for Evaluation of Privacy. Sixth International Workshop on Quality of Multimedia Experience (QoMEX 2014), Singapore, 2014. 25/11/14 - VideoSense Winter School 2014 49
Disclaimer Some of the illustrations may have copyright constraints 25/11/14 VideoSense Winter School 2014 50