We're performing a technical deep dive into differential privacy: preventing models from memorising private data. Theory + Colab notebook using Tensorflow Privacy! Social: Twitter: https://twitter.com/mukulrathi_ My website (+ blog): https://mukulrathi.com/ My email newsletter: https://newsletter.mukulrathi.com/ ----------------- Links: DP-SGD paper: https://arxiv.org/pdf/1607.00133.pdf Tensorflow Privacy Tutorials: https://github.com/tensorflow/privacy/tree/master/tutorials Tensorflow Privacy: https://github.com/tensorflow/privacy Pytorch Opacus: https://github.com/pytorch/opacus Moments accountant implementation: https://github.com/marcotcr/tf-models/blob/master/differential_privacy/privacy_accountant/tf/accountant.py GPT-2 memorises private data: https://ai.googleblog.com/2020/12/privacy-considerations-in-large.html Netflix dataset deanonymised: https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/ Netflix deanonymisation paper: https://www.cs.utexas.edu/~shmat/shmat_oak08netflix.pdf Strava heatmap leaks: https://www.zdnet.com/article/strava-anonymized-fitness-tracking-data-government-opsec/ ------------------------------------------------------------------------------------ Timestamps: 00:00 Introduction 00:32 Overview 01:26 Why Anonymisation Isn't Enough 02:38 Intuition for Differential Privacy 03:12 Example: Predict whether Bob has Cancer 04:11 Privacy Intuition 04:51 Privacy Loss Definition 05:26 Definition of Differential Privacy 06:40 Role of Noise in DP 07:08 Privacy Amplification Theorem 07:26 Fundamental Law of Information Recovery 07:51 Composition in DP 08:19 DP-SGD 09:20 Moments Accountant 12:55 Google Colab Notebook 14:39 Limitations of DP-SGD --------------------------------------------------------------------------- Music: Coffee Break by Pyrosion is licensed under a Creative Commons License. https://creativecommons.org/licenses/... https://soundcloud.com/pyrosion Support by RFM - NCM: https://bit.ly/2xGHypM