The presentation discusses the challenges of demonstrating mathematically novel attacks on industry infrastructure and the importance of considering bad inputs and model leakage. It also highlights the limitations of gradient-based attacks and the potential of quick and dirty attacks.
- Demonstrating mathematically novel attacks on industry infrastructure is challenging due to the additional layer of infrastructure and artificial constraints in the literature.
- Bad inputs and model leakage are two types of exploits that should be considered.
- Quick and dirty attacks are clever and do not require specific knowledge of algorithms or models.
- Gradient-based attacks constitute a much smaller percentage compared to other less mathy and quick and dirty attacks.
- Anecdotes include self-driving cars interpreting physical artifacts as part of the environment and malware classifiers being fooled by appending carefully selected strings to a malicious file.
- Deep fakes are more susceptible to gradient-based attacks, but larger models may be more robust to model leakage.
An example of bad inputs is with self-driving cars. Last year, the presenter had a talk at Defcon where they took some salt and poured it on the ground because it was cold and drove their friend's Tesla. Lo and behold, the Tesla started driving off into the sunset when it should not have been doing that. Another example is with malware classifiers, where appending carefully selected strings to a malicious file can fool the classifier into marking that file as benign.