Reverse Engineering Windows Defender's Emulator

The presentation discusses reverse engineering and vulnerability research on Microsoft Defender ATP.

• Reverse engineering of Microsoft Defender ATP
• AV instrumentation throughout the engine
• Vulnerabilities discovered by Google Project Zero
• Abuse of API call instruction by malware
• Importance of reverse engineering and vulnerability research

The presenter discussed how malware can abuse the API call instruction to exploit vulnerabilities in Microsoft Defender ATP. This highlights the importance of reverse engineering and vulnerability research to identify and address such vulnerabilities.

Windows Defender Antivirus's mpengine.dll implements the core of Defender's functionality in an enormous ~11 MB, 30,000+ function DLL. In this presentation, we'll look at Defender's emulator for analysis of potentially malicious Windows binaries on the endpoint. To the best of my knowledge, there has never been a conference talk or publication on reverse engineering any antivirus binary emulator before. We'll cover a range of topics including emulator internals—machine code to intermediate language translation and execution; memory management; Windows API emulation; NT kernel emulation; file system and registry emulation; integration with Defender's antivirus features; the virtual environment; etc.—building custom tooling for instrumenting the emulator; tricks that binaries can use to evade or subvert analysis; and attack surface within the emulator. Attendees will leave with an understanding of how modern antivirus software conducts emulation-based dynamic analysis on the endpoint, and how attackers might go about subverting or attacking these systems. I'll publish code for a binary for exploring the emulator from within, patches that I developed for instrumenting Defender built on top of Tavis Ormandy's loadlibrary project, and IDA scripts to help with analyzing mpengine.dll and Defender's "VDLLs"