Windows Offender: Reverse Engineering Windows Defender's Antivirus Emulator

The presentation discusses the reverse engineering of Windows Defender's emulator and the importance of emulation in the AV industry.

• Reverse engineering of Windows Defender's JavaScript engine
• Importance of emulation in the AV industry
• Challenges with signature-based heuristics
• Overview of emulation process

The presenter discovered that Windows Defender had a JavaScript engine used for analysis of potentially malicious code and found that the developers made the choice that seemed to emphasize security at the cost of performance. The AV industry moved into emulation to address the problem of signature-based heuristics, which are easily evaded with packed code and Apple binaries. Emulation involves taking potentially malicious unknown binaries and running them in a virtualized environment to see what they are doing at runtime.

Windows Defender's mpengine.dll implements the core of Defender antivirus' functionality in an enormous ~11 MB, 45,000+ function DLL. In this presentation, we'll look at Defender's emulator for analysis of potentially malicious Windows PE binaries on the endpoint. To the best of my knowledge, there has never been a conference talk or publication on reverse engineering the internals of any antivirus binary emulator before. I'll cover a range of topics including emulator internals (bytecode to intermediate language lifting and execution; memory management; Windows API emulation; NT kernel emulation; file system and registry emulation; integration with Defender's antivirus features; the virtual environment; etc.), how I built custom tooling to assist in reverse engineering and attacking the emulator; tricks that malicious binaries can use to evade or subvert analysis; and attack surface within the emulator. I'll share code that I used to instrument Defender and IDA scripts that can be helpful in reverse engineering it.