⚡ Lightning Talk: Securing Envoy: Catching Vulnerabilities With Continuous Fuzz Testing

Envoy developers use coverage-guided fuzz tests to automatically discover parser bugs and harden Envoy for production deployments.

• Service proxies are becoming increasingly complex data parsers
• Envoy supports HTTP protocol translation, payload decompression, gRPC-JSON transcoding, and many other features that operate directly on raw request bytes
• Malicious HTTP requests may trigger undefined behavior, resulting in service mesh downtime
• Coverage-guided fuzz tests capture intricate corner cases that a developer may have missed
• Integration with OSS Fuzz provides continuous fuzz testing and vulnerability reports
• Continuous fuzzing is essentially running fuzzers 24/7 in the background
• Coverage-guided fuzzers employ a feedback loop to generate inputs based on code coverage
• Optimization problem: fuzzers are trying to optimize against the input space and the loss function is inversely proportional to code coverage

The speaker gave an example of a C++ function that unescapes new line characters in the input string. The function worked well for the majority of inputs, but there were a few problems with it. The indexing was incrementing in two places, which could cause invalid behavior in C++. To catch edge cases like this, the speaker wrote a fuzzer that could generate random strings. The fuzzer used a feedback loop to generate inputs based on code coverage, and it caught the error in the library under test when it generated a string with a trailing backslash. This example illustrates the importance of continuous fuzz testing and coverage-guided fuzzers in discovering parser bugs and hardening Envoy for production deployments.

Service proxies are becoming increasingly complex data parsers. Out of the box, Envoy supports HTTP protocol translation, payload decompression, gRPC-JSON transcoding, and many other features that operate directly on raw request bytes. Coupled with the deployment of Envoy as a "front" edge proxy, these parser-like features become an attack surface for external clients — a malicious HTTP request may trigger undefined behavior, resulting in service mesh downtime. Envoy developers use coverage-guided fuzz tests to automatically discover parser bugs. When run with C++ sanitizers, fuzz tests capture intricate corner cases a developer may have missed. Integration with OSS Fuzz provides continuous fuzz testing and vulnerability reports, allowing developers to fix such bugs before the features are released. Join this talk for a case study on Envoy's multi-year adoption of fuzz testing. Learn about how we harden Envoy for production deployments and the surprising bugs our fuzz tests discovered!