Synfuzz: Building a Grammar Based Re-targetable Test Generation Framework

The presentation discusses the challenges in verifying the correctness of parsers and proposes the use of fuzzing to generate test cases.

• Verifying the correctness of parsers is challenging due to differences in parsing algorithms and poor documentation
• Tree generation flaws and opaque tokens also contribute to the difficulty in testing parsers
• Fuzzing can be used to generate a large number of test cases, but traditional fuzzing methods may not be effective for testing parsers
• A targeted approach to fuzzing can be developed to test parsers more effectively

The presenter mentions crawling the web and looking at Stack Overflow and open-source projects to gather test cases, but notes that this method may not provide enough variety and coverage. They suggest using a buzzer to generate test cases at a higher rate, but note that traditional fuzzing methods may not be effective for testing parsers. The presenter proposes developing a targeted approach to fuzzing to address this issue.

Fuzzers have played an important role in the discovery of reliability and security flaws in software for decades. They have allowed for test case generation at a rate impossible by hand and the creation of test cases humans may never conceive of. While there are many excellent fuzzers available most are designed for mutating source files or input in random ways and attempting to discover edge cases in the handling of them. Some others are designed with structured input in mind and use grammars to more strategically generate and mutate possible inputs that adhere to the format defined. These specifically are the ones we care about for the goals of identifying differences between multiple implementations of a single language, finding bugs in parse tree generation/handling of tokens, and handling of the data at runtime once it has been successfully lexically and syntactically analyzed. We'll look at some of the shortcomings of existing fuzzers and discuss the implementation for a new platform designed to make fuzzer creation easier with the goal of being able utilize grammars from the implementations of the languages themselves.