Experimenting with Real-Time Event Feeds

Today, defenders in a typical security operation center rely on their SIEM to do forensics on past logs, and to define real-time detections. This assumes that the SIEM was configured ahead of time to collect the subset of logs that are useful. But how does one decide what is useful? Further, some data comes at such high-volume that storing it in raw form is prohibitively expensive. Such data must be prefiltered and summarized before storage for query.We present tools and a method of comparing various options of filtering and pre-processing real-time feeds of data before storage. This can be done in isolated environments without SIEM coverage, such as labs/honeypots for researching Malware or Proof of Concept (POC) for exploits.The learnings of the method can be applied to understanding novel threats and creating true-real-time detections that work directly on the stream of events (no storage involved).