Can You Roll Your Own SIEM?

The presentation discusses the decision to roll out a homegrown security information and event management (SIEM) system at Two Sigma, a financial services company. The decision was based on cost, infrastructure, and business drivers, and the new system provides flexibility, extensibility, and improved speed and reliability of streaming alerting. The system also allows for onboarding and validating new data sets and extending response pipelines.

• The decision to roll out a homegrown SIEM system was based on cost, infrastructure, and business drivers
• The new system provides flexibility, extensibility, and improved speed and reliability of streaming alerting
• The system allows for onboarding and validating new data sets and extending response pipelines

Two Sigma's engineering and business teams were already making heavy investments into Google Cloud Platform (GCP) as their cloud platform, so they were able to hang on coattails. BigQuery offers two options to run searches on-demand queries which can be very expensive since you pay per feeds and free our security analysts to experiment on developing additional strategies that they weren't able to before. These include adding new external datasets to help match IOCs and the like as well as adding larger internal feeds like firewall logs and certain network telemetry that enables more sophisticated analysis to detect anomalies and threats.

At Two Sigma, we had sunk over $1 million in licensing for a popular third-party SIEM product and were paying an additional $200,000 in annual maintenance. We were limited on what data sources we could leverage as our license was restricted to a low daily ingestion rate. As our company began to explore cloud transformation broadly, we in Security began investigating competitive options for our event collection and analysis platform. We wanted to know if we could roll our own cloud-native SIEM more efficiently while providing greater access to our data, and be as effective as the vendor's solution.To figure that out, we asked:1. Does the vendor SIEM product cover enough of our threat landscape to make it worth the cost? 2. If not, has our organization made strategic investments in alternate platforms which could be leveraged instead? 3. If yes, does our team have the skills required to implement and mold these platforms to our needs?The answers led us to roll our own SIEM. In our presentation, we'll dig into these questions and decisions in-depth, as well as describe our architecture and several use cases. At the end of the day, we've been running our GCP SIEM for over a year and have moved off the vendor platform. To get started, we wrote less than 6,000 lines of code across a handful of simple tools. We ingest 5TB of data per day and have over 2PB of historical data stored and instantly searchable. In the end, we spent ~$500,000 to build our own SIEM that would have cost us $4 million if we used our third-party vendor. We're also saving an estimated $600,000 year over year in maintenance and subscription fees, plus reducing hardware capital expenditure.