Bridging Security Infrastructure Between the Data Center and AWS Lambda

The presentation discusses the benefits and challenges of using serverless computing with AWS Lambda, and how to integrate it with existing data center infrastructure using an envoy service mesh. The main goal is to maintain the benefits of Lambda while treating it equally as other workloads and ensuring compatibility with data center infrastructure.

• Serverless computing with AWS Lambda allows developers to focus on business logic and not worry about infrastructure
• Lambda is fast, scalable, and well-connected with the AWS ecosystem
• Lambda functions are called without assigned infrastructure and run on small VMs using firecracker VM and sandbox
• Lambda is incompatible with data center infrastructure due to differences in workload duration and deploy process
• To integrate Lambda with data center infrastructure, an envoy service mesh is used to connect services and maintain security
• The main challenge is to ensure Lambda functions can communicate securely with any application connected to the mesh and access application secrets in a data center-based system while maintaining response time and scalability
• Hybrid environments require interoperability and a buffer pouch to let environments support each other and avoid blocking

The presentation gives an example of how to call into the envoy service mesh from Lambda to communicate with a service in the data center without changing the code or infrastructure for the service. The speaker also discusses the importance of identity and security in the network, using an mTLS system that assumes the network is compromised and relies on identities for trust.

While serverless is all the rage, creating secure infrastructure that integrates serverless technology with existing Data Center (DC) services remains a challenge. Square's DC uses a microservice architecture. Services communicate over an envoy service mesh with short-lived mTLS certificates using SPIFFE identity for zero-trust based authentication. To achieve higher flexibility and scalability we have been migrating to the cloud, a gradual process that is still in progress.Why bother? Workloads have different characteristics, while a payment system might be required to be available all the time and have predictable traffic, other applications might have unpredictable bursts of use but otherwise receive no traffic. This flexibility draws developers to Lambda. Applications can scale up immediately, but also scale down when demand is low. However, these characteristics also make security engineering challenging. In this talk we will explain what challenges we were confronted with and how we solved them, bridging security properties we require in the DC to be compatible with AWS Lambda.How is this challenging? Complications with AWS Lambda fall broadly into two categories. One, operational requirements for Lambda that make developers choose it in the first place, such as near immediate response time. Two, providing parity with our DC security infrastructure so developers can engineer in a fashion they are used to while observing security best practices. Simply reimplementing the systems we use in the DC was not possible, but using native cloud functionality alone wasn't possible either.How did we solve this? This talk will cover how we architected workload identity in AWS Lambda, sharing identity between DC and the cloud. We built a custom certificate issuance on top of AWS Private CA that is compatible with the SPIFFE standard, but issues certificates ahead of time to not block Lambda startup. To offer application secrets, we built a syncing system that integrates with Keywhiz while using cloud native functionality to accelerate secret access.Why is this secure? We architected in a way to use the strengths of both our DC and cloud native functionality. We will discuss pros and cons of multiple approaches we considered and explain why we picked the ones we ended up with.Was this successful? Our CA architecture has subsequently influenced a SPIRE proposal for serverless issuance which will mirror ours, becoming a standard for serverless SPIRE certificate issuance. Both our certificate issuance and Lambda secrets are in use for our production systems, including Square Financial Services, a bank that is a subsidiary of Square.