Conference:  Black Hat Asia 2023

Authors: Simon Scannell, Valentina Palmiotti, Juan José López Jaimez

2023-05-11

Extended Berkeley Packet Filter (eBPF) is a technology that provides capabilities to programmers seeking to make use of kernel layer performance and functionality. Fundamentally, eBPF allows users to load programs into kernel space and attach them to hook points. This allows for loading kernel code at runtime without needing to modify the kernel source code itself or develop a kernel module. eBPF programs are written in a high-level language and then compiled into assembly-like bytecode. At load time, the bytecode is JIT-compiled into the native architecture which allows for the program to be kernel and architecture-independent. The instruction set is minimal but allows programmers to call outside kernel functions, read and store data in various data structures and perform pointer arithmetic and operations.Programs that run in the kernel must be carefully analyzed to ensure that these programs follow rules to guarantee the integrity and security of the kernel running the program. A single code flaw in any of the components involved in program parsing, analysis, optimization, and compilation may lead to a compromise of the kernel running an eBPF implementation.As eBPF becomes more prevalent, the goal of our talk is to share the history of eBPF vulnerabilities, bug classes, mitigations and provide an outlook for the future. We will also share our insights into automated vulnerability discovery. We will introduce listeners to advanced concepts of structured fuzzing such as designing and implementing an Intermediate Language. We will also discuss identifying roadblocks such as bug detection and give practical examples of how to overcome them. This will enable anyone to apply these concepts to their own fuzzing campaigns. The source code of our fuzzer will also be made available.

Conference:  KubeCon + CloudNativeCon North America 2022

Authors: Shane Corbett, Wil Reed

2022-10-26

Lessons learned from misadventures in running a large-scale multi-tenant Kubernetes cluster in production

• Misapplying Kubernetes concepts to Linux performance rules is a big mistake
• Thinking in cores can be dangerous, as Linux thinks in time
• Configuring cores actually converts into time
• Properly scaling on the right metric can greatly simplify cluster setup and reduce churn
• Measuring what's going on is necessary to understand best practices for a cluster
• Prometheus is a good tool for measuring cluster performance

Conference:  KubeCon + CloudNativeCon North America 2021

Authors: Martynas Pumputis, Aditi Ghag

2021-10-15

Debugging Kubernetes networking issues with eBPF

• Debugging Kubernetes networking can be difficult due to the complexity of Linux kernel networking
• Traditional tools like tcpdump and logging-based methods are not sufficient for debugging
• eBPF can be used to efficiently troubleshoot K8s networking issues
• Packet inspection across layer 2, layer 3 and policy routing, socket, and so on, regardless of the CNI
• Real-life examples of K8s networking problems and how they were debugged with eBPF