GitHub itself relies on github.com to host its source code, creating a circular dependency: if github.com goes down, developers can't access the code needed to fix it. This is just one example of a broader challenge in deployment safety. To tackle such issues, GitHub turned to eBPF, a powerful Linux kernel technology that can monitor and block specific system calls during deployment. This article explores the problem of circular dependencies in deployment scripts and how eBPF provides a fine-grained solution.
What is the main circular dependency GitHub faces in its deployment process?
GitHub hosts all its source code on github.com itself. This creates a simple circular dependency: to deploy updates to GitHub, engineers need access to the code on github.com. If the site experiences an outage, they lose access to the source code required to fix it. Historically, GitHub mitigated this by maintaining a mirror of the code for emergency fixes and pre-built assets for rollbacks. However, deeper circular dependencies exist within deployment scripts—these can cause failures even when the primary code repository is accessible. For instance, a deployment script might attempt to download a binary from GitHub, or an internal service might check for updates on the platform. If GitHub is partially unavailable, such calls can hang or fail, blocking the entire deployment. This issue motivated GitHub to explore runtime monitoring and blocking of network calls using eBPF.
What are the three types of circular dependencies identified by GitHub?
GitHub categorizes circular dependencies into three types: direct, hidden, and transient. A direct dependency occurs when a deployment script itself attempts to fetch a resource from GitHub—for example, pulling the latest release of an open-source tool. If GitHub is down, the script fails outright. A hidden dependency involves a tool already present on the machine that, when run, secretly checks for updates from GitHub. If the check times out or fails, the tool may hang or error. Finally, a transient dependency arises when a deployment script calls an internal API (like a migrations service), which in turn tries to fetch something from GitHub. The failure propagates back, halting the script. All three types can derail a deployment during an outage. GitHub needed a way to detect and block these calls without modifying each script or tool.
How did GitHub traditionally handle these circular dependencies before eBPF?
Before adopting eBPF, the responsibility fell on each team that owned stateful hosts to manually review their deployment scripts and identify potential circular dependencies. This approach was error-prone and labor-intensive. Engineers had to trace every external call made by the script, including those made by helper tools or services invoked. Moreover, dependencies could change over time as tools added new features like automatic update checks. The review process had to be repeated for each deployment pipeline, and there was no automated way to enforce that a script would not inadvertently contact GitHub or other internal services during an outage. GitHub recognized that a runtime enforcement mechanism would be far more reliable. They needed something that could observe system calls in real time and block specific network requests, without requiring changes to the scripts or applications themselves.
How does eBPF enable GitHub to deploy more safely?
eBPF (extended Berkeley Packet Filter) allows GitHub to write tiny programs that run inside the Linux kernel, attached to system call hooks. During deployment, these eBPF programs listen for specific network calls—like attempts to connect to external hosts or to download files from GitHub. If a deployment script or any tool it invokes tries to make such a call, the eBPF program can block it or return an error, preventing the circular dependency from causing a hang or failure. GitHub designed their eBPF programs to be selective: they only block calls that would create a dependency on GitHub or internal services during a critical deployment window. This approach gives them fine-grained control with minimal performance overhead. Because eBPF runs in kernel space, it can intercept calls before they happen, making it an ideal tool for enforcing deployment safety without modifying application code.
Can you give an example of a hidden dependency that eBPF helps catch?
Consider a MySQL deployment script that uses a local servicing tool to apply configuration changes. That tool might have been written with an automatic update feature: every time it runs, it contacts an update server (or even GitHub) to check for a newer version. During normal operations, this check is harmless and may even be helpful. But during an outage—say, a MySQL failure that takes github.com partially offline—the tool’s update check could time out or fail, causing the tool to hang. The deployment script then stalls, waiting for the tool to finish. With eBPF, GitHub can intercept the network call made by the servicing tool and block it entirely, returning a synthetic response (like “no update available”) or just failing the call immediately. This prevents the deployment from being blocked by a dependency that the script author may not have even known existed. eBPF makes hidden dependencies visible and manageable at the kernel level.
Why did GitHub choose eBPF over other approaches for deployment safety?
GitHub evaluated several alternatives before settling on eBPF. One option was to use a proxy or a container isolation strategy, but these would require modifying every deployment script or tool, adding complexity and overhead. Another approach was to use static analysis to detect calls before deployment, but that couldn't catch runtime behaviors like hidden update checks or transient API calls. eBPF stood out because it operates at the system call level, allowing GitHub to monitor and block calls without any changes to the application code. It provides low-latency, high-performance filtering directly in the kernel. Moreover, eBPF programs can be loaded and unloaded dynamically, making them ideal for deployment windows where enforcement is needed temporarily. The ability to write custom logic in C and compile it to bytecode gave GitHub the flexibility to precisely target the calls they wanted to block—such as DNS lookups for specific domains—while leaving everything else untouched.