eBPF https://ebpf.foundation Linux Foundation Project Thu, 12 Mar 2026 20:34:42 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://ebpf.foundation/wp-content/uploads/sites/9/2022/06/cropped-favicon-1-150x150.png eBPF https://ebpf.foundation 32 32 eBPF Fellowship Update: Podcasts, Labs, and Community Building https://ebpf.foundation/ebpf-fellowship-update-podcasts-labs-and-community-building/ Wed, 11 Mar 2026 17:35:40 +0000 https://ebpf.foundation/?p=1777 This work is supported by the eBPF Foundation’s Community & Advocacy Fellowship Program, written by eBPF Community & Advocacy Fellow Teodor Janez Podobnik.

In October 2025, I was honored to receive the Inaugural Community & Advocacy Fellowship from the eBPF Foundation in recognition of my ongoing contributions to the eBPF community. 

As part of this work, I hosted three podcast conversations focused on different aspects of eBPF: 

Three video interview thumbnails featuring individuals seated indoors, each labeled with interview titles, dates, and interviewer information.I also released several hands-on eBPF Coding Labs on the Iximiuz Labs platform, enabling developers to learn and experiment with eBPF directly in the browser through practical, interactive exercises: 

Diagram comparing traditional Linux networking and eBPF-based XDP, illustrating packet flow, backend interaction, and differences in load balancing methods using boxes, arrows, and icons.

In addition, I developed and presented my personal eBPF project, Kayrndor: GitHub Action Runtime Security Agent, at the Backend Meetup in Slovenia. 

The fellowship’s financial support also made it possible for me to organize the first-ever eBPF Meetup in Slovenia, where engineers from across the industry gathered to explore both beginner and advanced eBPF topics, exchange experiences, and strengthen the local community. 

A speaker presents to an attentive audience, standing beside a projector screen displaying diagramsOverall, during this first part of the fellowship period, the fellowship’s financial support has enabled me to dedicate significantly more time to eBPF-related initiatives and community efforts.

]]>
Why eBPF Is Redefining How Software Runs Inside the Linux Kernel https://ebpf.foundation/why-ebpf-is-redefining-how-software-runs-inside-the-linux-kernel/ Thu, 26 Feb 2026 18:15:49 +0000 https://ebpf.foundation/?p=1770 Introducing the eBPF Meetup Program https://ebpf.foundation/introducing-the-ebpf-meetup-program/ Wed, 25 Feb 2026 19:04:04 +0000 https://ebpf.foundation/?p=1759 The eBPF ecosystem is growing, and so is the demand for strong local communities where developers can meet, share ideas, and create the next generations of eBPF projects, products, and companies.

The eBPF Foundation is excited to launch the eBPF Meetup Program, a new initiative designed to support and grow local eBPF communities around the world. This program empowers community members to organize local events focused on eBPF use cases and real-world production experience.

As eBPF continues to grow, knowledge sharing is becoming just as important as code, and local meetups play a critical role in that evolution.

How the Program Supports Organizers

The eBPF Meetup Program provides financial support to approved eBPF-focused events to help reduce the burden of organizing local gatherings. This support can be used for typical meetup related expenses such as food, venue costs, swag, and other essentials.

Events must be hosted on community.cncf.io or meetup.com and primarily focused on eBPF technology or use cases.

See the full reimbursement guidelines.

Building the next phase of the ecosystem

As the Foundation continues to invest in research, upstream development, and programs like grants and fellowships, the Meetup Program represents another step toward strengthening the global eBPF community.

If you’re using, developing, or just interested in eBPF, consider hosting a meetup and sharing your journey. If you’re new to the space, join a local event and start learning. If you want to meet other eBPF enthusiasts in your area, there are already meetups in San Francisco, Tokyo, and Vienna and you can always start your own!

We look forward to seeing more eBPF communities take shape around the world.

Questions? Reach out to [email protected].

]]>
eBPF report shows efficiency, security gains at scale https://ebpf.foundation/ebpf-report-shows-efficiency-security-gains-at-scale/ Fri, 13 Feb 2026 12:30:11 +0000 https://ebpf.foundation/?p=1752 The RISC architecture frontier: Is eBPF ready for ARM64 and RISC-V? https://ebpf.foundation/the-risc-architecture-frontier-is-ebpf-ready-for-arm64-and-risc-v/ Thu, 12 Feb 2026 08:39:08 +0000 https://ebpf.foundation/?p=1749 New “eBPF In Production” Report Showcases Production Enterprise Outcomes Across Networking, Security, and Observability https://ebpf.foundation/new-ebpf-in-production-report-showcases-production-enterprise-outcomes-across-networking-security-and-observability/ Thu, 12 Feb 2026 08:00:05 +0000 https://ebpf.foundation/?p=1740 TL;DR

New research from the eBPF Foundation highlights how enterprises are achieving measurable cost savings, performance gains, risk reduction, and operational efficiencies with eBPF in production. The eBPF In Production: An Overview of Compelling Enterprise Outcomes Using eBPF report showcases real-world deployments from organizations including Cloudflare, Netflix, ByteDance, and Rakuten, and demonstrates how eBPF has become a foundational technology for modern infrastructure. The report is available now as a free download.

FOR IMMEDIATE RELEASE

Latest research from the eBPF Foundation demonstrates how organizations are achieving measurable ROI with eBPF at global scale

SAN FRANCISCO – February 12, 2026 – The eBPF Foundation, which advances eBPF’s upstream development and community and promotes its secure adoption across platforms, today announced the availability of its newest research report, eBPF In Production.

Designed for executive and senior technical leaders, the report focuses on tangible business outcomes and their associated ROI. It consolidates real-world examples of organizations deploying eBPF in production and documents measurable improvements in performance, infrastructure efficiency, security posture, and operational scale.

As the report, authored by tech journalist Bill Doerrfeld, explains eBPF has evolved into a mature, production-grade infrastructure technology with clear benefits over existing solutions. Embedded within products, projects, and tools spanning networking, observability, runtime security, and application governance, eBPF enables organizations to implement advanced functionality in the Linux kernel without disruptive kernel rewrites, risks of crashing the system, or excessive overhead.

“eBPF has moved decisively from experimentation to enterprise standard,” said Bill Mulligan, eBPF Governing Board Member from Isovalent. “This report demonstrates that leading organizations are not just piloting eBPF, they are standardizing on it to reduce costs, mitigate risk, and improve system performance at massive scale.”

Key Insights From the Report

Drawing on documented case studies and public production benchmarks, the report highlights consistent patterns of success across industries:

Reduced Infrastructure Costs

Organizations are leveraging deep kernel-level visibility to optimize CPU, GPU, and network utilization:

  • Datadog reduced CPU usage by 35% using an eBPF-based connection tracker.
  • Meta reduced CPU cycles by up to 20% using its eBPF-driven Strobelight profiler.
  • Polar Signals cut cross-zone traffic costs by 50% with eBPF-based observability.

Increased Operational Efficiency and Scale

eBPF’s low-overhead execution model enables leaner, more scalable infrastructure:

  • LinkedIn reduced Kafka log volume by 70% using an eBPF observability agent.
  • Seznam.cz doubled throughput while reducing CPU usage by 72x with eBPF load balancing.
  • DoorDash achieved 40% less memory usage, 98% fewer restarts, and 80% faster deployments after migrating to eBPF-based monitoring.

Risk Mitigation and Attack Prevention

Kernel-level enforcement and telemetry enable faster, more precise security controls:

  • SentinelOne detects and stops ransomware attempts in under one second using eBPF-based architecture.
  • Cloudflare mitigated DDoS attacks peaking above 7 Tbps using eBPF-based XDP programs.
  • CoreTech mitigated a 1 Tbps DDoS attack without downtime using eBPF-powered scrubbing.

Improved Developer Velocity

By centralizing observability and policy enforcement at the kernel layer, organizations reduce instrumentation burden, troubleshooting complexity, and compliance friction, enabling faster innovation across platform and security teams.

Featured Enterprise Case Studies

The report takes a deep dive into four global organizations using eBPF as a strategic infrastructure layer:

  • Cloudflare, where eBPF underpins networking, observability, and terabit-scale DDoS mitigation.
  • Netflix, which uses eBPF for network defense, “noisy neighbor” detection, and large-scale telemetry collection supporting more than 325 million subscribers.
  • ByteDance, which improved throughput by 10% across infrastructure supporting approximately one million servers by adopting eBPF-based networking.
  • Rakuten Mobile, where eBPF powers anomaly detection, security enforcement, and observability within a cloud-native telecom environment.

Together, these case studies demonstrate that eBPF is no longer an isolated tool for networking or observability, it is becoming a modular foundation that unifies performance, security, and telemetry at the kernel level.

A Strategic Infrastructure Foundation

The report concludes that eBPF is now woven into the fabric of modern Linux-based systems and should be viewed as a strategic infrastructure layer rather than a niche technology. Organizations that treat eBPF as a unified foundation across networking, observability, and security are realizing compound benefits in efficiency, visibility, and risk reduction.

As steward of the eBPF ecosystem, the eBPF Foundation plays a central role in advancing upstream innovation, supporting collaboration across vendors and end users, and accelerating the adoption of secure, programmable infrastructure.

Download the Report

eBPF In Production is available to download for free from the eBPF Foundation.

eBPF Foundation Resources

About the eBPF Foundation

The eBPF Foundation was created to advance eBPF as an open, shared technology for programmable infrastructure. It brings together a cross-platform community of maintainers and organizations working upstream to evolve eBPF’s capabilities while ensuring its safety, security, and performance. Foundation members collaborate on common technical priorities, security best practices, community development, and promotional opportunities supporting eBPF across kernels, operating systems, and enterprise environments. Find further information here: https://www.ebpf.foundation 

]]>
The eBPF Foundation’s 2025 Year in Review https://ebpf.foundation/the-ebpf-foundations-2025-year-in-review/ Thu, 18 Dec 2025 18:56:45 +0000 https://ebpf.foundation/?p=1700 As 2025 comes to a close, the eBPF Foundation is proud to reflect on a year marked by deep technical progress, expanding global adoption, and growing collaboration across research, industry, and community spaces. What stood out this year wasn’t a single breakthrough feature, but a steady accumulation of work that makes eBPF something operators can rely on, from better testing to safer runtimes, stronger tooling, and a community that now spans kernel developers, cloud providers, researchers, and practitioners running this technology at serious scale.

That shift shows up clearly in this year’s State of eBPF report, which highlights how eBPF is becoming the strategic platform of choice for infrastructure teams. In response, the Foundation invested in areas of the ecosystem that required the most support. Notable efforts include upstream development that improves the developer experience, high-impact academic research, launching a new fellowship program and meetup program, and publishing real-world use cases from some of the world’s largest tech companies. These milestones highlight eBPF’s growing importance as a core enabler of secure, observable, and efficient cloud native infrastructure and how eBPF is becoming a permanent part of how modern infrastructure is built and operated.

Funding Upstream Development 

Key development priorities for the community can fall between the cracks of what different companies and individuals are focusing on. Testing infrastructure, cross-architecture parity, and long-term maintainability rarely map cleanly to product roadmaps, but without them, everything else becomes fragile. In 2025, the eBPF Foundation continued to focus on funding projects to fill these gaps, with an emphasis on improving the day-to-day experience of developing and running eBPF. Major milestones included:

  • Improving the eBPF tests in the kernel: This post covers test suite expansion and rewriting legacy BPF samples to improve CI alignment and catch regressions early. The work increased automated test coverage, which improves stability and reduces the risk of regressions as the eBPF subsystem continues evolving.
  • Bouncing on trampolines to run eBPF programs: We also supported improvements to how eBPF programs attach to kernel functions, prioritizing clean abstractions and CI coverage, which resulted in reduced overhead in performance when attaching many programs, more predictable behavior, and simplified maintenance of attach logic when kernel functions change.
  • The Dwarves Beneath the Kernel: Forging BTF for eBPF: BTF tooling improvements and strengthening BTF support for introspection and tooling automation resulted in improved BTF portability, reduced image size compared to DWARF, and laid the groundwork for better developer ergonomics and wider architecture support.

Work continues toward achieving feature parity between x86-64 and ARM64 architectures as eBPF adoption grows across cloud and edge devices.

Strengthening Upstream Security

We received a $228,200 grant from the Alpha-Omega Foundation this year to support critical upstream security work for eBPF. This work aims to harden the eBPF runtime and ensure long-term safety as adoption grows across architectures. The funding supports two major efforts:

  • Defensive runtime tooling: Enabling KASAN for JIT-compiled eBPF programs to validate memory accesses at runtime.
  • Security auditing: Reviews of x86-64, arm64, and riscv64 JIT compilers, including instruction encoding, register allocation, and immediate value handling, checking verifier to JIT integrity, and assessing unprivileged surfaces.

Advancing Safety and Efficiency with Academic Research

In 2025, the eBPF Foundation continues its commitment to academic collaboration by awarding $100,000 in grants to research teams focused on advancing eBPF’s safety and efficiency:

  • University of Michigan: The ePASS (eBPF Policy-Aware Secure Systems) project is an in-kernel LLVM-like compiler framework that extends static verification in eBPF by integrating a verifier-cooperative runtime enforcement approach. The goals of ePass are to improve programmability without compromising safety, and to enhance safety beyond what static verification alone can provide.
  • University of California, Riverside: eBPF Governors is a framework for dynamic, low-latency, QoS-aware power management in Linux data centers. The project works to address the growing energy demands of AI and cloud based workloads, with the goal of providing an extensible framework for power management governors across power states and processing units by leveraging eBPF.

This year’s grants build on the success of the 2024 research, and we are looking at the road ahead for more opportunities to support the growing academic research ecosystem around eBPF.

Real World Success Stories With eBPF

Starting this year, the eBPF Foundation began publishing detailed case studies from companies using eBPF in production. Their stories show how eBPF delivers measurable impact across networking, observability, profiling, and security:

Whether it’s optimizing load balancing or cutting cloud costs, these use cases highlight what’s possible with eBPF today, and the performance benefits in production.

eBPF for the Infrastructure Platform Whitepaper

eBPF for the Infrastructure Platform explores how eBPF has become the foundation for modern infrastructure teams, powering everything from L7 networking and runtime security to LLM observability and FinOps. It highlights how eBPF is becoming the strategic platform of choice for infrastructure teams.

It provides companies with guidance on how to implement eBPF in their infrastructure, product categories that will be affected by eBPF, and gives an outlook towards the changes that will ease developer adoption and shape its future use cases. 

You can download the full report to learn how companies like Netflix and Cloudflare are using eBPF today, and what’s next for programmable infrastructure.

Global Community Support

Conference Sponsorships

The eBPF Foundation was a platinum sponsor of both the Linux Plumbers Conference (LPC) and Linux Storage, Filesystem, Memory Management & BPF Summit (LSFMM+BPF) this year. The community came together at LPC in a dedicated eBPF track and at LSFMM+BPF with a BPF track, bringing speakers from our global community to share their use cases and new ideas.

Introducing the Community & Advocacy Fellowship

We recently launched the eBPF Community & Advocacy Fellowship, designed to recognize and support individuals who help to grow the eBPF ecosystem for an increasingly global and diverse set of users, contributors, and end users. The inaugural fellowship recipients, Teodor Janez Podobnik and Yusheng Zheng, are engaging developers, translating content, and helping newcomers find their footing in the eBPF ecosystem.

As we move toward 2026, we are excited to expand upon the program and build the next generation of technical leaders and community builders.

Supporting eBPF Meetups

The eBPF Foundation is investing in the global community with the announcement of a new eBPF Meetup Program. The program offers an opportunity to grow the eBPF community in your area by providing financial support to community members organizing local meetups.

Looking Ahead

As we look forward to the coming year, we embrace opportunities to expand upon our current efforts of community building and education. We will continue to publish findings from new research, support upstream development and architecture parity, and build and grow programs like the new Fellowship and Meetup programs.

As eBPF becomes a permanent fixture in modern infrastructure, the ecosystem is moving from experimentation to stewardship. All of the work that the Foundation is doing makes a clearer path from idea to production without losing the flexibility that made eBPF compelling in the first place. Additionally, programs like the Fellowship and Meetup initiatives are helping ensure that knowledge and community, not just code, continue to grow.

Now it’s your turn to help move the eBPF ecosystem forward. If you’re using eBPF in production, submit a case study to be featured on our website and share your experiences with others. If you are new to the community, join a meetup around you or check out the resources on ebpf.foundation and ebpf.io to learn more.

Finally, consider becoming a member to help support and shape the future of the technology that is defining the next generation of infrastructure platforms.

]]>
eBPF research shows kernel tech powering modern AI https://ebpf.foundation/ebpf-research-shows-kernel-tech-powering-modern-ai/ Tue, 16 Dec 2025 10:20:11 +0000 https://ebpf.foundation/?p=1695 Research Update: Managing Server Power with eBPF https://ebpf.foundation/research-update-managing-server-power-with-ebpf/ Thu, 11 Dec 2025 07:00:24 +0000 https://ebpf.foundation/?p=1645 This work is supported by the eBPF Foundation Academic Research Grants.

1. Project Overview

With the explosive growth in cloud and ML/AI data centers, energy consumption is now a first-order design constraint. However, existing Linux power management subsystems have not evolved to handle the demands of modern data centers. They face many challenges, such as (1) slow and uncoordinated power management decisions, (2) growing difficulty in meeting workload performance (Quality-of-service) targets, and (3) increasingly volatile data center workloads. 

To this end, our project proposes eBPF Governor, an eBPF-based power management framework to deliver highly responsive, low-overhead, QoS-aware dynamic power management. This project enables the creation of custom power management governors in eBPF to control (and coordinate) various processor low-power states, such as frequency states and idle sleep states. 

This post serves as the first installment in our series to provide an overview of Linux power management subsystems, their limitations, and how eBPF can modernize the existing power management subsystems. In subsequent blogs, we will discuss how eBPF Governors can be made generalizable, extensible, and coordinated across the various Linux power management subsystems that are normally siloed. We aim to demonstrate that eBPF can modernize the Linux power management subsystems for better power savings and meet the demands of modern data centers. 

2. Linux Power Management – An Overview

In Linux, CPU power management is handled by two main components: CPU frequency scaling by the CPUFreq (P-states) subsystem and CPU idle sleep states by the CPUIdle (C-state) subsystem. These subsystems consist of two main components that handle performance scaling and idle state selection: drivers that interface with the hardware and governors that manage the states. The following figure illustrates the existing Linux power management subsystems.

Linux power management overview

Figure 1. Overview of Linux power management subsystem.

CPU power management is interfaced with the Linux kernel through drivers (e.g. intel_pstate for frequency scaling and intel_idle for idle sleep states), which directly interacts with the processor through model-specific registers (MSRs) to set the hardware’s frequency and voltage states or idle sleep states. For example, we can adjust processor power performance states (P-states) by writing specific frequency values to model-specific registers (MSRs), e.g., setting MSR_IA32_PERF_CTL to 2.4 GHz.

Governors implement algorithms to select the processor’s power performance or idle sleep states. Common governors include the menu governor (which predicts idle duration) for idle sleep state selection, schedutil (which uses CPU utilization data available from the CPU scheduler), ondemand (which uses current CPU load), and powersave (which uses the lowest frequency) for frequency scaling selection. Since these governors reside completely in the driver, they make use of the limited heuristics available in the kernel, such as the Linux scheduler, to make power management decisions. 

2.1 QoS-aware Challenges

While these scaling governors can achieve modest power savings, they are not aware of application-level QoS because their only performance feedbacks are kernel-visible load and utilization, which are not necessarily strong indicators of application request latency. For example, the assumption of a linear relationship between CPU load and required frequency does not hold true for all workloads, leading to suboptimal performance or unnecessary energy consumption. Furthermore, many data center workloads are latency-critical and have strict timing requirements. Without direct QoS feedback from applications, scaling governors are not aware of the latency impact of P-state selection decisions. 

To enable QoS-aware dynamic power management, existing best practices move the P-state selection algorithms out of kernel space and into userspace through the userspace scaling governor. This allows power management to consider application-specific needs, such as QoS, which kernel-space governors overlook. The userspace governor directly controls processor P-states by writing to a sysfs file, which can lead to high syscall overheads, which can further limit the responsiveness of power management decisions to workload fluctuations, thereby missing crucial opportunities for power savings.

Furthermore, enabling application-level QoS feedback often adds significant complexity to system design. As a result, the entire feedback loop, from capturing application performance to communicating metrics and triggering a power state adjustment, can introduce enough delay to make a timely response impossible, especially for fast-changing or bursty workloads. This motivates the need for new approaches that can provide both fine-grained QoS awareness and minimal overhead with high responsiveness.

2.2 Need for Responsiveness

While in-kernel CPUFreq governors offer better responsiveness than userspace governors, they are not designed to incorporate application feedback efficiently. Thus, there is an essential tradeoff between QoS-awareness and power management responsiveness. This landscape reveals a clear gap: no existing approach provides both the fast, low-overhead control of hardware-based power management solutions and the fine-grained, QoS-aware decision-making required by modern workloads. Bridging this gap calls for a new paradigm in dynamic power management, one that combines rapid response with application-level insight, without introducing additional overhead.

A closer look at the latency of different power management actions highlights the magnitude of this overhead. We observed that setting frequency directly from userspace takes, on average, 31 milliseconds. This number represents only the time to execute the setting operation itself and does not measure the full time until the change in frequency is observed by the hardware. As a result, existing CPUFreq techniques cannot sustain power savings with sub-50ms volatility.

Clearly, userspace-based power managers are fundamentally limited in their ability to respond quickly to changing workload conditions.  By comparison, we found that updating frequency from kernel space through eBPF programs is two orders of magnitude faster, requiring only ~200 microseconds.

There is a clear opportunity to leverage the flexibility and low-latency capabilities of eBPF to enable fast, QoS-aware power management in kernel space. Beyond solving the immediate shortcomings of current frameworks, this design also opens up new opportunities for future innovation in power and resource management.

2.3 Extensibility – Looking ahead

Extending for power management coordination: Even if frequency updates can be significantly reduced in legacy systems (i.e. by directly writing to MSRs), the existing Linux power management subsystem remains siloed, each making its own independent power management policies for frequency scaling, idle sleep states, and accelerator (such as GPU) power states. Our prior research findings have identified that short, volatile workloads (from functions/microservices or GPU kernels with short service time) can incur significant frequency state change overhead and that idle periods tend to become fragmented, which prevents idle state entry. Therefore, there exist significant power-saving opportunities when coordinating policies for idle sleep state and GPU states.

Extending beyond CPUs: Currently, the Linux power management subsystem only manages the CPU’s power states. Modern data center servers consist of significant GPU resources, or more generically, accelerator resources, such as TPUs. These accelerators can account for greater than 50% of server power consumption, with CPUs only consuming ~10%. GPU power management is currently handled independently by the hardware and can be guided by system management interfaces (e.g., nvidia-smi) by way of the GPU driver. With the significant rise of non-CPU power, power management of such components should become first-class citizens within the Linux kernel.  

3. The Internals of eBPF Power Management

We now present a quick overview of how eBPF Governors can (1) enable custom power management governor policies, (2) provide low-overhead QoS feedback, and (3) interface with power management drivers. To enable custom power management policies, instead of utilizing a userspace governor, we offload custom power management policies into an eBPF program. This enables extensible and customizable policies that we can adapt to different workload requirements and hardware configurations. As of this writing, we have successfully implemented control of frequency states through eBPF and will continue to extend eBPF Governors to target idle sleep states as the project progresses.

The internals of eBPF power managementFigure 2: eBPF Governor enables custom eBPF programs to guide processor power management.

3.1 eBPF Program Considerations

The eBPF environment lacks support for floating-point arithmetic due to a restricted instruction set. We address this constraint by modifying latency handling and power state selection calculations to avoid floating-point operations. Specifically, we process latencies reported at microsecond granularity, preserving precision without requiring floating-point representation. Power state selection is then calculated using fixed-point arithmetic, where we scale the numerator by a constant (e.g., 1000) and adjust accordingly. This approach ensures precise slack-based decisions without floating-point operations, aligning with eBPF’s computational constraints while maintaining necessary precision.

In multi-core systems, applications may be assigned to multiple CPU cores, each potentially generating concurrent QoS feedback. Because each core independently evaluates QoS and adjusts the performance preference hints for the application’s assigned cores, conflicting decisions may arise. For example, one core might detect a latency violation and boost performance, while another, observing acceptable latency, lowers it to save power. This concurrency introduces race conditions, resulting in inconsistent preference hint settings across the application’s cores, undermining coordinated power management. To ensure safe, coordinated updates to hardware registers across multiple cores, we use the bpf_spin_lock primitive provided by eBPF. This spin lock guarantees that only one core at a time can modify the performance preference hints stored in the model-specific registers, preventing race conditions and inconsistent settings. 

3.2 Providing QoS-awareness to eBPF Governor

To provide low-overhead QoS feedback to the eBPF governor, we utilize eBPF’s asynchronous buffers, a shared memory mechanism between user space and kernel space that allows applications to write data directly to a memory region accessible by eBPF programs. 

We utilize a BPF ring buffer of type BPF_MAP_TYPE_USER_RINGBUF. This enables applications to write QoS feedback data directly to a memory region accessible by the eBPF program, enabling efficient and low-overhead data exchange. On the eBPF program side, the eBPF program reads the performance metrics from the ring buffer, processes them, and adjusts power states accordingly. This ring buffer lets userspace and eBPF share data asynchronously, providing fast communication for QoS feedback.

3.3 Hooking into Power Management Drivers with kfuncs

Kfuncs are our main mechanism for integrating eBPF Governors with the existing kernel power management drivers. A kfunc is a kernel function that is explicitly marked as callable from eBPF. By marking specific kernel functions as callable from eBPF, we enable eBPF Governors to hook into the same paths that traditional governors use. The eBPF code can then implement QoS-aware policies that respond quickly to changes in workload latency and utilization, without paying the overhead of user space round trips. This makes it possible to react to QoS changes at microsecond time scales, instead of being limited by sysfs polling and context switches.

From the perspective of the hardware, all of this complexity collapses into a few MSR writes. Once the kernel has decided on a target P-state or C-state, it encodes that choice into an MSR and writes it. Because this is the universal endpoint of existing governors, we expose those MSRs through kfuncs. The eBPF Governor computes the desired state and then calls an MSR kfunc to commit it. Specifically, we employ rdmsrl_on_cpu to read current settings from Model Specific Registers (MSRs) and wrmsrl_on_cpu to write updated configurations. These functions facilitate efficient, per-CPU adjustments of HWP preference hints. By restricting access to rdmsrl_on_cpu and wrmsrl_on_cpu via wrappers that expose only necessary functionalities, we ensure safe, controlled operations, avoiding interference with unrelated system components.

3.4 Generalizability challenge

While we have demonstrated the feasibility of power management control from eBPF programs, there still exist many challenges towards generalizing eBPF control for generic power management. First, there exist vendor-specific MSRs and features that deviate between processors. Thus, to effectively manage the power of specific CPUs requires us to dive into the power management driver code to identify the proper MSR register and expected data format that needs to be written to it. Secondly, modern processors exhibit various power management modes (active vs. passive mode) and abstracted performance states that may not necessarily map to specific frequency states. For example, while the processor may run at various frequency levels, only a subset of these frequency levels are exposed to CPUFreq drivers, or abstracted only as bias hints. Thus, existing power management drivers perform extensive mapping of performance states to raw frequency states. When building eBPF Governor policies, we must be cognizant of the mode of operation and performance/frequency states that the processor exposes. If we want to realize the goal of a generalizable and extensible power management governor (à la sched_ext for Linux scheduler), there exist many outstanding challenges that need to be addressed. 

4. Summary

The ultimate goal of eBPF Governors is to provide an extensible framework for power management governors across power states (such as frequency states and idle sleep states) and processing units (such as CPUs and GPUs). There are many technical challenges towards demonstrating the feasibility of controlling and coordinating each of these power states and processing units through eBPF facilities. By pushing the boundaries of what eBPF can do with power management, we hope to inspire more use cases of eBPF server resource management in general to make data center servers more efficient. 

]]>
ePass: Verifier-Cooperative Runtime Enforcement for eBPF https://ebpf.foundation/epass-verifier-cooperative-runtime-enforcement-for-ebpf/ Thu, 04 Dec 2025 19:42:54 +0000 https://ebpf.foundation/?p=1648 Contributed by Ryan Huang, University of Michigan.

This work is supported by the eBPF Foundation Academic Research Grants.

eBPF has grown into one of the most exciting technologies in modern systems, providing a safe and expressive way to dynamically extend the Linux kernel. It now powers observability, networking, tracing, profiling, security, and more. As the eBPF ecosystem expands its use cases, developers increasingly run into a tension: the verifier, the component that checks a given eBPF program abides by safety rules before running the program, can sometimes be too restrictive; paradoxically, the verifier can also miss certain unsafe patterns. 

Our team has been developing ePass, a new framework that helps address this tension by introducing verifier-cooperative runtime enforcement. ePass improves programmability without compromising safety, and enhances safety beyond what static verification alone can provide. This blog post provides an overview of ePass, its current status, and future roadmap. 

When Static Verification Isn’t Enough

The verifier is a central component of the eBPF ecosystem. It uses static analysis to inspect eBPF bytecode at load time and decides whether it is safe to run. To stay sound, it must over-approximate possible program behaviors. In practice, that means when it cannot conclusively prove something is safe on all paths, it rejects the program even when the actual runtime behavior would be fine. 

As a result, developers can run into the problem of false rejections. Typical examples include programs whose loops have data-dependent bounds that the verifier cannot track precisely, or programs that index into arrays or maps using values derived through helper calls or bit-masking. In these situations, users often know their program logic is safe, but they must contort the code to convince the verifier: splitting logic across multiple tail calls, adding manual masks, or cloning nearly identical code paths. These workarounds add complexity, hurt readability, and sometimes introduce new bugs of their own. 

At the same time, static checking alone cannot capture every nuance of runtime behavior. The verifier operates on bytecode and abstract register states; it cannot see, for example, which exact stack bytes were initialized, or every subtle way misclassified helper arguments might interact with kernel code. Real CVEs have arisen from these gaps, including uninitialized stack reads, pointer leaks, and cases where invalid arguments to helpers slipped past the verifier but caused exploitable behavior during execution.

Efforts have been made to improve the verifier’s analysis, which helped alleviate some of these issues. However, the core problem remains. The verifier itself has also become difficult to evolve. Adding new helpers, map types, or program types often implies more special cases and more intricate state tracking. There is a growing sense that static verification alone cannot scale to meet both programmability and safety demands while keeping the complexity manageable. 

The ePass Vision: Verifier-Cooperative Runtime Enforcement

 

overview

Figure 1. Overview of ePass

ePass is built around a simple insight: some safety properties are fundamentally hard to prove statically but are reasonable to enforce at runtime. Instead of making the verifier ever more complicated, we can keep it as the core root of trust and augment it with structured runtime enforcement. 

To do this, ePass extends the eBPF loading pipeline. It first introduces an SSA-based Intermediate Representation (IR). This IR is designed specifically for eBPF: it has virtual registers, explicit control-flow graphs, and annotations that preserve the connection back to the original bytecode and the verifier’s analysis. When a program is loaded, ePass lifts the bytecode into the IR. Once in IR form, ePass runs a series of transformation passes that can insert runtime checks, repair verifier-triggered issues, or optimize code. After these passes run, ePass compiles the IR back to eBPF bytecode and sends it through the verifier again before JIT compilation.

Categories API Description
Instruction Manipulation ir_insn *epass_create_ja_insn(epass_env *env, ir_insn *pos_insn, ir_bb *to_bb, enum insert_position pos) Create a direct jump (JA) instruction
jumping to to_bb
before or after (defined by pos)
pos_insn.
void epass_ir_remove_insn(epass_env *env, ir_insn *insn) Remove an instruction from IR.
BB Manipulation ir_bb *epass_ir_split_bb(epass_env *env, ir_function *fun, ir_insn *insn, enum insert_position insert_pos) Split the basic block into two basic blocks at a given position.
Verifier Integration ir_insn *epass_ir_find_ir_insn_by_rawpos(ir_function *fun, u32 rawpos) Get the IR instruction that the bytecode at rawpos corresponds to.
vi_entry *get_vi_entry(epass_env *env, u32 idx) Get the VI map (Section Supervisor) entry from the instruction index idx.
Miscellaneous array epass_ir_get_operands(epass_env *env, ir_insn *insn) Get all the operands of an instruction.
void epass_ir_replace_all_uses(epass_env *env, ir_insn *insn, ir_value rep) Replace all the uses of a VR with another value.

Table 1. Example of ePass IR APIs

A key design decision is that ePass always operates inside the verifier pipeline. It does not bypass verification or silently sneak in unverified code. Every transformed program is re-verified. The verifier remains the central trusted component, while the checks inserted by ePass are treated like any other eBPF instructions and must comply with all existing safety rules. 

To make this practical, ePass includes two main components. The “core” is a compact compiler framework that implements the IR, the lifter, the code generator, and a library of reusable transformation primitives. The “supervisor” sits in the kernel alongside the verifier and coordinates when and how ePass runs. It also enforces policies, such as system-wide limits on instruction counts or which passes may be enabled by regular users versus administrators.

core design

 Figure 2. Architecture of ePass core. The green arrow means that the IR is passed through the IR checker. 

Current Status

Although the project is still evolving, our current prototype demonstrates that this verifier cooperative approach is feasible and useful. We have implemented a dozen passes, covering three main themes: repairing verifier false rejections, strengthening runtime safety, and optimizing code. 

One example is the instruction counter pass. Many developers have encountered confusing “program too large” or instruction-limit errors even when they know their loops will terminate quickly at runtime. ePass tackles this by moving part of that responsibility to runtime. The counter pass instruments programs with an instruction counter that is incremented at carefully chosen locations. If the total executed instructions exceed a configured limit, the program is safely terminated. This approach allows us to keep a global budget on execution without forcing the verifier to conservatively assume worst-case iteration counts for every path. In tests, this pass enabled real world programs rejected by the verifier to execute safely without exceeding actual limits and with negligible runtime overhead.

sample transform

Figure 3. A sample workflow of running the counter pass. The code is simplified for readability. ❶ is the input eBPF bytecode, ❷ is the lifted E PASS IR, ❸ is the transformed IR after running the pass, and ❹ is the compiled eBPF bytecode. Highlighted lines are the instructions added by the pass.

Another example is the MSan pass for uninitialized stack handling. The verifier historically tracks only how deep the stack is used, not whether every byte has been written before being read. ePass introduces a shadow memory region to track initialization status. When the program writes to the stack, the corresponding shadow bytes are marked initialized; when it reads, ePass checks the shadow state and terminates the program if it accesses uninitialized memory. This directly addresses a class of bugs and vulnerabilities where programs, previously accepted by the verifier, used uninitialized stack contents at runtime.

ePass also tackles helper-related exploits through a helper validation pass. The verifier’s range analysis is used to infer what arguments to helpers should look like. ePass then inserts runtime guards before those helper calls to ensure the values stay within the inferred safe ranges. If a malicious or buggy program later manages to steer arguments outside those bounds, the runtime check will detect and stop it before the helper executes in an unsafe way.

Beyond these, ePass includes passes that insert index-masking logic for dynamic array or map access, passes that simplify and canonicalize stack usage, and classic compiler transformations like dead-code elimination and compactness optimizations that merge redundant operations.

To understand how well ePass works in practice, we evaluated it on several aspects: programmability, safety, correctness, and performance. For programmability, we collect 23 real world eBPF programs that had been rejected by the verifier despite being semantically safe. These came from GitHub, Stack Overflow, and open source projects such as Cilium and Falco. With ePass enabled and appropriate passes configured, 21 of the 23 programs are successfully transformed, re verified, and allowed to run. The remaining two cases involve limitations of the current prototype across tail calls. In both cases, we verify that ePass does not compromise correctness by accepting unsafe variants.

For safety, we evaluate both fuzzing and real vulnerabilities. Using an eBPF fuzzing toolchain, we generated thousands of programs that attempted out-of-bounds accesses, uninitialized reads, or invalid pointer manipulations. Programs that were correctly rejected by the verifier stayed rejected under ePass, while programs that passed verification but would misbehave at runtime were caught by the inserted checks. We then analyzed recent eBPF-related CVEs and found that 14 of them fall into categories that our existing passes can mitigate.

Performance is often a concern when adding any runtime checks, particularly for performance critical environments like XDP. We therefore measured both throughput and latency for representative XDP programs, with and without ePass passes enabled.

throughput latency

Figure 4. Throughput and latency of four XDP programs. Blue (left) is without ePass, orange bar (middle) is with counter green bar (right) is with MSan pass.

Even with heavier instrumentation like MSan, the impact is under 2% on average for throughput and latency. Lighter passes such as the instruction counter have even smaller overhead. For regular system workloads, we run Falco in the background with ePass and benchmark the system using lmbench and Postmark. The overheads are negligible.

From a correctness perspective, we validated ePass on 504 human-written programs drawn from real projects, as well as 10,000 fuzzer generated programs. For all the tests where programs were valid, the transformed versions behave identically to the originals: same verifier decisions, same outputs, and no crash or divergence introduced by ePass itself.

Roadmap 

The current results show that our approach is promising. There is much more we plan to do. The next phase of the project focuses on turning ePass into a robust and highly usable framework. In the near term, we are working to extend and refine the ePass IR, stabilize the kernel integration, and increase compatibility with recent kernels. One high-priority task is to strengthen the testing framework so that bugs in the passes or ePass components can be caught early. We will also make the APIs easier to use. We aim to only require tens of lines of code for developing a typical new pass. In parallel, we will look for more compelling use cases and grow the library of transformation passes to cover diverse scenarios.

We are also thinking about introducing a policy framework for administrators. Rather than enabling passes one by one, operators will be able to select profiles that each map to a curated set of passes. This will make it easier for distributions, cloud providers, and platform teams to adopt ePass in ways that match their performance and feature requirements.

Finally, we will improve the ePass documentation, add examples and tutorials, share technical papers, present in conferences, and engage with the community for feedback. Our long-term goal is for ePass to integrate smoothly with the evolving eBPF ecosystem.

Get Involved

ePass is open source and available at:

https://github.com/OrderLab/ePass

We invite eBPF users, kernel developers, and platform builders to try it out, report issues, and share real programs that are incorrectly rejected by the verifier, use workarounds due to programmability restrictions, or expose tricky unsafe behavior. Those real-world examples are the best way for us to shape ePass into something that is truly useful to the community. 

With the support from the eBPF Foundation, we are excited to keep pushing in this direction. We hope that verifier-cooperative runtime enforcement becomes a core part of the eBPF ecosystem: unlocking more expressive programs, tightening safety guarantees, and keeping kernel extensibility both powerful and trustworthy.

]]>