https://www.ebpf.top/en/post/ Recent content in Posts on Head First eBPF Hugo -- gohugo.io en Tue, 27 Oct 2026 11:15:27 +0000 https://www.ebpf.top/en/post/bpf_capabilities_debug/ Mon, 16 Sep 2024 10:02:10 +0800 https://www.ebpf.top/en/post/bpf_capabilities_debug/ This article is available at: https://www.ebpf.top/post/bpf_capabilities_debug Author: kira skyler Introduction Unleash the Power of eBPF to Track Capability Changes Introduction In the Linux operating system, “capabilities” are a permission mechanism used for all privileges in the Linux system, finely divided into multiple independent permission bits. This way, users or processes can be granted only the specific permissions needed to perform specific tasks, without requiring all permissions. In the Linux capabilities system, permission assignments are divided into different sets, such as the Inheritable set, Permitted set, Effective set, Bounding set, and Ambient set. Each set controls the permissions of processes or threads in different scenarios. These capabilities may change under different circumstances, such as switching users, where a new user may likely have a different set of capabilities, and these sets may change according to different rules when creating child processes or executing new programs. https://www.ebpf.top/en/post/bpf_sched_ext_dive_into/ Fri, 30 Aug 2024 19:56:59 +0800 https://www.ebpf.top/en/post/bpf_sched_ext_dive_into/ This article can be found at: https://www.ebpf.top/post/bpf_sched_ext_dive_into Linux Process Scheduler CFS Scheduler EEVDF Scheduler Implementation Mechanism of BPF Scheduler Extender sched_ext Addition 1: SCHED_EXT Scheduling Class Addition 2: eBPF Custom Scheduler Functions Workflow of SCHED_EXT Scheduling Class Scheduling Cycle Workflow Switching to sched_ext Summary In the article Linus Strongly Pushes for Inclusion: BPF Empowers Scheduler Success, we reviewed the journey of BPF in the scheduler integration process within the community. Patch V7 is prepared for merging into 6.11, and subsequently, the code repository has also changed to kernel git address. It is only a matter of time for the merge to happen. This blog post will focus on the implementation principles of sched_ext. sched_ext is an extensible scheduler class jointly introduced by Meta and Google, referred to as ext_sched_class or sched_ext. This mechanism allows users to optimize scheduler strategies for specific workloads or scenarios by implementing scheduling classes through defined BPF programs. https://www.ebpf.top/en/post/bpf_sched_ext/ Sun, 30 Jun 2024 19:56:59 +0800 https://www.ebpf.top/en/post/bpf_sched_ext/ Read more at: https://www.ebpf.top/post/bpf_sched_ext 1. The Emergence of Pluggable Scheduler [2004] In 2004, Con Kolivas from the Linux community proposed the idea of a pluggable scheduler, envisioning multiple schedulers in the kernel that users could choose during boot. The principle behind the patch submission involved splitting a significant amount of code into a common part in kernel/sched.c and a private part. It also included pointers in the scheduler.c file that directed functions handling scheduling tasks, which were invoked for various process events (fork(), exit(), etc.), to gather scheduling-related information. Implementing a new scheduler simply required writing replacement functions and integrating them. However, this submission faced strong opposition from community developer Ingo Molnar, who believed that having pluggable schedulers would discourage patches for scheduling domains and instead lead to separate schedulers for specific scenarios like NUMA scheduling and SMP scheduling. Ingo Molnar’s standpoint was clear: If everyone focuses on their own little family, the scheduler as a big family will lack organization and code contributions, leading to the existence of schedulers tailored to specific scenarios. https://www.ebpf.top/en/post/bpf_rawtracepoint/ Sat, 09 Mar 2024 12:34:12 +0800 https://www.ebpf.top/en/post/bpf_rawtracepoint/ Read the article at: https://www.ebpf.top/en/post/bpf_rawtracepoint/ 1. Common Hook Types in eBPF Trace 2. BPF Rawtracepoint 2.1 Trace Performance Enhanced by 20% 2.2 Rawtracepoint Tracking Event Inspection and Quantity Statistics 2.3 Passing Parameter Changes 3. Example of Using rawtracepoint in BPF Programs 3.1 libbpf Library (Based on CO-RE) 3.2 Bpftrace Sample Code See Also 1. Common Hook Types in eBPF Trace eBPF allows tracing events in various categories within the trace domain as follows: Kernel static trace points tracepoint/rawtracepoint/btf-tracepoint Refer to /sys/kernel/tracing/available_events Kernel dynamic trace points k[ret]probe, fentry/fexit (based on BTF) Kprobe /sys/kernel/tracing/available_filter_functions User-space static trace points USDT Viewing method: readelf -n or bpftrace tool bpftrace -l 'usdt:/home/dave/ebpf/linux-tracing/usdt/main:*' User-space dynamic trace: u[ret]probe, obtainable via nm hello | grep main Performance monitoring counters PMC perf_event This article will focus on rawtracepoint within kernel static tracing, concluding with practical code examples using the libbpf development library and bpftrace. https://www.ebpf.top/en/post/bpfman_fedora_40/ Tue, 30 Jan 2024 13:04:00 +0800 https://www.ebpf.top/en/post/bpfman_fedora_40/ Original Article: https://www.ebpf.top/post/bpfman_fedora_40 1. Background 2. Introducing bpfman 3. Standalone Deployment Process 4. Kubernetes Cluster Deployment Process 5. Summary Appendix: bpfman Single Machine Verification Development Environment Setup Code Download and Compilation of bpfman Test program management functionality 1. Background Fedora 40 proposes bpfman as the default program manager. The open-source project bpfman provides a deeper understanding of the eBPF runtime state, making it easier to manage eBPF programs (including loading, unloading, and viewing runtime status). This proposal requires approval from the Fedora Engineering Steering Committee (FESCo), but if successful, bpfman is likely to appear in Fedora 40 in April to enhance eBPF management. So, what exactly is bpfman? This article will give you a brief introduction to bpfman and its working principles. 2. Introducing bpfman Originally named bpfd, bpfman is developed based on the Rust Aya library, using the Rust programming language. https://www.ebpf.top/en/post/network_and_bpf_2024/ Sun, 21 Jan 2024 22:21:30 +0800 https://www.ebpf.top/en/post/network_and_bpf_2024/ Article address: https://www.ebpf.top/post/network_and_bpf_2024 1. eBPF 1.1 Exponential Growth of eBPF 1.2 eBPF Application Market 1.3 Wider Application of eBPF in Mobile Devices 1.4 Risks of eBPF Abuse 2. Observability 2.1 The Most Popular Topic: Observability 2.2 Reducing Observability Overhead 2.3 Context-Aware Kubernetes Workloads 2.4 AI-Assisted Network Troubleshooting 3. Networking 3.1 Container Networking Performance Matching Host Networking Performance 3.2 Transformation in the Networking Industry 3.3 Cilium in Home Environments 3.4 Network Operators Seeking LLM Help - Not All Roses 4. Cloud Native 4.1 Kubernetes Users Pushing Back on Complexity 4.2 IPv6-Only Kubernetes Clusters Becoming More Common 4.3 Rapid Growth of WSAM 4.4 The Not-to-Be-Forgotten Heterogeneous Networks 4.5 The Challenges of Platform Engineering and Network Growth In early 2024, Nico Vibert, Senior Sales Engineer at Isovalent, made some predictions about networks and eBPF. Here, we’ll briefly outline some important conclusions, mainly related to eBPF/Cilium/cloud-native/network/observability, and security areas. https://www.ebpf.top/en/post/top_and_tricks_for_bpf_libbpf/ Fri, 12 Jan 2024 10:08:49 +0800 https://www.ebpf.top/en/post/top_and_tricks_for_bpf_libbpf/ Program Framework (Skeleton) Combine the Open and Loader Stages Selective Attachment (Attach) Custom load and attach Multiple BPF handlers for the same event Map Reduce pre-allocation overhead Determining Map Size at Runtime Per-CPU Global Variables Note on accessing fields directly through pointers Conclusion Link to this article: https://www.ebpf.top/post/top_and_tricks_for_bpf_libbpf Original article: https://www.pingcap.com/blog/tips-and-tricks-for-writing-linux-bpf-applications-with-libbpf/ In the early days of 2020, when I was using the BCC tool to analyze our database’s performance bottlenecks and pulled code from GitHub, I unexpectedly found an extra libbpf-tools directory in the BCC project. I studied the articles on BPF Portability and BCC to libbpf Conversion and transformed the previously submitted bcc-tools into libbpf-tools based on the knowledge I had gained. Finally, I completed the conversion work for nearly 20 tools (see Why We Switched from BCC-Tools to libbpf-Tools for BPF Performance Analysis). During this process, I was fortunate to receive a lot of help from Andrii Nakryiko (the person in charge of the libbpf + BPF CO-RE project). https://www.ebpf.top/en/post/lsm_bpf_intro/ Thu, 04 Jan 2024 15:49:27 +0800 https://www.ebpf.top/en/post/lsm_bpf_intro/ This article is available at: https://www.ebpf.top/post/lsm_bpf_intro 1. Background on Security 2. General Framework of Kernel Security Policy Module LSM 2.1 Introduction to LSM Framework 2.2 Architecture of LSM 2.3 Hook Functions in LSM 3. LSM BPF 3.1 BCC Practice 3.2 libbpf-bootstrap Framework Practice 4. Summary 5. Appendix: Process of Finding LSM Hot Patch Kernel Vulnerabilities Monitoring Hook Points 1. Background on Security Internationally, computer security is summarized by three main characteristics: Confidentiality, Integrity, and Availability (CIA). Confidentiality means that data is not visible to unauthorized individuals. Integrity refers to information not being altered during storage or transmission. Availability implies that one’s devices can be used when needed. There are roughly four methods that computer systems employ to address security challenges: isolation, control, auditing, and obfuscation. Access control involves controlling access, representing the subject’s actions on objects. Access control primarily involves defining subjects and objects, operations, and setting access policies. https://www.ebpf.top/en/post/cisco_and_isovalent/ Fri, 22 Dec 2023 18:23:32 +0800 https://www.ebpf.top/en/post/cisco_and_isovalent/ Article link: https://www.ebpf.top/en/post/cisco_and_isovalent On December 21, 2023, Thomas Graf, CTO & Co-founder of Isovalent, and Tom Gillis, Senior Vice President and General Manager of Cisco’s Security Business Group, announced on their respective company websites Cisco’s plan to acquire Isovalent. The acquisition price was not disclosed by either party. Following the completion of the acquisition, the Isovalent team is set to join Cisco’s Security Business Group, with the acquisition expected to be finalized in the third quarter of the 2024 fiscal year. Cisco has a history with Isovalent, having participated in Isovalent’s $29 million Series A funding at the end of 2020. Subsequently, in 2022, Cisco, along with Microsoft, Google, and other companies, added $40 million in the Isaolvent’s Series B funding. Cisco aims to enhance its capabilities in multi-cloud networking and security through this acquisition. The collaboration between Cisco and Isovalent will leverage the power of Cilium’s open-source technology to create uniquely advanced multi-cloud security and networking functionalities, aiding customers in simplifying and accelerating their digital transformation journey. https://www.ebpf.top/en/post/cpu_io_wait/ Fri, 22 Dec 2023 18:14:18 +0800 https://www.ebpf.top/en/post/cpu_io_wait/ Read more at: https://www.ebpf.top/post/cpu_io_wait 1. Definition of I/O Wait 2. Test and Verification 3. Further Clarification on Disk Throughput and Processes with High I/O FrequencyAfter identifying process I/O wait conditions through the vmstat b column, we can further define them using iostat and iotop. 4. Analysis of Kernel CPU Statistics Implementation 5. Conclusion References 1. Definition of I/O Wait I/O Wait is a performance metric for a single CPU, indicating the idle time consumed when threads in the CPU dispatch queue (in the sleep state) are blocked on disk I/O. The CPU’s idle time is divided into truly idle time and time spent blocked on disk I/O. A higher CPU I/O Wait time indicates a possible bottleneck in the disk, causing the CPU to wait idle. If you find this definition a bit confusing, then please continue reading. I believe that after you read the testing and verification process in this article, your understanding of the above definition will be different. https://www.ebpf.top/en/post/ebpf_and_kernel_feature/ Sat, 08 Jul 2023 18:28:42 +0800 https://www.ebpf.top/en/post/ebpf_and_kernel_feature/ This article link: https://www.ebpf.top/en/post/ebpf_and_kernel_feature BPF kfuncs Bloom Filter Map: 5.16 Compile Once – Run Everywhere: Linux 5.17 [Kernel Space] bpf_loop() Helper Function: 5.17 BPF_LINK_TYPE_KPROBE_MULTI: 5.18 Dynamic Pointers and Type Pointers: 5.19 USDT: 5.19 BPF panic: 6.1 BPF Memory Allocator, Linked Lists: 6.1 User Ring Buffer: 6.1 User-defined linked list support: 6.2 User-defined rbtree support: 6.3 BPF Generic Iterator: 6.4 References: In 2022, the Linux kernel primarily released versions 5.16-5.19, 6.0, and 6.1, each introducing numerous new features for eBPF. This article briefly introduces these new features. For more in-depth information, please refer to the provided links. Overall, eBPF remains one of the most active modules in the kernel, with its functionality continuously evolving at a rapid pace. In a sense, eBPF is rapidly evolving towards a comprehensive kernel-state programmable interface. BPF kfuncs The BPF subsystem exposes many aspects of kernel https://www.ebpf.top/en/post/xdp_lb_demo/ Sat, 06 May 2023 08:19:24 +0800 https://www.ebpf.top/en/post/xdp_lb_demo/ This article is available at: https://www.ebpf.top/post/xdp_lb_demo Author: Qiu Kang Creating network environment SLB Cluster Routing NAT Mode Load Balancing Algorithms With the progress of eBPF, we can now deploy eBPF/XDP programs directly on regular servers to achieve load balancing, saving the need for dedicated machines for LVS deployment. The previous article shared how to use xdp/ebpf to replace LVS for SLB. It adopted the independent machine deployment mode for SLB and loaded xdp program using bpftool and hardcoded configuration, which was version 0.1. Version 0.2 modified the 0.1 version to a programmatic loading mode based on BPF skeleton. To experience this workflow easily without changing the overall deployment mode of version 0.1, you can check out https://github.com/MageekChiu/xdp4slb/tree/dev-0.2 Version 0.3 added support for dynamically loading SLB configurations in the form of configuration files and command-line parameters. This article belongs to version 0. https://www.ebpf.top/en/post/get_start_chatgpt/ Fri, 03 Feb 2023 09:09:58 +0800 https://www.ebpf.top/en/post/get_start_chatgpt/ This article can be found at: https://www.ebpf.top/post/get_start_chatgpt 1. Introduction 2. Direct Experience 3. Official Account Registration 3.1 Account Registration 3.2 International SMS Verification 3.3 Testing the Effect 4. Summary References 1. Introduction ChatGPT is likely familiar to most people, sweeping the entire Internet and becoming a hot technology pursued by tech companies and a topic of great interest in the IT industry. ChatGPT is an AI chatbot program developed by OpenAI, launched in November 2022. ChatGPT is a large language model developed and trained by OpenAI, using a large language model based on the GPT-3.5 architecture and trained through reinforcement learning. It can generate high-quality text and performs well in various language tasks such as conversation generation, text summarization, text classification, etc. Some say it’s a career killer for coders, a firefighter of inspiration for poets, a dream crusher for “town problem solvers”… Give it a simple command, and it can code, write poetry, or research papers. https://www.ebpf.top/en/post/ebpf_rust_aya/ Sun, 06 Nov 2022 15:23:26 +0800 https://www.ebpf.top/en/post/ebpf_rust_aya/ This article can be found at: https://www.ebpf.top/post/ebpf_rust_aya 1. Introduction 2. Setting Up Rust Development Environment 2.1 Create a VM Virtual Machine 2.2 Install Rust Development Environment 2.3 Install Dependencies for bpf-linker and bpftool 3. Aya Guide to Create eBPF Programs 3.1 Creating a Project Using the Guide 3.2 Compile eBPF Program 3.3 Run User-space Program 4. Conclusion References 1. Introduction A significant change in Linux Kernel version 6.1 is the introduction of support for the Rust programming language. Rust is a system programming language that offers robust compile-time guarantees and precise control over memory lifetimes. Integrating Rust language into kernel development will bring additional safety measures to the early stages of kernel development. eBPF is a technology in the kernel that enables running user-defined programs based on events, with a validator mechanism ensuring the security of eBPF programs running in https://www.ebpf.top/en/post/bpf2pbpf_tail_call/ Sat, 24 Sep 2022 20:51:59 +0800 https://www.ebpf.top/en/post/bpf2pbpf_tail_call/ Site link: https://www.ebpf.top/post/bpf2pbpf_tail_call Author: Richard Li (Original author’s permission obtained) Original article link: https://blog.csdn.net/weixin_43705457/article/details/123474244 1. Introduction 2. Tail Call 3. BPF to BPF Calls 4. CO-RE Sample 5. Tail Call Costs in eBPF 6. Summary 7. References 1. Introduction This article first introduces the general restrictions and usage of tail calls, compares them with BPF to BPF calls, and finally provides a modified version I made of the tail call sample in the kernel source code (using CO-RE). (When learning about tail calls, I struggled with not having a simple and understandable example that could run, so I ended up creating one myself. I believe this version is the most beginner-friendly and logically clear among all examples currently available). 2. Tail Call BPF provides a capability to safely inject code when kernel events and user program events occur, allowing non-kernel developers to control the kernel. https://www.ebpf.top/en/post/ftrace_kernel_dynamic/ Sun, 01 May 2022 10:51:00 +0800 https://www.ebpf.top/en/post/ftrace_kernel_dynamic/ This article can be found at: https://www.ebpf.top/post/ftrace_kernel_dynamic 1. Basic Knowledge 1.1 Default Compilation 1.2 Using the -pg Option 1.3 Using the -pg and -mfentry Options 1.4 Kernel Verification 2. Validation of kprobe tracing mechanism in ftrace 3. Validation using a Kernel Module 3.1 Using kallsyms_lookup_name 3.2 Using Kernel Function Addresses Directly (Pitfalls, Optional) 4. Verification using gdb + qemu Reference In the Geek Time’s “Mastering Container Battles” by Teacher Li Chengyuan, a pondering question was left in the extra session 04 | Understanding ftrace(2): How to Understand the Technology Behind ftrace Tracepoint and kprobe?: Consider this, how can we observe that the first instruction of the corresponding kernel function has been replaced after we register a probe with kprobe for it? Kprobe is a mechanism for dynamic tracking of kernel functions. By using this mechanism, almost all kernel functions can be tracked (excluding those annotated with __kprobes/nokprobe_inline and those marked with NOKPROBE_SYMBOL). https://www.ebpf.top/en/post/no_space_left_on_devices/ Sat, 26 Mar 2022 10:20:46 +0800 https://www.ebpf.top/en/post/no_space_left_on_devices/ This article can be found at: https://www.ebpf.top/post/no_space_left_on_devices 1. Understanding “no space left on device” Error 2. Problem Analysis and Localization 2.1 Preliminary Identification of Problematic Function 2.2 Locating the Root Cause of the Issue 2.3 Identifying the Root Cause of the Issue 3. Analysis of Discrepancies Between Code Flow and Tracing Process References Recently, there have been cases of failures in creating containers with the error “no space left on device” in the production environment. However, during the investigation, it was found that disk space and inodes were quite normal. In cases where conventional troubleshooting methods have failed, is there a quick and universal approach to pinpointing the root cause of the problem? This article records the analysis and troubleshooting process using eBPF + Ftrace in a separate environment. Considering the general applicability of this approach, it has been organized in the hope of serving as a stepping stone for further exploration.