# bpftrace development guide

This document features basic guidelines and recommendations on how to do
bpftrace development. Please read it carefully before submitting pull requests
to simplify reviewing and speed up the merge process.

## Building

bpftrace uses [git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules),
so ensure they are initialized when checking out the code:

```bash
git clone --recurse-submodules https://github.com/bpftrace/bpftrace
cd bpftrace
```

For minimum kernel version requirements, see our
[dependency support policy](dependency_support.md#linux-kernel). Your kernel
should be built with the necessary BPF options enabled. Verify this by running
the `check_kernel_features` script from the `scripts` directory.

### Nix build (recommended)

[Nix](https://nixos.org/nix/) is the most convenient way to build and test
bpftrace. Nix manages all build and runtime dependencies automatically. It is
also used by CI, so you are more likely to catch errors before submitting your
change.

All build and test commands using Nix must be run inside the Nix dev shell
(e.g., `nix develop --command bash -c "<command>"`), or from within an
active `nix develop` session.

```bash
nix develop          # enter dev shell
mkdir build
cmake -B build -DCMAKE_BUILD_TYPE=Debug
make -C build -j$(nproc)
```

To develop with a different LLVM version: `nix develop .#bpftrace-llvm21`

For more Nix examples (static builds, fuzzing, flake management), see
[nix.md](./nix.md).

### Distro build

The distro build relies on you installing all of bpftrace's build and runtime
dependencies on your host and then calling `cmake`.

Please be aware that bpftrace has strict dependencies on new versions of
`libbpf` and `bcc`. We track their upstream closely, so while the distro build
works well on distros with newer packages, developers on distros that lag behind
(e.g. Debian) may want to use the Nix build or manually build and install
`bcc` and `libbpf`.

For a suitable build environment, see our Dockerfiles for detailed dependency
examples:
- [Ubuntu](https://github.com/bpftrace/bpftrace/blob/master/docker/Dockerfile.ubuntu)
- [Fedora](https://github.com/bpftrace/bpftrace/blob/master/docker/Dockerfile.fedora)
- [Debian](https://github.com/bpftrace/bpftrace/blob/master/docker/Dockerfile.debian)

Once all dependencies are installed:

```bash
mkdir build
cmake -B build -DCMAKE_BUILD_TYPE=Release
make -C build -j$(nproc)
```

Key cmake options: `-DBUILD_TESTING=ON` (default),
`-DLLVM_REQUESTED_VERSION=<major>`.

### Build output

The built binary is at `build/src/bpftrace`.

### Troubleshooting

**Kernel Lockdown:** If your system has kernel lockdown enabled (often with
Secure Boot), bpftrace will be blocked. To disable:
- Disable Secure Boot in UEFI, or
- Run `sudo mokutil --disable-validation` and reboot, or
- Temporarily lift lockdown with `SysRQ+x` (until next boot)

## [Tests](../tests/README.md)

Every contribution should (1) not break the existing tests and (2) introduce new
tests if relevant. See existing tests for inspiration on how to write new ones. [Read more on the different kinds and how to run them](../tests/README.md).

## Performance

We aim to not be wasteful, but always keep in mind that performance of the BPF
programs and runtime are the things in the critical path. Often, simplicity and
understandability on non-critical paths is often more important than
performance. That said, occasionally it is useful to measure the performance of
different parts of the pipeline. You may run bpftrace using
`--mode compiler-bench` in order to see the performance of the various
passes during compilation.

Similarly, you may benchmark the code generated by bpftrace using `BENCH` probes
and `--mode bench`:

```
$ bpftrace --mode bench -e 'BENCH:my_benchmark { @a++; }'
Attached 1 probe


+--------------+--------------+
| BENCHMARK    | AVERAGE TIME |
+--------------+--------------+
| my_benchmark | 270ns        |
+--------------+--------------+
```

## Continuous integration

CI executes the above tests in a matrix of different LLVM versions on NixOS.
The jobs are defined in `.github/workflows/ci.yml`.

### Running the CI

CI is automatically run on all branches and pull requests on the main repo. We
recommend to enable the CI (GitHub Actions) on your own fork, too, which will
allow you to run the CI against your testing branches.

### Debugging CI failures

It may often happen that tests pass on your local setup but fail in one of the
CI environments. In such a case, it is useful to reproduce the environment to
debug the issue.

To reproduce the NixOS jobs (from `.github/workflows/ci.yml`):

1. Acquire the job environment from the GHA UI: ![](../images/ci_job_env.png)
1. Run `.github/include/ci.py` with the relevant environment variables set

Example `ci.py` invocations:

```
$ NIX_TARGET=.#bpftrace-llvm10  ./.github/include/ci.py
```

```
$ NIX_TARGET=.#bpftrace-llvm11  \
  CMAKE_BUILD_TYPE=Release \
  RUNTIME_TEST_DISABLE="probe.kprobe_offset_fail_size,usdt.usdt probes - file based semaphore activation multi process" \
  ./.github/include/ci.py
```

### Virtual machine tests (vmtests)

In CI we run a subset of runtime tests under a controlled kernel by taking
advantage of nested virtualization on CI runners. For these tests, we use
[vmtest](https://github.com/danobi/vmtest) to manage the virtual machine.

The instructions in the above "Debugging CI failures" section also work
for the vmtest-ed runtime tests. But if you want to manually try something
quick and dirty in a CI kernel, you can do something like the following:

```bash
$ nix develop

(nix:nix-shell-env) $ vmtest -k $(nix build --print-out-paths .#kernel-6_12)/bzImage -- ./build/src/bpftrace -V
=> bzImage
===> Booting
===> Setting up VM
===> Running command
bpftrace v0.21.0-344-g3acb
```

While we'll defer to `vmtest` documentation for full details, one neat fact
worth pointing out is that `vmtest` will map the current running userspace into
the VM. This means you can run binaries built on your host from inside the
guest, eg. your development build of bpftrace.

## Coding guidelines

This is not about the formatting of the source code (we have `clang-format`
for that). Rather, it's about the semantics of the code and what language
features we try to use / avoid.

Please see [coding_guidelines.md](./coding_guidelines.md) for a full treatment
on the topic.

## Code style

We use clang-format with our custom config for formatting code.

[git clang-format](https://github.com/llvm/llvm-project/blob/main/clang/tools/clang-format/git-clang-format)
can be used to easily format commits, e.g. `git clang-format upstream/master`.

For bpftrace code (standard library and scripts), it should be formatted by
bpftrace itself. You can use `bpftrace --fmt` or `scripts/bpftrace_tidy.sh`.

Note that these are both checked by tests; if the changes don't adhere to our
style CI jobs will fail.

### Avoid 'fix formatting' commits

We want to avoid `fix formatting` commits. Instead every commit should be
formatted correctly.

### Comment style

Strongly prefer C++-style comments for single line and block comments. C-style
comments are still useable for nested comments within a single line, e.g. to
leave an annotation on a specific argument or parameter. In the future, there
may be considerations for automated documentation based on comments, but this
is not currently done.

`bpftrace` itself supports both C-style and C++-style comment blocks. There is
currently no decision on recommended comment style, and both are used freely.

## Merging pull requests

Please squash + rebase all pull requests (with no merge commit). In other words,
there should be one commit in master per pull request. This makes generating
changelogs both trivial and precise with the least amount of noise.

The exception to this is PRs with complicated changes. If this is the case and
the commits are well structured, a rebase + merge (no merge commit) is acceptable.
The rule of thumb is the commit titles should make sense in a changelog.

## Changelog

The changelog is for end users. It should provide them with a quick summary of
all changes important to them. Internal changes like refactoring or test changes
do not belong to it.

### Maintaining the changelog

To avoid having write a changelog when we do a release (which leads to useless
changelog or a lot of work) we write them as we go. That means that every PR
that has a user impacting change must also include a changelog entry.

As we include the PR number in the changelog format this can only be done after
the PR has been opened.

If it is a single commit PR we include the changelog in that commit, when the PR
consists of multiple commits it is OK to add a separate commit for the changelog.

## bpftrace internals

For more details on bpftrace internals, see
[internals_development.md](internals_development.md).
