How does bpf2c handle memory model differences?

[lmb]

I recently attended a talk by Paul McKenney on the work on a BPF memory model. This exists so that programs accessing memory concurrently have meaningful semantics.

If I understand bpf2c correctly it works by translating a BPF instruction into an equivalent C statement. All statements are then compiled via MSVC, resulting in x64 assembly in a native .sys file. In terms of memory models we get the following:

Source C (Clang memory model) -> BPF (BPF MM) -> Generated C (MSVC MM) -> assembly (hardware MM)

How do you ensure that this transformation of BPF to C preserves BPF memory model semantics in the final assembly?

See https://datatracker.ietf.org/meeting/118/materials/slides-118-bpf-bpf-memory-model-00 for some context.

[shankarseal]

Thanks for bringing this up @lmb. Someone from the eBPF maintainers group will look into the presentation and reply back to this discussion.

[lmb]

@Alan-Jowett one thought I had: instead of relying on MSVC to lower C to assembly, use the JIT you already have and embed its output as inline assembly in the driver. That removes compiler level reordering from the picture. You still have to worry about the JIT, but that's probably easier to tackle.

[Alan-Jowett]

Thanks for the suggestion. This is something we looked into when we originally developed this solution, but this approach would require a lot of additional work.

1. JIT only supports x64 Windows calling conventions, not ARM64 Windows calling conventions.
2. JIT doesn't emit the required debug information.
3. JIT doesn't emit the required pdata and xdata required for correct stack walking.
4. JIT is slower than MSVC generated code.
5. JIT doesn't emit the correct CFG and stack protections that MSVC emits.
6. JIT doesn't emit the correct Spectre mitigations.

In addition, there was a strong pushback from internal folks that we should be using MSVC for native generation so that we would automatically pick-up the latest security mitigations from the compiler.

In short, MSVC gets us machine code with all the latest compiler enhancements and protections that we would need to go and build for the JIT path. This is one of the primary reasons why we are moving towards a native only approach.

[lmb]

Thanks for the context! I guess by (1) you mean that there isn't a JIT for arm64?

Would it be possible to take a hybrid approach where each function is jitted individually (kind of what Linux already does) and then the C is something like:

void ebpf_func_1() {
// inline asm
}

void ebpf_func_2() {
// inline asm

ebpf_func_1()

// etc
}

So control flow would go via the C compiler and therefore gets CFG integrity, debug metadata, etc. Doesn't solve Spectre, arm64 and speed but would give me a better feeling (for lack of a better word) about the correctness of the emitted code. Maybe I'm seeing ghosts though (ha!)

[Alan-Jowett]

Bearing in mind that both compiler and CPU can re-order instructions, I am not sure anything short of explicit memory barriers really make senses long term, especially on architectures like ARM64 which have an even weaker memory model.

There is an ongoing discussion in the BPF mailing list for a proposal for load/acquire and store/release semantics, which I think would be better as it would prevent compiler re-ordering around those instructions.