GSoC 2026 · konflux-ci (Red Hat)

Reproducible Builds: Pipeline Wiring, Verification Tooling, and Benchmark Suite

What I set out to do

Konflux is a build system. You give it your source code, and it turns that into a container image you can run. My project was about a specific property those images didn't have: if you build the exact same source code twice, you should get the exact same image back, byte for byte. Right now, in most build systems including Konflux's default setup, you don't. Build the same code twice and you get two images that behave the same but aren't actually identical underneath.

Why does this matter? If a build is reproducible, anyone can take your source code, build it themselves, and check whether the image they get matches the one you shipped. If it matches, they know the image really came from that source and nothing was added or swapped in along the way. If builds aren't reproducible, that check is impossible, because you'd expect the two images to differ even when nothing suspicious happened.

My job was to make Konflux builds reproducible on an opt-in basis, wire that into the pipelines people actually use, and build a way to check reproducibility automatically instead of just claiming it works.

The design

Before writing code, I wrote up a proposal (Konflux calls these ADRs, architecture decision records) so the maintainers could weigh in on the approach first. This became ADR-0069, merged June 24 and my first-ever contribution to the project.

Here's the core idea a reproducibility check relies on: every container image has a digest, a number computed from every byte in it. Change one byte anywhere inside the image, even a timestamp buried in a file nobody looks at, and the digest changes completely. That's what makes reproducibility checkable at all: build the same source twice, compare the two digests, and if they match, the images are identical down to the byte.

The problem is that build tools normally stamp the current time into an image while building it, and record some history about the build itself. Build the same source at two different times and you get two different digests, even though nothing about the code changed. The ADR proposes three settings to fix that, all off by default:

  • source-date-epoch: a fixed timestamp to use instead of "whatever time it is right now."
  • rewrite-timestamp: makes sure files inside the image's layers also get stamped with that fixed time, not just the image's own metadata. Timestamps hiding inside individual files are exactly the kind of thing that quietly breaks reproducibility if you only fix the outer metadata.
  • omit-history: drops the build-history record from the image, which can also carry the real build time.

Before writing any of this up, I tested whether it would actually work. I built three kinds of images twice each on a test cluster: a plain Alpine image that just copies files in, a Go program built with a couple of extra flags, and a Fedora image that installs a package with dnf. The first two came out byte-identical both times once those three settings were on. The Fedora one didn't. It turned out dnf writes a log file with a timestamp inside it as part of installing the package, and that log ends up baked into the image regardless of the three settings. I wrote that down as a known limitation instead of hiding it, along with the workaround: delete that log file in the same build step that created it.

Getting it into real pipelines

The three settings already existed at the level of Konflux's build task, from work that landed before I started. What was missing was the pipeline layer: the actual pipeline definitions that Konflux users build their images with never asked for those settings, so nobody could turn them on even though the underlying support already existed.

#3670, merged July 15, added the three parameters to the docker-build family of pipelines and passed them through to the build task. All three still default to off, so nothing changes for anyone who doesn't ask for it. I checked this actually worked by building the same commit twice through the real pipeline on a local cluster, for two different kinds of images, and getting matching digests both times.

That covered pipelines going forward, but a lot of users don't pull a fresh pipeline from Konflux directly. They have a copy that was generated into their own repo earlier, and that copy doesn't automatically pick up changes made upstream. #24, merged July 21, adds a migration so those existing copies get the same three parameters too, still off by default.

Refining the CLI

konflux-build-cli is the small command-line tool that Konflux's build task actually runs to do the real work with buildah, the underlying build engine. The rewrite-timestamp setting needs a number to clamp file times to, and the obvious number to use is the timestamp of the git commit being built. My first instinct was to have rewrite-timestamp automatically pull that timestamp in behind the scenes whenever it was turned on.

Discussion on #3670 talked me out of that. One setting silently reaching over and filling in a different setting can surprise someone reading the pipeline later, and it also means you can't ask for "use the commit's timestamp" without also asking for the file-rewriting behavior, even if that's not what you wanted. So the design changed to something a user has to type on purpose: a special value, :from-commit-timestamp:, that you can write into the source-date-epoch field to mean "use the commit's timestamp." Nothing happens automatically. #225 is the code that recognizes that value and substitutes the real timestamp before buildah ever sees it, following three rules: an actual number you type always wins, the special value gets replaced by the real commit timestamp (with a clear error if that timestamp is missing or broken), and leaving the field empty still does nothing, same as before.

You might be wondering why the build step can't just read the commit timestamp off the checkout directly. On a pull-request build, the code you're looking at is a merge commit created fresh at build time, so its own timestamp is always "just now," not the timestamp of the actual commit being tested. An earlier step in the pipeline, the one that clones the code, captures the real commit's timestamp before doing that merge, and #225 is what carries that captured value through to the build step. As of writing this, reviewers have approved #225 and it's marked ready to merge, just not merged yet.

A finding along the way

While testing all of this, I ran into a separate bug. When Konflux builds an image for more than one CPU architecture, like amd64 and arm64, it assembles them into one "index" that lists all of them together. Buildah loops over those architectures using a Go map internally, and Go maps don't guarantee you get items back out in the same order you put them in. That means the same build could produce that list of architectures in a different order on different runs, changing the digest even though none of the actual images changed.

I wrote a fix for konflux-build-cli that explicitly sorts the images by architecture before adding them to the index, so the order would always come out the same. I opened #226 with it.

A maintainer reviewing the PR asked a direct question: doesn't Konflux's pipeline already add these images one at a time, in a fixed order, through separate commands, rather than looping over a map all at once? I went and checked. He was right. The specific way this pipeline calls buildah never touches the code path where that map-ordering problem happens. It goes through a different path that adds each image individually, in the order it's given. I'd reproduced the bug, but it lived in a part of buildah that this pipeline simply doesn't use.

So I closed my own PR. The right response to finding a bug isn't always to ship the fix, sometimes it's to check whether the system you're working on actually exercises the code path where that bug lives, and this one didn't.

Verification tooling and the benchmark suite

Everything above makes a build reproducible if you ask for it, but none of it checks that a given image actually is. My first attempt at that check was #3803, a Tekton pipeline that took every build parameter as a manual input and rebuilt the image into a scratch registry to compare digests.

A reviewer pushed back on it directly: nobody would actually use something that unwieldy, and it would go stale fast, since every new build parameter would need someone to remember to add it by hand. Konflux's own architecture discussion had already asked for something simpler: a tool that takes just the image and its own signed build record, and rebuilds from that alone. So I closed the PR and built that instead.

That tool is verify-repro. Every Konflux build already produces a SLSA provenance attestation, a signed record of exactly which parameters that build ran with. verify-repro reads that record straight off the image, clones the source at the exact commit it names, reruns the build locally with those same parameters, and compares the digest it gets against the image's own digest. Nothing to type in by hand, because the attestation already has it all. It's the same shape Red Hat's own Hummingbird project already runs in production, which is exactly what the reviewer's feedback was pointing toward.

Getting the comparison right took real digging. buildah keeps layers decompressed on disk internally, and different tools recompressing that same content can produce different bytes depending on the exact path they take. So the tool doesn't just list the rebuilt image, it pushes it to a local OCI directory with no registry involved, because that's what forces the same code path a real registry push would use.

I wrote the same kind of limitations into this tool's README that I wrote into the old PR. It can't fetch a hermetic build's prefetched dependencies yet, so a hermetic build will likely fail or mismatch until that's added. It only rebuilds a single-architecture manifest, not a multi-platform index. And like before, a matching digest here only proves the same platform reproduces its own bytes. It can't catch that platform itself being compromised in a way that's internally consistent. The stronger check, rebuilding independently on a different platform, is still just an idea, not built yet.

Testing it end to end against a real Konflux-built image turned up one unexplained gap. Every layer and every file matched byte for byte, except the timestamp on one pre-existing directory that the build never actually modifies. The most likely reason is that the machine I tested on falls back to a different storage driver than a real cluster uses. I wrote that down as observed but unresolved instead of guessing at a cause I hadn't confirmed.

This is also where the benchmark suite lives now. verify-repro ships with 13 automated tests covering how it maps build parameters, matches the right task inside a provenance record, parses a signed attestation, and handles the one setting that can't be resolved from provenance alone. Same kind of repeatable, automated proof as before, just checking a design people will actually use.

Where things stand, and what's left

  • #225 (CLI sentinel): approved by reviewers, waiting to be merged.
  • verify-repro: published, tested end to end against a real Konflux-built image.

A few pieces are still on the list. Extending verification to multi-architecture images is one. The stronger independent-platform rebuild check described above is another, still just an idea. And there's one open question I wrote down but haven't answered yet, whether the SBOM (the automatically generated list of everything inside an image) itself comes out byte-identical across two runs. If it doesn't, the way those SBOMs get tagged and stored will need to change too.

I also plan to update ADR-0069 itself, since right now it only describes the pipeline version of verification, the one I ended up closing. It needs to describe this build-twice-and-compare approach instead.

One more piece is a smaller lift than it looked a few days ago. Wiring in a tool called diffoci, so a mismatch shows exactly which layers differ instead of just "the digest didn't match," used to mean asking another team's maintainers to approve a new tool in a repo I don't own. Now that verification lives in my own repo, that's no longer someone else's call to make. I haven't shipped it yet, but it's mine to build whenever I get to it.

The work, linked directly