Three SBOM Generators, Three Different Claims About the Same Source

Evidence boundary. Under their clean_deep campaign, average JavaScript lockfile-package coverage was 98.72%, 43.78% and 32.71% respectively; Rust was 99.34%, 100% and 83.85%. These are scoped study results, not universal accuracy rankings or proof of deployed binary completeness. The study excludes dynamic and runtime-resolved dependencies and examines source repositories, not supplier-specific production images.

The same source can support different inventories

A product owner may receive an SBOM as if it were an objective list of everything in a release. This study shows why the production contract must say more. The researchers froze a corpus, regenerated lockfiles, compared three generators across two ecosystems and traced divergences in scope, identifier normalization and graph representation. A lockfile is a useful controlled baseline for their research, not a ground truth for everything that actually executes in a built product.

The high JavaScript divergence is partly a scope choice: development dependencies are excluded differently by Syft and Trivy. It would be wrong to declare the larger SBOM automatically better for a runtime exposure question, or the smaller one automatically complete. The supplier and consumer must agree whether the record covers source, build, deployment or runtime and what it includes. They also need to state whether direct, transitive, development, build and optional dependencies are represented.

Make the generation claim inspectable

1. Input and phaseBind the source revision, lockfile, installed tree or image digest to a named generation stage.

2. Scope and identitySpecify dependency classes, nested inventories, package aliases, versions and the graph root.

3. Checks and exceptionsCompare representative outputs with known dependencies, fields and unresolved gaps before using the SBOM in vulnerability decisions.

A CUP proposed acceptance contract derived from the empirical failure modes. It is not a prescribed standard or a benchmark result for a particular supplier.

One concrete edge case is an npm alias: the paper shows tools can report an alias or the real package name for the same lockfile entry, producing different PURLs. A vulnerability lookup keyed to one identifier can then disagree without any source change. Another is a directory exclusion in the tested cdxgen version that silently skipped lockfiles under a directory named examples. These details justify regression tests for the supplier's actual tool version and repository layout rather than trust in a single count.

With no lockfile, the authors observed empty SBOM outputs for Syft and Trivy in the manifest-only campaign. cdxgen tried to install JavaScript dependencies but, for Rust, used only manifest declarations and reached 3.2% coverage against their resolved-lockfile baseline. This is not a universal statement about every invocation or current release; it is evidence that input availability and generation mode are part of the SBOM's claim.

Field presence is a separate gate

The study also checked metadata fields. In both analyzed ecosystems, none of the three generated a Supplier field under the authors' NTIA-oriented measure, even where names, versions and dependency relationships were present. That is a gap in automatically generated evidence, not an assertion that every SBOM fails every current legal requirement. Supplier identity may need to be supplied and checked by the producer's process. The source, tool version, configuration and declared scope must travel with the artifact so consumers can interpret such fields correctly.

The competitive decision

A procurement or release gate should ask competing tools to produce an SBOM for the same frozen representative product, with the same documented input and scope. Check known direct and transitive dependencies, aliases, missing lockfiles, nested projects, package identifiers and fields needed by the recipient. Require explanations of discrepancies and reproducible exceptions, then choose a generation pipeline for the intended evidence claim. Repeat the test when tool versions or build inputs change. This is an engineering proposal; the study did not evaluate a specific automotive or OT product and does not certify any of the three tools.

The October CUP discussion about a zero-result vulnerability search asked whether an individual product ticket could be closed. This article asks an earlier and different decision: what must an SBOM-generation pipeline prove before any downstream absence or applicability query is trusted. Avoid turning the result into another generic 'SBOMs are incomplete' warning.

Written by

Antonio González, PhD

Cybersecurity practitioner focused on product security, OT/ICS and cyber-physical systems. Cybersecurity Under Pressure translates incident evidence into engineering and business decisions.

About →LinkedIn ↗

Source record