Skip to content

feat: private crate registry via JFrog Artifactory (#12) - #684

Merged
BernardJen merged 5 commits into
mainfrom
feat/jfrog-cargo
Sep 3, 2026
Merged

feat: private crate registry via JFrog Artifactory (#12)#684
BernardJen merged 5 commits into
mainfrom
feat/jfrog-cargo

Conversation

@BernardJen

@BernardJen BernardJen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Closes the Cargo half of Lab271/labs-jfrog-poc#12.

Eighteen releases in, publish = false and no crates.io presence meant no colleague could depend on this crate. This wires it to Lab271's Artifactory tenant as a private Cargo registry — opt-in, so a default clone is entirely unaffected.

The public-repo trade-off, and why the config is not committed

Cargo has no per-project registry config that lives outside the working tree. npm has .npmrc, and the JFrog CLI can sidestep even that with jf npm-config. There is no jf cargo-config and no jf cargo command at alljf --help says so outright. So .cargo/config.toml and CARGO_* env vars are the only mechanisms on offer.

That matters because this repository is public and the Artifactory Cargo index is not anonymously readable:

$ curl -o /dev/null -w '%{http_code}' \
    https://schubergphilis.jfrog.io/artifactory/api/cargo/lab-cargo-dev/index/config.json
401

The index also advertises "auth-required": true, which Cargo honours for crate downloads as well as index reads. A committed source replacement would therefore 401 every anonymous cargo build — every outside contributor, every fork — at dependency-fetch time, with an authentication error that gives the contributor no hint that the fix is to delete a file they did not add.

So: .cargo/config.toml.example is committed, .cargo/config.toml is gitignored. ADR-0038 records the decision and the four alternatives rejected.

package.publish stays false

It does not need to change. cargo package works under it, and the resulting .crate deploys with jf rt upload, which Artifactory indexes into a valid registry entry on its own (~5s later, with deps, features and cksum parsed from Cargo.toml).

publish = ["lab-cargo-dev"] would also work, and is narrower than it looks — cargo publish --registry crates-io still refuses with "The registry crates-io is not listed in the package.publish value", and a bare cargo publish auto-targets the single allowed registry. But it is unnecessary, and only the upload path produces build-info. Whether this crate should be publishable at all is a separate decision and not this PR's to make.

Two things worth knowing before relying on this

  • sqlite-rs is already taken on crates.io by an unrelated crate — 19 versions, up to 0.3.7. lab-cargo-dev merges its crates.io cache with our local repo, so one index path serves 20 versions from two different projects with nothing distinguishing them. Ours resolves correctly today only because the version ranges do not overlap. lab-cargo-prod has no remote and serves 0.18.10 alone. Renaming the crate is the real fix, before any public release.
  • Consumers should use the named-registry form, not the source replacement. Under source replacement the consumer's Cargo.lock records our private crate as source = "registry+https://github.com/rust-lang/crates.io-index" — untrue, and it makes deny.toml's sources check (unknown-registry = "deny", allow-registry = crates.io only) pass a private-registry dependency silently.

Test plan

Verified against the live tenant, not reasoned about:

cargo fetch --locked through lab-cargo-dev 116 crates resolved, Cargo.lock unchanged — all 116 source entries still point at crates.io
published lab-cargo-dev-local/crates/sqlite-rs/sqlite-rs-0.18.10.crate, sha256:eb1deeae…
build-info artifact + 116 deps + commit e04330e8 in one GET /api/build/sqlite-rs/2?project=lab
promoted --copy=true lab-cargo-prod-local, digest sha256:eb1deeae…identical, no rebuild
consumed a throwaway crate declared sqlite-rs = "0.18", resolved 0.18.10 from the registry, compiled and linked it, and ran

Anonymous access was confirmed broken before choosing the mechanism (the 401 above), which is the whole basis for ADR-0038.

Notes for a human

  • The jfrog-prod environment does not exist yet. GitHub creates it on this workflow's first run with no protection rules, so the promote job is currently ungated. Add required reviewers under Settings → Environments → jfrog-prod before treating it as an approval gate. Until then the environment records who triggered the run, not who approved it.
  • JF_ACCESS_TOKEN is a repo secret carrying project-admin rights on lab, so any workflow in this repository can promote to prod, and one credential covers both build and promote. That is a least-privilege regression against the OIDC design in labs-jfrog-poc#6 and is recorded there, not fixed here.
  • Anyone opting in locally needs their own identity token. Nothing in this repository can mint one.
  • Verified in CI, not reasoned about. Commits 2 and 4 add and then remove a temporary push trigger for this branch — workflow_dispatch only fires for a workflow that already exists on the default branch, so there was no other way to make the publish and promote jobs actually run. All four jobs went green in run 33790369579 (Promoted bytes identical to dev — no rebuild in the path), and the overwrite guard was verified skipping cleanly in run 33790771562. The trigger is gone from the final state; on: is back to tags and workflow_dispatch.
  • Two real defects were found only in CI, both fixed in-branch: Cargo needs [registry] global-credential-providers = ["cargo:token"] named explicitly before CARGO_REGISTRIES_*_TOKEN is consulted at all (not mentioned in Artifactory's Set Me Up), and re-publishing a version silently overwrites released bytes — see the third commit, which is the one worth reading.

Xray

Reported in full in the PoC findings, but the short version, since it bears on this repo's existing supply-chain gate: jf audit does not recognise a Rust project. It reports the tree as [unknown] and generates an SBOM with no library components, so the 116-crate closure that make check-deny and sqlite-rs-dev.cdx.json already cover is never examined. Xray does have Cargo CVE data and applies it to a .crate scanned with jf scan — confirmed against a deliberately vulnerable time 0.1.44, which reports CVE-2020-26235, type cargo — but this crate has zero runtime dependencies, so there is nothing there to find.

make check-deny remains the supply-chain gate. Artifactory adds distribution, not assurance. No change to deny.toml, the SBOMs, or any existing gate is proposed here.

Spend: matched estimate.

🤖 Generated with Claude Code

BernardJen and others added 3 commits September 3, 2026 20:22
Eighteen releases in, `publish = false` and no crates.io presence meant
no colleague could depend on this crate. This wires it to Lab271's
Artifactory tenant as a private Cargo registry, opt-in.

Cargo has no per-project registry config that lives outside the working
tree — there is no `jf cargo-config` and no `jf cargo` command at all,
so `.cargo/config.toml` and `CARGO_*` env vars are the only mechanisms
on offer. That matters here because this repository is public and the
Artifactory Cargo index is not anonymously readable:

    $ curl -o /dev/null -w '%{http_code}' \
        .../api/cargo/lab-cargo-dev/index/config.json
    401

The index also advertises `"auth-required": true`, which Cargo honours
for crate downloads as well as index reads. So a committed
`.cargo/config.toml` carrying a source replacement would 401 every
anonymous `cargo build` — every outside contributor, every fork — and
would do it at dependency-fetch time, with an authentication error that
gives the contributor no hint that the fix is to delete a file they did
not add. The npm pilot of the same evaluation rejected a committed
`.npmrc` for exactly this reason.

Hence: commit `.cargo/config.toml.example`, gitignore
`.cargo/config.toml`. The default clone is untouched and resolves from
crates.io as before. ADR-0038 records the decision and the alternatives.

`package.publish` stays `false`, deliberately. It does not need to
change: `cargo package` works under it, and the resulting `.crate`
deploys with `jf rt upload`, which Artifactory indexes into a valid
registry entry on its own (~5s later, with deps/features/cksum parsed
from Cargo.toml). Setting `publish = ["lab-cargo-dev"]` would also work
and would *not* open a path to crates.io — `cargo publish --registry
crates-io` still refuses — but it is unnecessary, and whether this crate
should be publishable at all is a separate decision that is not this
one's to make.

Two things the docs call out because they are not obvious:

- `sqlite-rs` is already taken on crates.io by an unrelated crate, 19
  versions up to 0.3.7. `lab-cargo-dev` merges its crates.io cache with
  our local repo, so one index path serves 20 versions from two
  projects with nothing distinguishing them. Ours resolves correctly
  today only because the ranges do not overlap. `lab-cargo-prod` has no
  remote and serves 0.18.10 alone.
- Consumers should depend on us via the named-registry form, not the
  source replacement. Under source replacement the consumer's lockfile
  records our private crate as coming from crates.io, which is untrue
  and makes deny.toml's `sources` check pass it silently.

The workflow skips cleanly whenever JF_ACCESS_TOKEN is absent, which is
always the case on forked PRs — a JFrog-caused red build would count
against the evaluation's reliability criterion, whose threshold is zero.

Refs: Lab271/labs-jfrog-poc#12

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`workflow_dispatch` only fires for a workflow that already exists on the
default branch, so from a feature branch there is no way to make the
publish and promote jobs actually run. An unexecuted workflow is not
evidence that it works, and the promote job carries the evaluation's
hard-fail assertion (promoted digest identical to the dev digest), so it
needs a real run rather than a reading.

Reverted in the next commit, before merge.

Also makes the consume step skip when the current version is not in
lab-cargo-prod yet, which is the normal state on a PR that bumps the
version. Failing there would be a JFrog-caused CI failure, and the
evaluation's threshold for those is zero.

Refs: Lab271/labs-jfrog-poc#12

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Artifactory's Cargo index advertises `"auth-required": true`, and Cargo
then refuses to query it at all unless a credential provider is named:
"authenticated registries require a credential-provider to be
configured". `CARGO_REGISTRIES_*_TOKEN` alone is not enough — the
provider has to be listed before the env var is consulted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BernardJen and others added 2 commits September 3, 2026 20:28
Found the hard way, and it broke a real consumer:

  * `cargo package` is not byte-reproducible across machines. Commit
    e04330e produced sha256:eb1deeae… on a laptop and sha256:84074b9f…
    on a runner.
  * `lab-cargo-dev-local` and `lab-cargo-prod-local` are mutable.
    Re-uploading 0.18.10 overwrote the bytes silently — no error, no
    warning — including in prod, where that version already carried
    status `Released`.
  * Cargo pins checksums in Cargo.lock, so that is not a harmless
    re-upload. A consumer pinned to the old digest now fails outright:
    "checksum for `sqlite-rs v0.18.10` changed between lock files ...
    unable to verify that `sqlite-rs v0.18.10` is the same as when the
    lockfile was generated". Reproduced, not theorised.

Publish and promote now skip cleanly when the version already exists,
rather than clobbering it. Bumping the version is the way to release new
bytes.

Repository immutability on the prod local repo
(Lab271/labs-jfrog-poc#4) is the actual fix and needs platform admin we
do not have. Until then this guard is the only thing between a re-run
and a broken downstream lockfile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The publish and promote jobs have now run for real on this branch
(run 33790369579, all four jobs green, "Promoted bytes identical to dev
— no rebuild in the path"), and the overwrite guard has been verified
skipping cleanly on a re-run (run 33790771562). The trigger has served
its purpose.

Back to tags and workflow_dispatch only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@BernardJen
BernardJen merged commit 7701d18 into main Sep 3, 2026
10 checks passed
@BernardJen
BernardJen deleted the feat/jfrog-cargo branch September 3, 2026 19:28
iheitlager added a commit that referenced this pull request Sep 7, 2026
…supersedes #678) (#693)

* feature: spec extension for sqe

* spec: autoindex write-side rule (010/Req-8) + incremental row access (013/Req-7)

010 gains Requirement 8: a write MUST NOT succeed while leaving an index it
could not read unmaintained. Measured at 0.18.5 against stock sqlite3 3.51.0 --
inserting into a stock-created composite-PK table leaves rows out of
sqlite_autoindex_*, after which the oracle undercounts and integrity_check
reports rows missing, while the write returns rc=0. Names both acceptable
fixes (recover the autoindex column list from the declared constraint, or
refuse the write) and scopes autoindex *creation* out to V3/V7.

The embedding-api spec gains Requirement 7: statements must yield rows
incrementally. It also upgrades the composite-PK prerequisite from inferred to
measured, flagging it as the highest-priority item in or around that spec --
it is silent corruption of a valid SQLite file, not an ergonomic gap, and its
Req 6 byte-identity scenario cannot pass while it stands.

That spec is numbered 012 as of this commit and is renumbered to 013 by the
next one, after main landed its own spec 012.

Refs: 010/Req-8, 013/Req-7

* docs: renumber embedding-api spec to 013 and its ADR to 0034

Main landed its own spec 012 (`012-query-constraints`) and its own ADR-0033
(constant propagation / OR-to-IN) while this branch was open, so both numbers
collided on rebase. The newcomer moves:

- `.openspec/specs/012-embedding-api/` -> `013-embedding-api/`, heading updated
- `adr/0033-embedding-api-owns-the-connection-driver-out-of-tree.md` -> `0034-`,
  heading updated, and its five `spec 012` prose references retargeted to 013
- `adr/index.md` gains the 0034 row, which 0ee936b omitted entirely

Renumbering ADR-0034 does not violate the immutability convention: it has
never been on main and is still `Status: Proposed`.

Also fixes four `**Implementation:**` lines in spec 013 that were already
marked `(planned)` but written in a form `tools/assurance.py:474` does not
match -- its regex only accepts `(planned)` immediately after a *single*
backticked path, so `` `a`, `b` (planned) `` scored as active and its
not-yet-written test links counted as dead. With that fixed, the dashboard is
byte-identical to main: 86 active requirements, Completeness 85/86 (99%),
Coverage 270/270 (99%), zero dead links, planned 2 -> 10.

Refs: 013/Req-1, 013/Req-7

* docs: renumber the embedding-API ADR to 0041 and reconcile spec 013 with the tree (#678)

Takes over #678's spec so it can land off current `main`. Jacob's three
commits are cherry-picked verbatim above this one and keep their
authorship; everything here is the reconciliation they could not carry,
because the tree moved after they were written.

ADR renumber, 0034 -> 0041. 0034 was already taken on `main`
(`0034-index-range-seeks.md`) when #678 was opened, and 0038/0039/0040
have since gone to the Cargo registry decision (#684), `Value`'s `Arc`
payloads (#688) and the streaming primitive (#683). Three citations in
spec 013 and the `index.md` row move with it. The index gains a gap at
0039/0040 until those two branches land, which is the correct state for
this branch rather than a placeholder.

Four claims in spec 013 were true when written and are not now:

- **Item 7 said "nothing in `src/vdbe/` offers a step or iterator
  API".** `vdbe::Execution` is that API as of #683, with `run()`
  reimplemented as a wrapper over it. Requirement 7 is restated as a
  facade gap and its `Implementation:` line now points at
  `Execution::next_row` as the thing to build on -- #682 found the
  ordering matters, because a facade retrofitted onto `execute_with_db`
  cannot be made incremental afterwards.

- **Item 3 attributed the `!Send` problem to the pager alone.** So did
  ADR-0041. `Value::Text`/`Blob` held `Rc` payloads, which made a result
  row -- precisely the thing that has to leave a worker thread --
  unsendable too. Closed by #688/ADR-0039, which also narrows
  Requirement 4: the worker thread is still required for the pager, but
  it now hands rows across instead of copying them.

- **The composite-PK prerequisite was one item; it is two.** #685 fixed
  maintenance of an existing `sqlite_autoindex_*` and the read-only
  safety valve (spec 010/Req 8). Creating one on `CREATE TABLE` is
  still open as #687. SQE's two workarounds split the same way: the
  write-refusal one comes out now, the dropped-composite-key one waits
  on #687.

- **Requirement 7's acceptance scenario could not pass.** It asked that
  "peak allocation is proportional to the ten rows". Peak heap for a
  streaming read is a floor set by the page cache, not a slope in rows
  pulled, so no correct implementation satisfies that wording. Restated
  as independence from result size -- 8.68 MB flat against 137.7 MB
  materialized on 1,000,000 rows (#682) -- with
  `DEFAULT_PAGE_CACHE_CAPACITY` named as the knob that moves the floor
  (2000 pages -> 8.68 MB, 64 -> 291 KB, ~4.5% streaming cost). That is
  both the property a consumer needs and, unlike proportionality,
  testable.

Also corrected in both specs: measurements were cited against "stock
`sqlite3` 3.51.0", but the pinned oracle is 3.53.4
(`tests/corpus/oracle.rs:22`, `Cargo.toml [package.metadata.oracle]`).
On this machine a bare `sqlite3` is Apple's 3.51.0 codec build, which
`tools/gen_fixtures.sh` refuses by design. I re-derived the autoindex
rule against the pinned 3.53.4 across eleven DDL shapes, comparing
`pragma_index_info` key lists rather than index counts: every case
agrees, including the counter-intuitive ones (declaration order beats
primary-key-first; a rowid alias consumes no number). The citation was
wrong, not the rule -- but "measured against the pinned oracle" is this
repo's whole assurance basis, so it has to be accurate.

Spec 010/Req 8's three scenario `Tests:` links already name the exact
test functions #685 created. They stay `(planned)` here: the flip
belongs to the PR that discharges the requirement, and it cannot happen
on this branch because the tests do not exist on it.

`make check-assurance` passes at 86/86 and 276/276, unchanged -- every
requirement added here is `(planned)`, so it is excluded from scoring by
design and the dashboard cannot move until the implementing tickets
land. No dead links.

Refs: 010/Req-8, 013/Req-1, 013/Req-4, 013/Req-7, #678, #682, #683,
#685, #687, #688

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: PRAGMA synchronous already has a handler — a fifth stale claim in spec 013 (#678)

My reconciliation commit said four of spec 013's claims had stopped being
true. There were five.

Item 6 read "`Pager` syncs but nothing states what is guaranteed, and
`synchronous` has no handler". The second clause was already false when
#678 was written: #645 implemented `PRAGMA synchronous` in full — the
bare query form reporting `0`/`1`/`2` like stock SQLite, all three
levels, and a decided per-level fsync-skip policy recorded in ADR-0036
(`src/vdbe/pragma.rs:79`, dispatched at `src/vdbe/exec.rs:760`, with
unit tests). Nothing about it is a stub.

That also means Requirement 5 is further along than it claims. It asks
the API to "honor `PRAGMA synchronous` at least to distinguish FULL from
OFF", which is a weaker ask than what already exists, and says "what is
missing is a documented guarantee and any way to trade it" — the trade
mechanism is exactly what #645 added. Requirement 5's remaining work is
the written guarantee, the transaction surface, and the busy/retryable
error handling, not the PRAGMA.

I found this while answering "will this work with SQE yet", by checking
each of the spec's seven gaps against the tree instead of trusting the
list. Worth noting for anyone reviewing #693: the list was written
against 0.18.5 and the tree is 0.18.10, so treat every "is missing"
line as a claim to re-verify rather than a fact. The four I corrected
first were the ones my own branches falsified; this one had been stale
for longer and nothing I built touched it.

`make check-assurance` unchanged at 86/86 and 276/276, no dead links —
Requirement 5 stays `(planned)`, since the requirement as a whole is not
discharged even though this part of it is.

Refs: 013/Req-5, #645, #678

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Jacob Verhoeks <jjverhoeks@schubergphilis.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ilja Heitlager <iheitlager@schubergphilis.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant