feat(install,store,ci): Phase 66 Phase 4d — v1→v2 silent migration + default flag flip#33
Open
tolgaergin wants to merge 2 commits into
Open
feat(install,store,ci): Phase 66 Phase 4d — v1→v2 silent migration + default flag flip#33tolgaergin wants to merge 2 commits into
tolgaergin wants to merge 2 commits into
Conversation
…ing + v1→v2 cache-hit translation Closes the two perf gaps deferred from Phase 4b's handoff so that Phase 4d's default flip can land on an honest cold/warm-install benchmark comparison vs. v1. ## #5 — speculative pre-fetcher under v2 (was: drained as no-op) Pre-Phase-4d, the speculative dispatcher (which prefetches likely tarball downloads in parallel with resolution) explicitly drained its channel as a no-op when `LPM_STORE_VERSION=v2` was set, because its inner `stream_and_store_package` calls only know v1's `<HOME>/.lpm/store/v1/<pkg>/<version>/` slot. v2 installs paid full per-package fetch latency on the hot path because spec was effectively disabled. `spawn_speculation_dispatcher` now takes `Option<Arc<v2::Store>>`; both call sites (post-Phase-60 fused resolver + walker arm) pass `store_v2_handle` through. `speculative_download_and_store` branches on the v2 store handle: when `Some`, it collects the response body to memory and calls `extract_object_from_bytes` (idempotent on object hits); when `None`, it streams to disk via the legacy v1 path. The store-hit short-circuit at the top is also layout-aware: under v2 with a recorded SRI we check `paths().object_dir(sri).exists()` instead of v1's `(name, version)` lookup, so spec correctly identifies an already-populated v2 object. Spec sets are bounded (~hundreds of packages, each typically <500 KB compressed) and the upstream semaphore caps concurrent allocator pressure; the in-memory shape vs v1's streaming-to-disk is a non-issue at this scale. Streaming-to-disk into v2 is a Phase 4d/4f follow-up if benchmarks ever surface allocator pressure as a bottleneck. ## #6 — v1 → v2 cache-hit translation Pre-Phase-4d, the install pipeline's cache-hit gate force-fetched every CAS-backed package under v2 mode, even when v1 already had the extracted bytes. A user upgrading lpm-rs from v1 to v2 paid the full network cost on their first v2 install of every project — the opposite of the cross-project sharing v2 is supposed to enable. New `lpm_store::v2::Store::populate_object_from_v1(v1_pkg_dir, sri)` recursively copies `<v1>/<name>/<version>/` into a tmp staging dir, rewrites `.integrity` to the caller-supplied SRI (so it's byte- equivalent to what `extract_object` would write), preserves `.lpm-security.json` if present (regenerates if not), and atomic-renames into place at `~/.lpm/store/v2/objects/<sri>/`. Idempotent: a second call on a complete object dir is a no-op. `std::fs::copy` invokes `copy_file_range(2)` on Linux (CoW reflink where supported) and `fcopyfile(2)` on macOS — essentially free on reflink-capable filesystems, bounded by a single tar-extract's IO cost otherwise. We don't reach for `clonefile()` directly because translation is one-time-per-package (not the hot install path) and keeps the lpm-store crate free of macOS-specific bindings. Wired in `install.rs` immediately before the pre-existing v1 cache- hit gate: under v2 mode + non-local-source + recorded integrity + `store_has_source_aware`, attempt translation; on success take the `cached += 1; continue;` slot, on failure fall through to fetch. The v2 link dispatch reads `objects/<sri>/` directly so no additional plumbing is needed once translation succeeds. ## Tests - `lpm-store::v2::store::tests::populate_object_from_v1_copies_extracted_package_dir` — end-to-end happy path: v1 dir → v2 object dir, contents preserved, `.integrity` rewritten, idempotent. - `lpm-store::v2::store::tests::populate_object_from_v1_runs_analysis_when_security_cache_missing` — pathological-but-legal case where v1 didn't ship `.lpm-security.json`; translation regenerates it so v2's post-write contract holds. Pre-merge gate green: clippy --workspace --all-targets, fmt, all 5722 nextest pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…default flag flip
The Phase 4 series's user-visible payoff. After this commit, an
upgrade-in-place user runs `lpm install` once and gets the v2
virtual-store layout: project `node_modules/<dep>` becomes a symlink
into `~/.lpm/store/v2/links/<graph-key>/...`, the cross-project
sharing v2 was designed for kicks in, and v1's per-project
`<project>/.lpm/wrappers/` ceases to exist. Explicit
`LPM_STORE_VERSION=v1` remains as a downgrade-rollback path for users
hitting a v2 regression.
## Migration sequence (preplan §3.1, §3.2)
`commands::install::needs_v2_migration(project_dir)` returns true
iff the project still has v1 layout markers — `<project>/.lpm/wrappers/`
or `<project>/.lpm/hoisted/`. On match, `migrate_v1_to_v2` runs
before resolution:
1. `rm -rf <project>/.lpm/wrappers/`
2. `rm -rf <project>/.lpm/hoisted/`
3. `rm -rf <project>/node_modules/` (entirely, including `.bin/` —
shims regenerate from the post-migration install layout; preserving
`.bin/` would point shims at deleted wrapper paths and crash
every `npx <bin>`)
4. `rm -f <project>/.lpm/install-hash` (the v6 hash baked v1 layout
assumptions; force fresh hash regen)
Each step is idempotent; project-level sidecars `build-state.json`,
`trust-snapshot.json` etc. survive (only the layout-specific roots
are wiped). The store-side `~/.lpm/store/v1/` is intentionally
**not** wiped — other projects on the same machine may still depend
on it; users opt into pruning via `lpm cache prune --legacy-v1`
(Phase 4e).
## Freshness-gate dual wiring
The migration check has to fire BEFORE the install pipeline, but the
sync fast lane in `main.rs` runs `install_state::check_install_state`
ahead of the async install path. Both freshness probes
(`try_mtime_fast_path` + `check_install_state_with_content`) gain a
matching v2-migration gate: when the active `StoreVersion::from_env()`
is v2 AND `<project>/.lpm/{wrappers,hoisted}/` exists, return
`up_to_date: false` so the install path runs and triggers the
migration. Mirrors the Phase 61.3 D8c shape for legacy-isolated →
new-isolated migration.
## StoreVersion default flip
`lpm_store::StoreVersion`:
- `#[default]` moves from `V1` to `V2`.
- `parse(None) → V2`; empty/`v2`/`2` → V2; `v1`/`1` → V1; unknown
→ V2 + warning trace (was V1 + warning).
- `from_env()` doc rewritten to reflect the post-flip defaults.
The pre-flip parse rule treated unknown values as V1 because v2 was
opt-in dev-only — a typo silently kept users on v1. Post-flip the
safe default is V2 (the active production layout); typos fall through
to the production path with a warning rather than to a deprecated
downgrade.
## Test infrastructure scoping
Workflow tests assert on v1 layout shape (project-side real dirs,
hardlink-detach behavior, hoisted-flat `node_modules/<dep>` paths,
etc.). Pinning each test to `LPM_STORE_VERSION=v1` keeps their
contracts intact while v2's regression coverage continues to live in
the audit-fixture CI matrix.
- `tests/workflows/tests/support/mod.rs::lpm()` sets `LPM_STORE_VERSION=v1`
unconditionally (matches the existing pattern for `LPM_LINKER`,
`LPM_NPM_ROUTE`, `LPM_FORCE_FILE_AUTH`).
- `crates/lpm-cli/src/commands/install_global.rs::TestEnvGuard`
adds `LPM_STORE_VERSION=v1` since `lpm install -g`'s test suite
asserts on tarball-fetch counts that don't account for v2's
cache-translation + spec-prefetch idempotency.
- `install_state::tests::populated_new_layout_does_not_force_install`
is renamed + repurposed to assert the post-Phase-4d contract:
v1 wrappers populated under v2 default → migration owed → not
fresh.
## Speculation/cache-hit gate completeness
Three v2 cache-hit short-circuits in the install pipeline:
1. **v2 native hit** — `if v2_mode && p.integrity exists && objects/<sri>/ exists → cached`. Catches the speculative pre-fetcher's writes (#5) so the real fetch loop doesn't redownload.
2. **v1 → v2 translation** — `if v2_mode && v1 has bytes && SRI known → populate_object_from_v1; cached`. Catches the upgrade-in-place case where v1 store has bytes but v2 doesn't.
3. **v1 native hit** — pre-existing.
Order matters: native v2 check first (cheapest), then translation
(cheap copy), then v1 cache hit (only fires under v1 mode).
## CI matrix update
`.github/workflows/ci.yml` audit-fixtures matrix flipped:
- Pre-Phase-4d: `["v1", "v2"]` with v2 row setting `LPM_STORE_VERSION=v2`.
- Post-Phase-4d: `["v2", "v1"]` with v1 row setting `LPM_STORE_VERSION=v1`.
- v2 row now exercises the natural production default (env unset).
## Tests added
- `commands::install::tests::needs_v2_migration_detects_legacy_isolated_wrappers`
- `commands::install::tests::needs_v2_migration_detects_legacy_hoisted_metadata`
- `commands::install::tests::needs_v2_migration_returns_false_on_clean_v2_or_fresh_project`
- `commands::install::tests::migrate_v1_to_v2_wipes_all_required_paths`
- `commands::install::tests::migrate_v1_to_v2_is_idempotent_on_clean_state`
- `commands::install::tests::migrate_v1_to_v2_preserves_project_lpm_sidecars`
StoreVersion parser tests rewritten to reflect post-flip defaults
(unset → V2; aliases for V1/V2 swapped polarity).
## End-to-end smoke (manual; reproducible against the realworld-app fixture)
1. `LPM_STORE_VERSION=v1 lpm-rs install` → `<project>/.lpm/wrappers/` populated.
2. `lpm-rs install` (default v2) → migration message + clean reinstall.
3. `<project>/.lpm/wrappers/` is gone; `node_modules/<dep>` is a
symlink into `~/.lpm/store/v2/links/<graph-key>/...`.
4. `lpm-rs doctor` reports `node_modules: symlinks into ~/.lpm/store/v2/links/`.
## Pre-merge gate
- cargo clippy --workspace --all-targets -- -D warnings ✓
- cargo fmt --check ✓
- cargo nextest run --workspace --exclude lpm-integration-tests
→ 5728 tests pass, 7 skipped ✓
- cargo test -p lpm-auth (parallel-deterministic) ✓
- audit-fixtures default (v2) → 17 PASS / 1 SKIP / 0 mixed ✓
- audit-fixtures `LPM_STORE_VERSION=v1` (downgrade) → 17 PASS / 1 SKIP / 0 mixed ✓
## Out of scope for 4d (queued for 4e/4f)
- `lpm cache prune` (Phase 4e — orphan detection in `~/.lpm/store/v2/`,
`--legacy-v1` mode for retiring v1 store post-migration).
- Linker default flip from isolated → hoisted (Phase 4f, dual-gated
on bench: hoisted warm-install ≤ 200 ms absolute AND ≤ isolated +
30 ms paired).
- README bench refresh — wait until 4f's bench gate runs; 4d's
perf surface is per-fixture audit correctness, not warm-install
latency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Phase 4 series's user-visible payoff. After this lands, an upgrade-in-place user runs
lpm installonce and gets the v2 virtual-store layout: projectnode_modules/<dep>becomes a symlink into~/.lpm/store/v2/links/<graph-key>/..., the cross-project sharing v2 was designed for kicks in, and v1's per-project<project>/.lpm/wrappers/ceases to exist. ExplicitLPM_STORE_VERSION=v1remains as a downgrade-rollback for users hitting a v2 regression.Stacks on #32 (Phase 4c — read pipeline supports v1 and v2). Will auto-rebase after #30 → #31 → #32 land.
What's in this PR
Two commits:
Phase 43 — Tarball URLs in the lockfile #5 + fix(registry): honor --insecure end-to-end on tarball paths + Phase 43 gate #6 perf followups [b4ecebd] — speculative pre-fetcher routes through v2 instead of draining as no-op (Phase 43 — Tarball URLs in the lockfile #5); v1→v2 cache-hit translation via
Store::populate_object_from_v1so upgrade-in-place users don't pay full re-download cost on their first v2 install (fix(registry): honor --insecure end-to-end on tarball paths + Phase 43 gate #6). Both were deferred from Phase 4b's handoff so 4d's perf comparison can be honest.Migration + default flip [0f956bf] —
commands::install::needs_v2_migration+migrate_v1_to_v2per preplan §3.1/§3.2. Freshness gate dual-wired (install_stateslow path +try_mtime_fast_path).StoreVersiondefault flips from V1 → V2; parse rule re-pivots so unset/typos → V2, explicitv1/1→ V1. CI matrix flipped (v2 is now natural default; v1 row exercises explicit downgrade). 6 new migration unit tests; existing parser tests rewritten for post-flip semantics. Test-infra scoping: workflow helper + install_global TestEnvGuard pin toLPM_STORE_VERSION=v1since those tests assert on v1 layout shape.Migration sequence (preplan §3.2)
rm -rf <project>/.lpm/wrappers/rm -rf <project>/.lpm/hoisted/rm -rf <project>/node_modules/(entirely, including.bin/)rm -f <project>/.lpm/install-hashEach step is idempotent. Sidecars (
build-state.json,trust-snapshot.json) survive. The store-side~/.lpm/store/v1/is intentionally not wiped — other projects on the same machine may still reference it; users opt into pruning vialpm cache prune --legacy-v1(Phase 4e).v2 cache-hit gate completeness
Three short-circuits in install-pipeline order:
objects/<sri>/already exists (catches the speculative pre-fetcher's writes).objects/<sri>/.Without #1, mocked workflow tests (each tarball
expect(1)) double-fetched once spec was wired through.Pre-merge gate
End-to-end migration smoke
Out of scope (queued)
lpm cache prune(orphan detection in~/.lpm/store/v2/,--legacy-v1retirement of v1 store).Test plan
LPM_STORE_VERSION=v1downgradeaudit-fixtures (v1)andaudit-fixtures (v2)rows🤖 Generated with Claude Code