W3-W6: SoA/AoS layout helpers — SoaVec + soa_struct! + aos_to_soa + soa_to_aos + bulk_apply (scalar; SIMD deferred)

[AdaWorldAPI]

Summary

Establishes SoA/AoS layout-handoff helpers across the codebase. Scalar-only by design — public API forward-compatible with future bench-justified per-arch SIMD acceleration via the existing LazyLock dispatch layer, but no SIMD bodies in this PR.

W3 + W5 + W6 land as a combined deliverable in src/hpc/soa.rs; W4 lands separately in src/hpc/bulk.rs. W7 (cognitive bulk ops over &[Plane] / &[Fingerprint256] / etc.) is explicitly deferred until a bench harness identifies measured hot paths to accelerate.

What's in

Wave	File	LOC	Tests	Symbols
W3	src/hpc/soa.rs	854	29 unit + 10 doctests	SoaVec<T,N>, SoaChunks, soa_struct! macro
W5+W6	src/hpc/soa.rs (same file)	(incl.)	(incl.)	aos_to_soa<T,N,F>, soa_to_aos<T,N,F>
W4	src/hpc/bulk.rs	326	16 unit + 2 doctests	bulk_apply, bulk_scan

Plus four supporting knowledge docs in .claude/knowledge/:

• w3-w6-soa-aos-design.md — design contract (v2, after savant patches)
• w3-w6-plan-review.md — savant's pre-spawn audit
• cognitive-distance-typing.md — typed-distance rule (palette-256 ≠ HDR popcount ≠ Base17 L1; no roundtrips, no umbrella API)
• w3-w6-codex-audit.md — post-sprint codex verdict

Two-layer rule (re-affirmed)

user code (hpc/soa, hpc/bulk, downstream crates)
   ↓ allowed imports only
crate::simd, crate::simd_ops      ← dispatch layer (LazyLock-frozen function-pointer tables)
   ↓
simd_avx512.rs, simd_avx2.rs, simd_neon.rs  ← per-tier impls, these carry #[target_feature]

W3-W6 lives at the user-code level. Zero #[target_feature], zero cfg(target_feature), zero direct simd_{type}.rs imports, zero raw intrinsics. Verified by codex audit (grep checks all returned empty).

Distance typing guardrail

cognitive-distance-typing.md (committed in this PR) establishes that distance metrics in this codebase are typed:

• Palette-256 distance carries (PaletteIdx, PaletteIdx, &Buckets, EulerGammaOffset) → PaletteDistance — buckets and Euler offset are integral
• HDR popcount early-exit IS the cosine replacement (Level 1 of the cascade), not a Fisher-z'd palette result
• Fisher-z is variance-stabilization on palette OUTPUT, not a distance itself
• BF16 mantissa direct-transform is the typed fast path inside palette space (one hop, no cascade)

The worst-case roundtrip (palette → fisher-z → "cosine" → hamming → popcount → palette) is explicitly illegal because each arrow erases the typing the previous step earned.

W3-W6 helpers are layout-only and explicitly do NOT bake in any distance metric. Both module headers warn against extension toward distance and point at the typing doc. W7 (when it lands) will ship per-metric named bulk fns (bulk_hdr_popcount_early_exit, bulk_palette256_distance, bulk_palette256_bf16_mantissa_transform), never a bulk_distance<T> umbrella.

Process — multi-agent sprint

Per the established protocol: plan → savant review → correct → sprint → codex audit → fix P0 → commit → repeat.

1. Design v1 committed as worker contract
2. Plan-review savant spawned — returned READY-WITH-DOC-FIXES with 1 P0 (helpers belong in hpc::soa not simd_ops.rs) + 7 P1
3. Design v2 absorbed all P0/P1 patches + open-question rulings
4. Two workers in parallel in isolated worktrees:
   • Worker A: src/hpc/soa.rs combined (W3+W5+W6) — single commit
   • Worker B: src/hpc/bulk.rs (W4) — single commit
5. Worker commits cherry-picked onto branch
6. Codex audit spawned on combined diff — returned READY-FOR-PR with 0 P0, 1 P1 (usize::MAX docstring gap), 3 P2 deferred
7. P1 patched (docs(hpc/bulk): F4 - document usize::MAX chunk_size semantics)

Codex audit verdict — verification

Command	Exit	Notes
cargo check -p ndarray --no-default-features --features std	0
cargo test --lib --no-default-features --features std hpc::soa	0	29 passed
cargo test --lib --no-default-features --features std hpc::bulk	0	16 passed
cargo test --doc --no-default-features --features std hpc::soa	0	10 passed, 1 intentional ignore
cargo test --doc --no-default-features --features std hpc::bulk	0	2 passed, 1 intentional ignore
cargo fmt --all -- --check	0
cargo clippy --no-default-features --features std -- -D warnings	0

What's deferred (P2 from the audit, explicit non-goals for this PR)

• bulk_scan naming (savant suggested bulk_for_each — kept for symmetry with bulk_apply; rename if downstream finds it misleading)
• SoaVec::iter_rows() row iterator (use soa.chunks(1) for now)
• #[derive(Clone, Debug)] on SoaVec (would require where T: Clone+Debug bounds; macro-generated structs already support derive passthrough)
• Per-arch SIMD acceleration of aos_to_soa / soa_to_aos (defer until bench harness identifies a hot path; the public API forward-compatible via grow-internal-arms)

W7 explicit deferral

Cognitive bulk ops on &[Plane] / &[Fingerprint256] / &[PaletteIdx] are NOT in this wave. Two reasons:

1. No bench data — would design SIMD primitives from imagination
2. Each metric needs its own typed bulk fn per the distance-typing rule; can't safely design without measured hot-path-vs-target-metric pairing

When W7 revisits, bulk fns will be one-per-metric (bulk_hdr_popcount_early_exit, bulk_palette256_distance, etc.), MAY internally use SoaVec + bulk_apply from this PR for staging.

Downstream impact

None. This PR is purely additive: two new modules under hpc/. No existing signatures change. burn-ndarray, candle, tract, ort, lance-graph continue to build unchanged.

The new symbols become available at ndarray::hpc::soa::* and ndarray::hpc::bulk::*. The soa_struct! macro is #[macro_export] so ndarray::soa_struct! works at the crate root.

Test plan

• cargo check clean
• cargo test --lib hpc::soa hpc::bulk — 45 passed / 0 failed
• cargo test --doc hpc::soa hpc::bulk — 12 passed / 0 failed
• cargo fmt --all -- --check clean
• cargo clippy --no-default-features --features std -- -D warnings clean
• Distance-typing guardrail: codex grep confirms zero umbrella API surfaces
• Layering rule: codex grep confirms zero #[target_feature] / cfg(target_feature) / per-arch imports / raw intrinsics
• CI matrix

---

Generated by Claude Code