feat(testutil/validatormock): port Charon validatormock to Rust#418
Draft
varex83 wants to merge 24 commits into
Draft
feat(testutil/validatormock): port Charon validatormock to Rust#418varex83 wants to merge 24 commits into
varex83 wants to merge 24 commits into
Conversation
Initial port of charon/testutil/beaconmock: BeaconMock with wiremock server, builder API, static endpoints, and deterministic attester/ proposer duties. Migrate eth2util, app/eth2wrap, core/deadline, and cli test code to the shared mock.
Prep for parallel work on headproducer, attestation store, options, validator-set API, and fuzzer. Public API (BeaconMock, MockState, Validator, ValidatorSet) unchanged.
…ions, fuzzer
Brings the Rust beaconmock to functional parity with charon/testutil/beaconmock:
- head producer: slot ticker + SSE on /eth/v1/events + deterministic
block-root endpoint (headproducer.rs).
- attestation store: deterministic AttestationData per (slot, committee),
by-root aggregate lookup, 32-slot trim (attestation.rs).
- builder options: endpoint_overrides, fork_version, sync committee
size/subnet count, no_proposer/attester/sync duties, deterministic
sync committee duties (options.rs + sync duties endpoint).
- fuzzer mode: random JSON for proposal, attestation, duties endpoints
behind a builder flag (fuzzer.rs).
- ValidatorSet: by_public_key, public_keys.
- default spec extended with mainnet keys real validator clients read.
- /eth/v2/beacon/blocks/{id} default endpoint.
Add Rust ports of the seven Go tests not covered by the agents' implementation-focused tests: deterministic attester/proposer duties, TestStatic, genesis_time/slots_per_epoch/slot_duration overrides, and default overrides.
…lidation Port charon's static.json + gen_static.sh approach: a Holesky beacon-node snapshot is committed in the crate and used as the baseline for the default spec; Charon-simnet overrides apply on top. A new build.rs validates the file at compile time (well-formed JSON, required endpoints present, required spec keys present) and triggers rebuilds whenever the snapshot changes — failures surface as compile errors instead of test failures. Drops the hand-curated ~80-key spec list previously inlined in defaults.rs in favor of the real beacon-node snapshot. scripts/gen_static_beaconmock.sh regenerates static.json from a live beacon node (`BEACON_URL=<url> ./scripts/gen_static_beaconmock.sh`), mirroring charon/testutil/beaconmock/gen_static.sh.
Restore the testdata/*.golden files from charon/testutil/beaconmock/ verbatim and switch the deterministic attester/proposer duties tests to golden assertions matching Go's RequireGoldenJSON. Adds a third golden test covering AttestationData for (slot=1, committee_index=2). Fixes a bug surfaced by the AttestationStore golden: the Rust port used saturating_sub(1) for previous_epoch, but Go wraps to u64::MAX at epoch 0 (see charon/testutil/beaconmock/attestation.go newAttestationData). Switch to wrapping_sub so the source.epoch and source.root match Go byte-for-byte.
…le epochs Charon's ValidatorSetA leaves ExitEpoch and WithdrawableEpoch unset, so they serialize as "0". The Rust port was forcing them to FAR_FUTURE_EPOCH (u64::MAX), which is more spec-correct but breaks parity with Charon's mock output.
…use CancellationToken Three intertwined issues are resolved by restructuring HeadProducer lifecycle: - Drop race: switching from Notify::notify_waiters() (which only wakes currently-registered waiters) to CancellationToken (cancel-then-await semantics) ensures shutdown is observed on the next poll regardless of registration timing. - Block-root race: HeadProducer::spawn now generates and stores the initial head event synchronously before returning, so /eth/v1/beacon/blocks/.../root and /eth/v1/events never observe a None current head. The 150ms test sleep workaround is removed. - std::thread::sleep in async handler: with the head guaranteed present, the busy-wait loop in wait_for_first_head is no longer needed and is deleted, eliminating the synchronous block on tokio worker threads.
… duty-root typo Charon's headproducer.go renders both current_duty_dependent_root and previous_duty_dependent_root from the same internal value (a Go typo carried in v1.7.1). Mirror that behavior by reducing HeadEvent to a single duty_dependent_root field and rendering it into both JSON fields. Also document that the Rust port uses ChaCha-based StdRng instead of Go's math/rand LCG. The byte sequences differ but Pluto does not assert on any specific head/state/dependent root value, and ChaCha is portable and well-tested; the head event JSON shape and per-slot determinism are preserved.
The arithmetic_side_effects lint denies + operator on SystemTime; the fallback already used checked_add elsewhere in the same function. Mirror that pattern.
…ties Charon's WithDeterministicProposerDuties iterates over mock.ActiveValidators, which only includes validators with an Active* status. Mirror that by filtering the validator set on validator.status.is_active() before assigning proposer duties. Add a regression test that adds a WithdrawalDone validator to set_a and verifies it is skipped while the three Active validators retain duties.
headproducer.rs had a private hex_0x copy identical to state::hex_0x. Import the shared pub(crate) helper instead. No behavior change.
Three identical parsers (epoch_from_path in defaults.rs, slot_from_path
and epoch_from_path in fuzzer.rs) all extract the trailing /{u64} segment
of a request path. Pull the parse into a single pub(crate) helper in
state.rs and delegate to it from the named wrappers (kept for call-site
readability).
…pect
Two defensive fallbacks were unreachable for inputs the code accepts:
- random_validator_status's unwrap_or("active_ongoing") covered an empty
STATUSES slice, which is a const &[&str; 9].
- default_genesis_time's None => Utc::now() covered DST ambiguity at a UTC
instant where DST does not apply.
Both swap to expect() with a message that explains the invariant, so a future
breakage prints a useful panic instead of silently producing surprising data.
Mirrors `charon/testutil/validatormock` (Go) into Rust. This first phase lays the foundation that the propose/attest/synccomm/component ports build on: - `meta.rs` - SpecMeta + MetaSlot + MetaEpoch value types (port of meta.go). - `sign.rs` - `Sign` trait + `Signer` + `SignFunc = Arc<dyn Sign>` (Go's SignFunc + NewSigner) backed by `pluto-crypto` BlstImpl. - `validators.rs` - local `ActiveValidators` newtype + `active_validators(client)` helper. Avoids a `pluto-testutil -> pluto-app` dep, since `pluto-app` already dev-depends on this crate. - `capture.rs` - test-only `SubmissionCapture` wiremock helper. Equivalent of Go's `beaconMock.SubmitAttestationsFunc = ...` callback fields - mounts a high-priority `Mock` on `BeaconMock::server()` and records POST bodies. - `error.rs` - module-wide `Error` + `SignError` (thiserror). - `testdata/TestAttest_*.golden` - byte-for-byte copies of the Go fixtures used by the Phase-2 attest port. The Cargo cycle `pluto-eth2util <-> pluto-testutil` is broken by moving `pluto-testutil` from `[dependencies]` to `[dev-dependencies]` in `crates/eth2util/Cargo.toml`. The three call sites in `eth2util` (signing, eth2exp, enr tests) are all `#[cfg(test)]`, so the move is a no-op for production code. 11 unit tests pass; clippy + nightly fmt clean.
Ports `charon/testutil/validatormock/synccomm.go` to Rust. SyncCommMember
drives PrepareEpoch → PrepareSlot → Message → Aggregate; per-slot ready flags
use Arc<tokio::sync::OnceCell<()>> to mirror Go's lazy `chan struct{}` maps.
TestGetSubcommittees ports the Go internal test verbatim.
…er reg) Ports `propose_block` and `register` from `charon/testutil/validatormock/propose.go`. Versioned dispatch matches the Go switch on `eth2spec.DataVersion`; blinded path is gated on the proposal's `execution_payload_blinded` flag (the Pluto generated client's analog of Go's `block.Blinded`). Phase0/Altair branches return `Error::UnsupportedVariant` until the Pluto typed surface for them is ready; Bellatrix .. Fulu (full + blinded) are implemented in full. `register` ports the *intended* behaviour of Go's `Register`: it switches on the input registration's `Version` (Go switches on the zero-value-initialized signed registration version, which always falls through to V1 — a Go quirk noted inline). Tests mirror `TestProposeBlock` and `TestProposeBlindedBlock`; variants whose random fixtures don't yet exist in `pluto-testutil::random` are `#[ignore]`d with a TODO pointing at the missing helper.
Ports `charon/testutil/validatormock/attest.go` to Rust. SlotAttester drives
Prepare -> Attest -> Aggregate for a single slot, gated by Arc<CloseOnce>
close-once flags (AtomicBool + Notify) that mirror Go's `chan struct{}` ready
signals; mutable state lives behind Arc<Mutex<_>> so all entry points stay
`&self`.
Because the Rust eth2api client encodes attestations using the Electra
SingleAttestation shape (incompatible with Go's VersionedAttestation JSON
captured in the goldens), the two submit endpoints
(`POST /eth/v2/beacon/pool/attestations` and
`POST /eth/v2/validator/aggregate_and_proofs`) are POSTed as raw JSON whose
structure matches `eth2spec.VersionedAttestation` /
`*SubmitAggregateAttestationsOpts`. All other beacon-node interactions use
the generated client.
TestAttest matches Go's golden output for DutyFactor {0, 1} on validator
set A via assert_json_eq, with submissions sorted by data.index to match
the Go test's deterministic ordering.
Also adds Eth2Exp and Submit variants to validatormock::Error, and a
ValidatorSet-driven POST `/eth/v1/beacon/states/head/validators` plus a
BeaconCommitteeSelections echo route in the tests because the beaconmock
defaults don't cover those DV-only paths.
Ports `charon/testutil/validatormock/component.go` to Rust, completing the validatormock module. `Component` drives the duty-by-duty workflow across a sliding window of attesters (per slot) and sync-committee members (per epoch) for the configured pubkeys. - `clock.rs`: `Clock` trait + `SystemClock` (production) + `FakeClock` (tests). `FakeClock` is an explicit-advance clock; pending sleepers register a `oneshot::Sender` and fire when `advance` / `advance_to` moves past their deadline. Avoids `tokio::time::pause()`, which interacts poorly with `wiremock::MockServer` (Plan agent flagged this). - `component.rs`: scheduler built around `tokio::sync::mpsc` + `tokio::spawn` + `tokio_util::sync::CancellationToken`. `Component::builder()` returns a handle that owns the consumer task; `shutdown().await` cancels gracefully and `Drop` cancels for the panic path. `run_duty_via_inner` matches the Go `runDuty` switch on `pluto_core::types::DutyType`. - `duties_for_slot` + `duty_start_times` mirror Go's `dutyStartTimeFuncsByDuty` table, including the half-/third-slot offsets for attest/aggregate/sync messages. Tests cover the start-up swallow window (delay_start_slots = 2), the duty-start-time arithmetic, and the duty-collection algorithm. Driving the full Component through 2 epochs with FakeClock would require comprehensive post_state_validators mounting on BeaconMock; the slot_ticked test asserts the epoch-window machinery instead and shuts down cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports
charon/testutil/validatormock(Go) to Rust undercrates/testutil/src/validatormock/. Mirrors the 5 Go files file-per-concern with full functional parity:meta.rs—SpecMeta/MetaSlot/MetaEpochslot/epoch arithmeticsign.rs—Signtrait +Signer+SignFunc = Arc<dyn Sign>(Go'sSignFunc+NewSigner)validators.rs— localActiveValidatorsnewtype + helper (avoids apluto-testutil → pluto-appdependency cycle)capture.rs—SubmissionCapturewiremock helper, the test-side replacement for Go'sbeaconMock.SubmitAttestationsFunc = ...callback fieldsattest.rs—SlotAttesterwithPrepare/Attest/Aggregate; matches Go goldens byte-for-byte viaassert_json_diffpropose.rs—propose_block+register; Bellatrix→Fulu, blinded + full variantssynccomm.rs—SyncCommMember+TestGetSubcommitteescomponent.rs— scheduler (tokio::sync::mpsc+CancellationToken+ injectableClock)clock.rs—Clocktrait +SystemClock+ explicit-advanceFakeClockerror.rs— unifiedthiserror::Errortestdata/TestAttest_{0,1}_{attestations,aggregations}.golden— byte-for-byte fromcharon/testutil/validatormock/testdata/Based on
bohdan/beaconmockapi(#416). Will rebase ontomainonce #416 merges.Design notes
close(chan struct{})ready signals are modelled per-module:attest.rsusesAtomicBool+tokio::sync::Notify;synccomm.rsusesArc<tokio::sync::OnceCell<()>>(lazy per-slot map).tokio::spawn+mpsc+CancellationToken.clockwork.FakeClockmaps to an injectableClocktrait.tokio::time::pause()was rejected — it interacts poorly withwiremock::MockServer.pluto-eth2util ↔ pluto-testutilCargo cycle was fixed:pluto-testutilmoved from[dependencies]to[dev-dependencies]incrates/eth2util/Cargo.toml. The 3 call sites in eth2util (signing, eth2exp, enr tests) are all#[cfg(test)], so this is a no-op for production code.Workflow
Built across 4 phases on parallel git worktrees:
Test plan
cargo +nightly fmt --all --check— cleancargo clippy --workspace --all-targets --all-features -- -D warnings— cleancargo test --workspace --all-features— all green; 73 tests pass inpluto-testutil(6#[ignore]d propose tests waiting on missingRandomCapella/Deneb/Electra/Fulu*Proposalhelpers inpluto-testutil::random— follow-up)cargo deny check— cleantestdata/TestAttest_{0,1}_{attestations,aggregations}.goldenbyte-for-byte (structural viaassert_json_diff)Known follow-ups (out of scope)
RandomCapella/Deneb/Electra/Fulu{Versioned,Blinded}Proposalhelpers topluto-testutil::randomto un-ignore 6 propose tests.POST /eth/v1/beacon/states/{state_id}/validatorsmount toBeaconMockso callers don't have to inline it.registerdeviates from Charon'sswitch signedRegistration.Version(always zero-valued V1) and instead switches on the input registration'sVersion. Inline comment notes the Go quirk; please confirm intent.