[IIRR-39] Add archived puzzle seeds for low success rate filter testing#77
Merged
Conversation
JuanVqz
changed the title
arielj
reviewed
May 19, 2026
| 3.times do | ||
| puzzle = Puzzle.all.sample | ||
| puzzle = Puzzle.approved.sample | ||
| next unless puzzle |
Contributor
There was a problem hiding this comment.
can this realistically ever happen? like... if the seed depends on approved puzzles we should be sure we have approved puzzles at this point instead of this conditional
it's also weird to have answers for approved puzzles, in real life we can only have answers for archived puzzles, approved puzzles have not been sent yet, so maybe the seed was always wrong creating answers for puzzles with any state other than archived
Contributor
There was a problem hiding this comment.
like maybe we should remove this block directly
Adds 7 archived puzzles: 5 with random low/mixed answer rates and 2 with 90% correct rate to support manual testing of the low success rate filter without needing production data.
- Remove random answers for approved puzzles. Real users only answer puzzles after they've been sent (archived state). - Add two cloned puzzles so the 'hide cloned puzzles' filter from #78 has data to filter.
Mix two pending clones with two archived clones so the 'hide cloned' filter can be exercised across both states.
The :one fixture is approved with no sent_at, so it's eligible for DailyPuzzleJob. The job picks one approved+unsent puzzle at random, which made this test flaky depending on which puzzle was picked. Clear the eligible pool before creating the puzzle under test, mirroring the pattern in 'does nothing when no approved unsent puzzles exist'.
JuanVqz
force-pushed
the
feature/seed-archived-puzzles
branch
from
0b513b6 to
36deac3
Compare
- Replace high_success boolean with per-puzzle success_rate (20, 40, 50, 70, 80 for low; 90, 100 for high). user_idx < rate/10 makes the seeded correct/incorrect distribution exact across runs. - Append '(clone)' to cloned puzzle questions so they are easy to spot in the archived table during manual testing. - Clones inherit their parent's success_rate via original_puzzle lookup.
Now that each archived puzzle carries an explicit success_rate, the correct-answer distribution no longer needs to ride along inside the user loop. Iterating per puzzle and picking the first N server users per puzzle reads more directly: 'for each puzzle, mark rate/10 users correct'. The user loop is now purely about creating users and linking them to the server.
Both groups already had the same shape; archived just carried extra fields (state, sent_at, success_rate). Folding them into one array removes the duplicated find_or_create_by! loop and gives a single place to add new puzzles. Entries without an explicit state fall back to the column default (:pending), preserving existing behavior.
arielj
approved these changes
May 19, 2026
Contributor
|
looks good! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://ombulabs.atlassian.net/browse/IIRR-39
Summary
db/seeds.rbwith deterministic success rates spanning the last 7 dayscorrect_answer_percentagereturns real datasuccess_rate(20, 40, 50, 70, 80, 90, 100). The seed assignsis_correctviauser_idx < rate / 10, so the resulting percentages are exact across runspending, 2archived) withoriginal_puzzle_idset, so the "Hide cloned puzzles" filter from IIRR-33: Filter out cloned archived puzzles #78 has data to filter. Clone questions are suffixed with(clone)to make them easy to spot during manual testingapprovedpuzzles. In reality only archived puzzles can have answers, so seeding them on approved was wrongContext
Follow-up to #76 (low success rate filter) and #78 (hide cloned puzzles filter). Both filters need archived puzzles with known success rates and a mix of cloned/uncloned originals to be exercised locally.
Also includes a small fix to
DailyPuzzleJobTest#test_marks_the_selected_puzzle_as_archived_and_sets_sent_at: clears any otherapproved+sent_at: nilpuzzles before invoking the job, so the test no longer depends on which eligible puzzle the job picks at random. This was a pre-existing flake exposed once #76 removedsent_atfrom the:onefixture.Test plan
bin/rails db:seed:replant(clone))reduce90% andvalidates100% puzzles disappear)bin/rails testand confirm all tests pass