perf(enrich): Phase A quick wins for OOM fix#145
Merged
Conversation
…ask A1) Each LanguageExtractor.Extract reparsed the source file at its top — on Python at ~13 nodes/file that meant ~13x over-parse. pprof on airflow flagged 91% of total allocations from tree-sitter. (*Tree).cachedNode driven by the per-node re-parse storm. Adds ExtractFromTree(ctx, tree, nodes) []Result to the LanguageExtractor interface. The orchestrator now parses the file once and calls ExtractFromTree(tree, allNodes) — the AST is walked multiple times for distinct node-kinds but never re-parsed. Extract is retained as a thin wrapper for single-node convenience callers and tests. Plan: docs/superpowers/plans/2026-05-13-enrich-oom-fix.md Task A1. Per-file caches: matchAllList (py), matchInterfaceAssertion (go), collectExports (ts) are computed once per file rather than once per matching node. Verification: - go test ./internal/intelligence/extractor/... -count=1: 28 pass - go test ./... -count=1: 875 pass
Previously the enricher spawned one goroutine per source file with no cap. On polyglot Python repos (airflow: 7,456 files) that produced 7k+ concurrent live tree-sitter Trees + file content strings, driving the OOM-prone RSS spike pprof exposed. Adds a semaphore-bounded fan-out at 2*runtime.GOMAXPROCS(0). Tasks still write to indexed slots, so determinism (sorted file path order) is preserved. Polyglot real-world targets see materially lower peak RSS at no measurable wall-time cost. Plan: docs/superpowers/plans/2026-05-13-enrich-oom-fix.md Task A2. Verification: - New TestEnricher_BoundedConcurrency asserts peak in-flight calls <= 2*GOMAXPROCS by driving 4*cap files through a tracking extractor. - go test ./... -count=1: 876 pass.
…sk A3)
kuzu.DefaultSystemConfig() allocates 80% of system RAM as the buffer
pool (~12 GiB on a 15 GiB host) before any enrich work runs. Combined
with Go-side enricher memory that's enough to OOM the process. The
default also allocates full GOMAXPROCS worth of internal threads,
amplifying COPY-side working set.
Adds OpenOptions struct + OpenWithOptions(path, opts). Open(path)
now applies safe defaults via OpenWithOptions(path, OpenOptions{}):
- BufferPoolBytes: 2 GiB (DefaultBufferPoolBytes)
- MaxThreads: min(4, GOMAXPROCS)
OpenReadOnly is unchanged externally (same signature) but routes
through OpenWithOptions internally — read paths inherit the same
buffer pool cap (2 GiB is plenty for read-side caching at our graph
scale).
Plan: docs/superpowers/plans/2026-05-13-enrich-oom-fix.md Task A3.
Future polish: surface --max-buffer-pool and --copy-threads CLI flags
for power-user tuning (deferred).
Verification:
- go test ./internal/graph/... -count=1: 44 pass
- go test ./... -count=1: 876 pass
GraphBuilder.Snapshot extracted deduped nodes/edges into sorted slices but left builder.nodes and builder.edges maps holding references to the same objects. With the slices and maps coexisting for the rest of the enrich pipeline (~30 sec wall time on ~/projects/), ~280 MB of duplicate references stayed live needlessly. Clear the maps inside Snapshot before returning. Snapshot is now single-shot — calling it twice on the same builder returns an empty snapshot (acceptable; the only caller is analyzer.Enrich which calls once). Plan: docs/superpowers/plans/2026-05-13-enrich-oom-fix.md Task A4. Verification: - New TestSnapshotReleasesDedupMaps asserts both nodes + edges maps are nilled after Snapshot returns. - go test ./... -count=1: 876 pass (no regressions).
This was referenced May 13, 2026
aksOps
added a commit
that referenced
this pull request
May 14, 2026
Stale doc references after Phase 6 (Java deletion, #132) and the Kuzu 0.7.1 → 0.11.3 bump (#155 + #159). - CLAUDE.md / PROJECT_SUMMARY.md: bump Kuzu 0.7.1 → 0.11.3, go-sqlite3 1.14.22 → 1.14.44, cobra to 1.10.2; note native FTS. - AGENTS.md: rewrite "What this repo is" (no more "REST API"); flip `mvn -B -ntp clean verify` → `go test ./...`; clarify that REST + React SPA were deleted in Phase 6 and won't return. - SECURITY.md: rewrite scope. Drop the dead JAR / serve / REST API / React UI / H2 / Neo4j Embedded references. New in-scope list covers every codeiq subcommand, the 10 MCP tools (with `run_cypher` mutation gate called out), `.codeiq/cache/` (SQLite) + `.codeiq/graph/` (Kuzu), and `read_file` path sandboxing. Add the security CI workflows (CodeQL, Semgrep, OSV-Scanner, Trivy, Gitleaks, SBOM, Socket Security) + perf-gate to the hardening references. - CHANGELOG.md: populate [Unreleased] with the OOM-fix saga (PRs #145-#148), the five correctness fixes (#149-#153), the Kuzu 0.7.1 → 0.11.3 bump (#155-#158), the FTS migration (#159), the Dependabot config rewrite (#154), and the enrich CLI knobs. No code changes. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase A of the enrich OOM fix plan (docs/superpowers/plans/2026-05-13-enrich-oom-fix.md). Four surgical fixes that target the actual hot spots pprof exposed on a real-world polyglot Python target (airflow, 9,151 files → 3.8 GB peak RSS; trajectory extrapolates to OOM at ~/projects/ scale).
Tasks landed
Expected impact
Per the plan's success criterion: `~/projects/` peak RSS should drop from 9-15 GB (OOM-killed at exit 137) to ~2-4 GB. Real-world verification will run once Phase B + C land too.
Test plan
Next phases
This PR is Phase A of 4 from the plan. Phase B (TreeCursor migration), Phase C (streaming three-pass refactor), Phase D (perf-gate CI + real-world acceptance) ship as separate PRs after A merges.