feat(ui-automation): Add rs/1 runtime automation parity#416
Open
cameroncooke wants to merge 45 commits into
Open
feat(ui-automation): Add rs/1 runtime automation parity#416cameroncooke wants to merge 45 commits into
cameroncooke wants to merge 45 commits into
Conversation
commit: |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Validation artifacts with local paths committed to repo
- Removed entire out.nosync/ directory from git tracking and added it to .gitignore to prevent future commits.
- ✅ Fixed: Plan document embeds developer-local filesystem paths
- Replaced all machine-specific absolute paths with portable references or removed them entirely from the plan document.
Or push these changes by commenting:
@cursor push 07a7505ffa
Preview (07a7505ffa)
diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -116,3 +116,6 @@
.derivedData
/.pr-learning
/repros
+
+# Validation artifacts
+out.nosync/
diff --git a/docs/plans/ui-automation-agent-optimization-2026-05-13.md b/docs/plans/ui-automation-agent-optimization-2026-05-13.md
--- a/docs/plans/ui-automation-agent-optimization-2026-05-13.md
+++ b/docs/plans/ui-automation-agent-optimization-2026-05-13.md
@@ -12,7 +12,7 @@
- `wait_for_ui` polls snapshots and records the latest usable runtime snapshot while waiting (`src/mcp/tools/ui-automation/wait_for_ui.ts`).
- Prior art: `04b055210b111c8a2d70bdf5c19ca4c6b0d2a479` added RS/1 runtime automation parity, including batch execution, wait predicates, runtime refs, and screen-hash unchanged responses.
-Transcript evidence from `/Users/cameroncooke/Desktop/out.nosync`:
+Transcript evidence from validation run:
- Settings toggles were handled as repeated `tap -> snapshot_ui -> tap` loops (`0058` through `0087`) instead of one snapshot plus one batch.
- Batch syntax was guessed and failed (`tap e7`, `tap --element-ref e7`, `tap-target e21`) before the agent fell back to coordinates (`0281` through `0286`, `0312` through `0327`).
- Scroll and sheet expansion produced repeated gesture/swipe/screenshot/snapshot loops (`0110` through `0144`).
@@ -83,9 +83,9 @@
Use Claude Code, not Codex, for validation runs. Configure Claude Code to use the local source-built XcodeBuildMCP server, then repeat the Weather-app task from `0001_user_message.md`.
Reference commands and artifacts:
-- Existing parser: `/Volumes/Developer/parse_claude_conversation.py`
-- Example source conversation: `/Users/cameroncooke/.claude/projects/-Volumes-Developer-XcodeBuildMCP-example-projects-Weather/b3d6cae2-e274-4a72-92ea-25eaf4f6fcff.jsonl`
-- Export command example: `python3 /Volumes/Developer/parse_claude_conversation.py <claude-jsonl>`
+- Existing parser: `parse_claude_conversation.py`
+- Example source conversation: `.claude/projects/<project-path>/<session-id>.jsonl`
+- Export command example: `python3 parse_claude_conversation.py <claude-jsonl>`
Acceptance signals:
- The settings-toggle sequence is reduced to one `snapshot_ui`, one structured `batch`, and at most one verification call.
@@ -106,11 +106,11 @@
## Validation Results
-Validation artifacts were written under `/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522` with timestamped names. Prior Claude transcripts/exports were not deleted or overwritten.
+Validation artifacts were written under `out.nosync/validation-ui-automation-20260513-215522` with timestamped names. Prior Claude transcripts/exports were not deleted or overwritten.
### Automated checks
-All checks below were run with `XCODEBUILDMCP_AXE_SOURCE_PATH=/Volumes/Developer/AXe`:
+All checks below were run with `XCODEBUILDMCP_AXE_SOURCE_PATH` pointing to the AXe source build:
- Focused UI automation/config/factory tests: `npx vitest run src/mcp/tools/ui-automation/__tests__/batch.test.ts src/mcp/tools/ui-automation/__tests__/runtime-snapshot.test.ts src/mcp/tools/ui-automation/__tests__/snapshot_ui.test.ts src/mcp/tools/ui-automation/__tests__/tap.test.ts src/mcp/tools/ui-automation/__tests__/wait_for_ui.test.ts src/utils/__tests__/axe-helpers.test.ts src/utils/__tests__/config-store.test.ts src/utils/__tests__/project-config.test.ts src/utils/__tests__/session-aware-tool-factory.test.ts src/utils/responses/__tests__/next-steps-renderer.test.ts` passed: 10 files, 192 tests (`focused-vitest-20260513-215522.log`).
- `npm run lint` passed (`lint-20260513-215539.log`).
@@ -118,7 +118,7 @@
- `npm run typecheck` passed (`typecheck-20260513-215539.log`).
- `npm run build` passed (`build-20260513-215553.log`).
- `npm run test` passed: 186 files, 2049 tests (`test-20260513-215553.log`).
-- Post-review fixes were applied for switch batch delay handling and selector-scoped `gone` text waits. Final checks after those fixes passed: `npm run lint`, `npm run format:check`, `npm run typecheck`, `npm run build`, and `npm run test` (186 files, 2052 tests). The final full-test output was preserved at `/tmp/xcodebuildmcp-final-npm-test-20260513.txt`.
+- Post-review fixes were applied for switch batch delay handling and selector-scoped `gone` text waits. Final checks after those fixes passed: `npm run lint`, `npm run format:check`, `npm run typecheck`, `npm run build`, and `npm run test` (186 files, 2052 tests).
### AXe source-build proof
@@ -127,18 +127,18 @@
```json
{
"resolved": {
- "path": "/Volumes/Developer/AXe/.build/release/axe",
+ "path": "<axe-source-path>/.build/release/axe",
"source": "source"
},
"bundledEnvironment": {}
}-The resolved binary reported version staging-main-31-510d4df-dirty, proving the validation used the /Volumes/Developer/AXe source build rather than bundled or PATH fallback.
+The resolved binary reported version staging-main-31-510d4df-dirty, proving the validation used a local AXe source build rather than bundled or PATH fallback.
Claude Code E2E
-Claude Code ran the full original 18-step Weather/Safari task from /Users/cameroncooke/Desktop/out.nosync/0001_user_message.md against the local source-built XcodeBuildMCP server:
+Claude Code ran the full original 18-step Weather/Safari task against the local source-built XcodeBuildMCP server:
- MCP config:
claude-mcp-config-20260513-215817.json - Prompt:
claude-weather-safari-prompt-20260513-215817.md
@@ -153,9 +153,9 @@
{
"command": "node",
- "args": ["/Volumes/Developer/XcodeBuildMCP/build/cli.js", "mcp"],
+ "args": ["<xcodebuildmcp-path>/build/cli.js", "mcp"],
"env": {
- "XCODEBUILDMCP_AXE_SOURCE_PATH": "/Volumes/Developer/AXe",
+ "XCODEBUILDMCP_AXE_SOURCE_PATH": "<axe-source-path>",
"XCODEBUILDMCP_SENTRY_DISABLED": "1"
}
}
@@ -181,8 +181,5 @@
- Safari WebView contents, cookie/sign-in UI, and BBC in-page links were not exposed as tappable RS/1 targets, so Claude used the URL bar for Sport, Premier League, tables, and Brighton. This reached the equivalent end state but was not a real row/link click.
## References
-- Transcript folder: `/Users/cameroncooke/Desktop/out.nosync`
-- Validation artifact folder: `/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522`
-- Parser: `/Volumes/Developer/parse_claude_conversation.py`
-- AXe workspace: `/Volumes/Developer/AXe`
+- Validation artifact folder: `out.nosync/validation-ui-automation-20260513-215522`
- Prior RS/1 commit: `04b055210b111c8a2d70bdf5c19ca4c6b0d2a479`
diff --git a/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json b/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json
deleted file mode 100644
--- a/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json
+++ /dev/null
@@ -1,1008 +1,0 @@
-{
- "jsonl": "/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522/claude-session-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-215817.jsonl",
- "total_lines": 425,
- "total_tool_calls": 138,
- "all_tool_counts": {
- "ToolSearch": 3,
- "mcp__xcodebuildmcp-dev__session_show_defaults": 1,
- "mcp__xcodebuildmcp-dev__build_run_sim": 2,
- "mcp__xcodebuildmcp-dev__snapshot_ui": 21,
- "mcp__xcodebuildmcp-dev__wait_for_ui": 23,
- "mcp__xcodebuildmcp-dev__screenshot": 19,
- "Bash": 2,
- "TodoWrite": 14,
- "mcp__xcodebuildmcp-dev__stop_app_sim": 1,
- "mcp__xcodebuildmcp-dev__launch_app_sim": 1,
- "mcp__xcodebuildmcp-dev__swipe": 15,
- "mcp__xcodebuildmcp-dev__tap": 16,
- "mcp__xcodebuildmcp-dev__batch": 1,
- "mcp__xcodebuildmcp-dev__gesture": 2,
- "Grep": 3,
- "Read": 2,
- "mcp__xcodebuildmcp-dev__type_text": 6,
- "mcp__xcodebuildmcp-dev__button": 1,
- "mcp__xcodebuildmcp-dev__key_press": 5
- },
- "xcodebuildmcp_tool_counts": {
- "session_show_defaults": 1,
- "build_run_sim": 2,
- "snapshot_ui": 21,
- "wait_for_ui": 23,
- "screenshot": 19,
- "stop_app_sim": 1,
- "launch_app_sim": 1,
- "swipe": 15,
- "tap": 16,
- "batch": 1,
- "gesture": 2,
- "type_text": 6,
- "button": 1,
- "key_press": 5
- },
- "batch_calls": [
- {
- "ordinal": 33,
- "timestamp": "2026-05-13T21:01:15.625Z",
- "id": "toolu_01FsZYoJ6ddwjVLWKMwYoyU6",
- "kind": "structured",
- "step_count": 7,
- "steps": [
- {
- "action": "tap",
- "elementRef": "e130",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e133",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e137",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e140",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e142",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e145",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e148",
- "postDelay": 0.25
- }
- ],
- "result": {
- "line": 112,
- "timestamp": "2026-05-13T21:01:20.872Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null,
- "structured": null,
- "content": "{\"schema\":\"xcodebuildmcp.output.ui-action-result\",\"schemaVersion\":\"2\",\"didError\":false,\"error\":null,\"data\":{\"summary\":{\"status\":\"SUCCEEDED\"},\"action\":{\"type\":\"batch\",\"stepCount\":7},\"artifacts\":{\"simulatorId\":\"5213C8D8-61D0-4CD7-B468-B463C6206C7D\"}}}"
- }
- }
- ],
- "raw_string_batch_attempts": [],
- "errors": [],
- "settings_window_calls": [
- {
- "ordinal": 20,
- "timestamp": "2026-05-13T21:00:04.466Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e7",
- "direction": "up",
- "distance": 800
- },
- "result": {
- "timestamp": "2026-05-13T21:00:06.036Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 21,
- "timestamp": "2026-05-13T21:00:08.085Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:00:08.545Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 22,
- "timestamp": "2026-05-13T21:00:11.517Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:13.199Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 23,
- "timestamp": "2026-05-13T21:00:20.754Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "in_progress",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "pending",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "pending",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "pending",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
- {
- "content": "Go to tables",
- "status": "pending",
- "activeForm": "Going to tables"
- },
- {
- "content": "Click Brighton",
- "status": "pending",
- "activeForm": "Clicking Brighton"
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:00:20.762Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 24,
- "timestamp": "2026-05-13T21:00:23.387Z",
- "name": "mcp__xcodebuildmcp-dev__tap",
- "input": {
- "elementRef": "e113"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:24.176Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 25,
- "timestamp": "2026-05-13T21:00:26.779Z",
- "name": "mcp__xcodebuildmcp-dev__wait_for_ui",
- "input": {
- "predicate": "settled",
- "settledDurationMs": 500,
- "timeoutMs": 5000
- },
- "result": {
- "timestamp": "2026-05-13T21:00:28.153Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 26,
- "timestamp": "2026-05-13T21:00:33.949Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e123",
- "direction": "up",
- "distance": 400
- },
- "result": {
- "timestamp": "2026-05-13T21:00:35.629Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 27,
- "timestamp": "2026-05-13T21:00:37.754Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:00:38.303Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 28,
- "timestamp": "2026-05-13T21:00:42.712Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:43.315Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 29,
- "timestamp": "2026-05-13T21:00:49.819Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e115",
- "direction": "up",
- "distance": 500
- },
- "result": {
- "timestamp": "2026-05-13T21:00:51.550Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 30,
- "timestamp": "2026-05-13T21:00:54.206Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:54.821Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 31,
- "timestamp": "2026-05-13T21:01:01.034Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:01:01.575Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 32,
- "timestamp": "2026-05-13T21:01:11.819Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "completed",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "in_progress",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "pending",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "pending",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
- {
- "content": "Go to tables",
- "status": "pending",
- "activeForm": "Going to tables"
- },
- {
- "content": "Click Brighton",
- "status": "pending",
- "activeForm": "Clicking Brighton"
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:01:11.822Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 33,
- "timestamp": "2026-05-13T21:01:15.625Z",
- "name": "mcp__xcodebuildmcp-dev__batch",
- "input": {
- "steps": [
- {
- "action": "tap",
- "elementRef": "e130",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e133",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e137",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e140",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e142",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e145",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e148",
- "postDelay": 0.25
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:01:20.872Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 34,
- "timestamp": "2026-05-13T21:01:23.163Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:01:23.739Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 35,
- "timestamp": "2026-05-13T21:01:27.506Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:01:28.031Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 36,
- "timestamp": "2026-05-13T21:01:31.879Z",
- "name": "mcp__xcodebuildmcp-dev__tap",
- "input": {
- "elementRef": "e126"
- },
- "result": {
- "timestamp": "2026-05-13T21:01:32.759Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 37,
- "timestamp": "2026-05-13T21:01:35.902Z",
- "name": "mcp__xcodebuildmcp-dev__wait_for_ui",
- "input": {
- "predicate": "gone",
- "text": "Settings",
- "timeoutMs": 3000
- },
- "result": {
- "timestamp": "2026-05-13T21:01:39.013Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 38,
- "timestamp": "2026-05-13T21:01:49.208Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "completed",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "completed",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "completed",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "in_progress",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
... diff truncated: showing 800 of 12059 linesYou can send follow-ups to the cloud agent here.
Add batch execution, wait predicates, runtime snapshot refs, and screen-hash unchanged responses so agents can drive AXe with fewer process launches and less repeated snapshot output. Tighten action validation, stale-snapshot recovery, compact rendering, and fixture coverage so UI automation flows are easier for agents to use reliably. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Return compact runtime snapshot captures after UI actions and derive next-step guidance from foreground runtime elements. This reduces reliance on full snapshots and screenshot verification loops while preserving structured output contracts. Co-Authored-By: Codex <noreply@openai.com>
Add coverage for runtime captures, post-action snapshot behavior, compact next-step rendering, and session-aware structured output handling. Co-Authored-By: Codex <noreply@openai.com>
Record the runtime snapshot and next-step guidance improvements for the upcoming release. Co-Authored-By: Codex <noreply@openai.com>
cameroncooke
force-pushed
the
cam/feat/ui-automation-runtime-parity
branch
from
c0a5d33 to
15edeec
Compare
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: SIMULATOR_UUID placeholders leak in wait_for_ui manifest
- Removed static nextSteps block from wait_for_ui.yaml to eliminate SIMULATOR_UUID placeholder leak, matching the fix already applied to snapshot_ui.yaml.
Or push these changes by commenting:
@cursor push 46bb7686f9
Preview (46bb7686f9)
diff --git a/manifests/tools/wait_for_ui.yaml b/manifests/tools/wait_for_ui.yaml
--- a/manifests/tools/wait_for_ui.yaml
+++ b/manifests/tools/wait_for_ui.yaml
@@ -9,18 +9,6 @@
version: '2'
routing:
stateful: true
-nextSteps:
- - label: Refresh runtime snapshot
- toolId: snapshot_ui
- params:
- simulatorId: SIMULATOR_UUID
- when: success
- - label: Wait again
- toolId: wait_for_ui
- params:
- simulatorId: SIMULATOR_UUID
- predicate: settled
- when: success
annotations:
title: Wait for UI
readOnlyHint: trueYou can send follow-ups to the cloud agent here.
Allow existing coordinate-based UI action payloads and UI hierarchy captures in the v2 schemas. Avoid publishing wait-for-ui next-step params when the wait fails, and remove an unused simulator test import. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Schema relaxes video FPS from positive integer to any number
- Restored integer type constraint and minimum: 1 validation to the fps field in videoRecordingCapture schema definition.
Or push these changes by commenting:
@cursor push 5e661b4638
Preview (5e661b4638)
diff --git a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
@@ -183,7 +183,7 @@
"properties": {
"type": { "const": "video-recording" },
"state": { "enum": ["started", "stopped"] },
- "fps": { "type": "number" },
+ "fps": { "type": "integer", "minimum": 1 },
"outputFile": { "type": "string" },
"sessionId": { "type": "string" }
},You can send follow-ups to the cloud agent here.
Remove the wait_for_ui manifest nextSteps block so the tool relies on dynamic next-step params instead of leaking SIMULATOR_UUID placeholders. Co-Authored-By: Codex <noreply@openai.com>
Restore the capture-result schema fps constraint to a positive integer while keeping the UI hierarchy capture additions intact. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Redundant
stylevariable duplicates existingoutputStylecheck- Removed redundant
stylevariable and replaced usage withoutputStyle !== 'minimal'at line 412.
- Removed redundant
- ✅ Fixed: NaN silently passes through number array coercion for trailing commas
- Modified
coerceNumberArrayto filter out empty strings instead of converting them to NaN, preventing silent NaN propagation.
- Modified
Or push these changes by commenting:
@cursor push 52baba93d8
Preview (52baba93d8)
diff --git a/src/cli/__tests__/schema-to-yargs.test.ts b/src/cli/__tests__/schema-to-yargs.test.ts
--- a/src/cli/__tests__/schema-to-yargs.test.ts
+++ b/src/cli/__tests__/schema-to-yargs.test.ts
@@ -37,6 +37,6 @@
expect(coerce?.('23,18,14')).toEqual([23, 18, 14]);
expect(coerce?.('23, 18, 14')).toEqual([23, 18, 14]);
expect(coerce?.(['23', '18,14'])).toEqual([23, 18, 14]);
- expect(coerce?.('23,')).toEqual([23, Number.NaN]);
+ expect(coerce?.('23,')).toEqual([23]);
});
});
diff --git a/src/cli/register-tool-commands.ts b/src/cli/register-tool-commands.ts
--- a/src/cli/register-tool-commands.ts
+++ b/src/cli/register-tool-commands.ts
@@ -298,7 +298,6 @@
const outputStyle: OutputStyle = argv.style === 'minimal' ? 'minimal' : 'normal';
const socketPath = argv.socket as string;
const logLevel = argv['log-level'] as string | undefined;
- const style = argv.style as string | undefined;
const filePathRenderStyle = argv.filePathRenderStyle as FilePathRenderStyle | undefined;
const verboseOutput = argv.verbose === true;
@@ -409,7 +408,7 @@
runtime: 'cli',
outputStyle,
filePathRenderStyle,
- includeNextSteps: style !== 'minimal',
+ includeNextSteps: outputStyle !== 'minimal',
});
const writeJsonlFragment =
outputFormat === 'jsonl'
diff --git a/src/cli/schema-to-yargs.ts b/src/cli/schema-to-yargs.ts
--- a/src/cli/schema-to-yargs.ts
+++ b/src/cli/schema-to-yargs.ts
@@ -13,7 +13,8 @@
String(entry)
.split(',')
.map((item) => item.trim())
- .map((item) => (item === '' ? Number.NaN : Number(item))),
+ .filter((item) => item !== '')
+ .map((item) => Number(item)),
);
}You can send follow-ups to the cloud agent here.
Remove the redundant argv.style alias and use the normalized outputStyle value when deciding whether to include CLI next steps. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Duplicated schema definitions across two JSON schema files
- Extracted 7 shared type definitions to common.schema.json and updated both schemas to reference them, eliminating duplication.
Or push these changes by commenting:
@cursor push f0e0e2782a
Preview (f0e0e2782a)
diff --git a/schemas/structured-output/_defs/common.schema.json b/schemas/structured-output/_defs/common.schema.json
--- a/schemas/structured-output/_defs/common.schema.json
+++ b/schemas/structured-output/_defs/common.schema.json
@@ -564,6 +564,124 @@
"minLength": 1
},
"minItems": 1
+ },
+ "frame": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "x": { "type": "number" },
+ "y": { "type": "number" },
+ "width": { "type": "number" },
+ "height": { "type": "number" }
+ },
+ "required": ["x", "y", "width", "height"]
+ },
+ "runtimeActionName": {
+ "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
+ },
+ "runtimeElementRole": {
+ "enum": [
+ "application",
+ "button",
+ "cell",
+ "image",
+ "keyboard-key",
+ "list",
+ "menu",
+ "other",
+ "scroll-view",
+ "slider",
+ "switch",
+ "tab",
+ "text",
+ "text-field",
+ "window"
+ ]
+ },
+ "runtimeElementState": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "enabled": { "type": "boolean" },
+ "focused": { "type": "boolean" },
+ "selected": { "type": "boolean" },
+ "visible": { "type": "boolean" }
+ }
+ },
+ "runtimeElement": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
+ "role": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElementRole" },
+ "label": { "type": "string" },
+ "value": { "type": "string" },
+ "identifier": { "type": "string" },
+ "frame": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/frame" },
+ "state": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElementState" },
+ "actions": {
+ "type": "array",
+ "items": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeActionName" }
+ }
+ },
+ "required": ["ref", "frame", "actions"]
+ },
+ "compactRuntimeSnapshot": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "type": { "const": "runtime-snapshot" },
+ "rs": { "const": "1" },
+ "screenHash": { "type": "string", "minLength": 1 },
+ "seq": { "type": "integer", "minimum": 0 },
+ "count": { "type": "integer", "minimum": 0 },
+ "targets": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "scroll": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "text": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "udid": { "type": "string" }
+ },
+ "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
+ },
+ "recoverableUiError": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "code": {
+ "enum": [
+ "SNAPSHOT_MISSING",
+ "SNAPSHOT_EXPIRED",
+ "SNAPSHOT_PARSE_FAILED",
+ "ELEMENT_REF_NOT_FOUND",
+ "TARGET_NOT_FOUND",
+ "TARGET_AMBIGUOUS",
+ "TARGET_NOT_ACTIONABLE",
+ "WAIT_TIMEOUT",
+ "UI_STATE_CHANGED",
+ "ACTION_FAILED"
+ ]
+ },
+ "message": { "type": "string" },
+ "recoveryHint": { "type": "string" },
+ "elementRef": { "type": "string" },
+ "candidates": {
+ "type": "array",
+ "items": {
+ "oneOf": [{ "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }, { "type": "string" }]
+ }
+ },
+ "snapshotAgeMs": { "type": "integer", "minimum": 0 },
+ "timeoutMs": { "type": "integer", "minimum": 0 }
+ },
+ "required": ["code", "message", "recoveryHint"]
}
}
}
diff --git a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
@@ -9,17 +9,6 @@
}
],
"$defs": {
- "frame": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "x": { "type": "number" },
- "y": { "type": "number" },
- "width": { "type": "number" },
- "height": { "type": "number" }
- },
- "required": ["x", "y", "width", "height"]
- },
"uiHierarchyNode": {
"type": "object"
},
@@ -35,61 +24,11 @@
},
"required": ["type", "uiHierarchy"]
},
- "runtimeActionName": {
- "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
- },
- "runtimeElementRole": {
- "enum": [
- "application",
- "button",
- "cell",
- "image",
- "keyboard-key",
- "list",
- "menu",
- "other",
- "scroll-view",
- "slider",
- "switch",
- "tab",
- "text",
- "text-field",
- "window"
- ]
- },
- "runtimeElementState": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "enabled": { "type": "boolean" },
- "focused": { "type": "boolean" },
- "selected": { "type": "boolean" },
- "visible": { "type": "boolean" }
- }
- },
- "runtimeElement": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
- "role": { "$ref": "#/$defs/runtimeElementRole" },
- "label": { "type": "string" },
- "value": { "type": "string" },
- "identifier": { "type": "string" },
- "frame": { "$ref": "#/$defs/frame" },
- "state": { "$ref": "#/$defs/runtimeElementState" },
- "actions": {
- "type": "array",
- "items": { "$ref": "#/$defs/runtimeActionName" }
- }
- },
- "required": ["ref", "frame", "actions"]
- },
"runtimeActionHint": {
"type": "object",
"additionalProperties": false,
"properties": {
- "action": { "$ref": "#/$defs/runtimeActionName" },
+ "action": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeActionName" },
"elementRef": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
"label": { "type": "string" }
},
@@ -108,7 +47,7 @@
"expiresAtMs": { "type": "integer", "minimum": 0 },
"elements": {
"type": "array",
- "items": { "$ref": "#/$defs/runtimeElement" }
+ "items": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }
},
"actions": {
"type": "array",
@@ -127,31 +66,6 @@
"actions"
]
},
- "compactRuntimeSnapshot": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "type": { "const": "runtime-snapshot" },
- "rs": { "const": "1" },
- "screenHash": { "type": "string", "minLength": 1 },
- "seq": { "type": "integer", "minimum": 0 },
- "count": { "type": "integer", "minimum": 0 },
- "targets": {
- "type": "array",
- "items": { "type": "string" }
- },
- "scroll": {
- "type": "array",
- "items": { "type": "string" }
- },
- "text": {
- "type": "array",
- "items": { "type": "string" }
- },
- "udid": { "type": "string" }
- },
- "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
- },
"runtimeSnapshotUnchanged": {
"type": "object",
"additionalProperties": false,
@@ -200,43 +114,11 @@
"matches": {
"type": "array",
"items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
+ "oneOf": [{ "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }, { "type": "string" }]
}
}
},
"required": ["predicate", "matches"]
- },
- "recoverableUiError": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "code": {
- "enum": [
- "SNAPSHOT_MISSING",
- "SNAPSHOT_EXPIRED",
- "SNAPSHOT_PARSE_FAILED",
- "ELEMENT_REF_NOT_FOUND",
- "TARGET_NOT_FOUND",
- "TARGET_AMBIGUOUS",
- "TARGET_NOT_ACTIONABLE",
- "WAIT_TIMEOUT",
- "UI_STATE_CHANGED",
- "ACTION_FAILED"
- ]
- },
- "message": { "type": "string" },
- "recoveryHint": { "type": "string" },
- "elementRef": { "type": "string" },
- "candidates": {
- "type": "array",
- "items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
- }
- },
- "snapshotAgeMs": { "type": "integer", "minimum": 0 },
- "timeoutMs": { "type": "integer", "minimum": 0 }
- },
- "required": ["code", "message", "recoveryHint"]
}
},
"properties": {
@@ -274,7 +156,7 @@
},
{ "$ref": "#/$defs/uiHierarchyCapture" },
{ "$ref": "#/$defs/runtimeSnapshot" },
- { "$ref": "#/$defs/compactRuntimeSnapshot" },
+ { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/compactRuntimeSnapshot" },
{ "$ref": "#/$defs/videoRecordingCapture" },
{ "$ref": "#/$defs/runtimeSnapshotUnchanged" },
{ "$ref": "#/$defs/compactRuntimeSnapshotUnchanged" }
@@ -283,7 +165,7 @@
"diagnostics": {
"$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/basicDiagnostics"
},
- "uiError": { "$ref": "#/$defs/recoverableUiError" },
+ "uiError": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/recoverableUiError" },
"waitMatch": { "$ref": "#/$defs/waitMatch" }
},
"required": ["summary", "artifacts"]
diff --git a/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
@@ -9,92 +9,6 @@
}
],
"$defs": {
- "frame": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "x": { "type": "number" },
- "y": { "type": "number" },
- "width": { "type": "number" },
- "height": { "type": "number" }
- },
- "required": ["x", "y", "width", "height"]
- },
- "runtimeActionName": {
- "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
- },
- "runtimeElementRole": {
- "enum": [
- "application",
- "button",
- "cell",
- "image",
- "keyboard-key",
- "list",
- "menu",
- "other",
- "scroll-view",
- "slider",
- "switch",
- "tab",
- "text",
- "text-field",
- "window"
- ]
- },
- "runtimeElementState": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "enabled": { "type": "boolean" },
- "focused": { "type": "boolean" },
- "selected": { "type": "boolean" },
- "visible": { "type": "boolean" }
- }
- },
- "runtimeElement": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
- "role": { "$ref": "#/$defs/runtimeElementRole" },
- "label": { "type": "string" },
- "value": { "type": "string" },
- "identifier": { "type": "string" },
- "frame": { "$ref": "#/$defs/frame" },
- "state": { "$ref": "#/$defs/runtimeElementState" },
- "actions": {
- "type": "array",
- "items": { "$ref": "#/$defs/runtimeActionName" }
- }
- },
- "required": ["ref", "frame", "actions"]
- },
- "compactRuntimeSnapshot": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "type": { "const": "runtime-snapshot" },
- "rs": { "const": "1" },
- "screenHash": { "type": "string", "minLength": 1 },
- "seq": { "type": "integer", "minimum": 0 },
- "count": { "type": "integer", "minimum": 0 },
- "targets": {
- "type": "array",
- "items": { "type": "string" }
- },
- "scroll": {
- "type": "array",
- "items": { "type": "string" }
- },
- "text": {
- "type": "array",
- "items": { "type": "string" }
- },
- "udid": { "type": "string" }
- },
- "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
- },
"point": {
"type": "object",
"additionalProperties": false,
@@ -106,38 +20,6 @@
},
"direction": {
"enum": ["up", "down", "left", "right"]
- },
- "recoverableUiError": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "code": {
- "enum": [
- "SNAPSHOT_MISSING",
- "SNAPSHOT_EXPIRED",
- "SNAPSHOT_PARSE_FAILED",
- "ELEMENT_REF_NOT_FOUND",
- "TARGET_NOT_FOUND",
- "TARGET_AMBIGUOUS",
- "TARGET_NOT_ACTIONABLE",
- "WAIT_TIMEOUT",
- "UI_STATE_CHANGED",
- "ACTION_FAILED"
- ]
- },
- "message": { "type": "string" },
- "recoveryHint": { "type": "string" },
- "elementRef": { "type": "string" },
- "candidates": {
- "type": "array",
- "items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
- }
- },
- "snapshotAgeMs": { "type": "integer", "minimum": 0 },
- "timeoutMs": { "type": "integer", "minimum": 0 }
- },
- "required": ["code", "message", "recoveryHint"]
}
},
"properties": {
@@ -333,8 +215,8 @@
"diagnostics": {
"$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/basicDiagnostics"
},
- "capture": { "$ref": "#/$defs/compactRuntimeSnapshot" },
- "uiError": { "$ref": "#/$defs/recoverableUiError" }
+ "capture": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/compactRuntimeSnapshot" },
+ "uiError": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/recoverableUiError" }
},
"required": ["summary", "action", "artifacts"]
},You can send follow-ups to the cloud agent here.
There was a problem hiding this comment.
key_press and key_sequence missing routing.stateful: true, bypassing daemon in CLI mode
key_press.yaml and key_sequence.yaml omit routing: stateful: true, so CLI invocations take the direct path instead of routing through the daemon — inconsistent with every other UI-interaction tool in this PR (tap, touch, long_press, swipe, type_text, batch, snapshot_ui, wait_for_ui) which all declare routing.stateful: true.
Verification
Ran grep for routing: across all manifests/tools/*.yaml. key_press.yaml and key_sequence.yaml have no routing block. All other stateful UI automation tools introduced or modified in this PR (batch, tap, long_press, swipe, touch, type_text, snapshot_ui, wait_for_ui) include routing:\n stateful: true. In tool-invoker.ts executeTool(), the daemon branch is entered only when opts.runtime === 'cli' && tool.stateful. Without the manifest flag, tool.stateful is falsy and the CLI falls through to the direct invocation path, silently bypassing the daemon for HID key operations that require a live simulator session — violating the 'Stateful CLI tools route through daemon only when routing.stateful requires it' guardrail.
Identified by Warden xcodebuildmcp-runtime-boundary-review
Keep ref-based runtime UI actions as the public input shape while carrying resolved coordinates in structured action output when they are available. This preserves the v2 action payload contract without reintroducing coordinate-driven tool inputs. Co-Authored-By: Codex <noreply@openai.com>
Restore regression coverage for SwiftPM progress sorting, volatile build settings normalization, and compact frame formatting. These behaviors are still implemented by the production normalizer and need explicit tests. Co-Authored-By: Codex <noreply@openai.com>
Remove internal validation detail from the changelog, preserve existing duration validation compatibility for button and key press actions, and simplify UI wait predicate evaluation for readability. Co-Authored-By: Codex <noreply@openai.com>
Move the type_text changelog entry to the added section and keep user-facing notes free of internal runtime schema names. Preserve AXe fallback diagnostics when command output is empty and simplify small helper paths flagged by Warden. Co-Authored-By: Codex <noreply@openai.com>
Preserve zero-value compatibility for gesture duration and delta validation while keeping negative values rejected. Remove internal runtime snapshot wording from the public swipe manifest. Keep touch success text stable when the optional event is omitted and simplify runtime snapshot control flow. Co-Authored-By: Codex <noreply@openai.com>
Keep the swipe manifest description app-agnostic and free of runtime snapshot terminology. Use a neutral reserved email address in runtime snapshot fixtures to avoid personal-looking test data. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Weather app hardcoded to mock, README still documents launch arg
- Restored conditional logic in WeatherApp.swift to check CommandLine.arguments for --mock-weather-api flag, allowing production weather service to be used when the argument is not present.
Or push these changes by commenting:
@cursor push c819d28354
Preview (c819d28354)
diff --git a/example_projects/Weather/Weather/WeatherApp.swift b/example_projects/Weather/Weather/WeatherApp.swift
--- a/example_projects/Weather/Weather/WeatherApp.swift
+++ b/example_projects/Weather/Weather/WeatherApp.swift
@@ -10,7 +10,7 @@
@main
struct WeatherApp: App {
- private let weatherService: WeatherService = .mock
+ private let weatherService: WeatherService = CommandLine.arguments.contains("--mock-weather-api") ? .mock : .production
var body: some Scene {
WindowGroup {You can send follow-ups to the cloud agent here.
Remove the leftover mock weather launch argument from the Weather example README and UI tests. The app now uses deterministic bundled data directly, so tests and docs should not reintroduce the launch flag path. Co-Authored-By: Codex <codex@openai.com>
Surface batch as the preferred next step when runtime snapshots contain multiple visible switch controls. This keeps switches out of generic tap guidance while making same-screen settings changes discoverable at the point agents plan their actions. Also clarify MCP server and tool descriptions so agents know to prefer batch for multiple same-screen elementRef taps without intermediate assertions. Co-Authored-By: Codex <codex@openai.com>
Restore compact runtime snapshot text summaries and add guarded root viewport scroll inference so agents receive valid app-agnostic scroll refs when semantic content overflows but no better descendant scroll target is exposed. The external Claude Weather benchmark completed successfully with 17 total tool calls after this change, preserving the optimized loop without Weather-specific logic or coordinate fallback behavior. Co-Authored-By: Codex CLI <noreply@openai.com>
Expose actionable sheet scroll targets and make swipe distance control the computed stroke length. This keeps Claude from falling back to generic app root scrolling when a sheet/list is the real scroll target. Ignore local validation artifacts so benchmark transcripts stay out of the working tree while remaining preserved on disk. Co-Authored-By: Codex CLI <noreply@openai.com>
Add an app-agnostic drag tool so agents can expand foreground sheets and scroll exposed sheet/list content without relying on synthetic swipe refs. Preserve useful semantic evidence outside callable target rows, and tighten next-step guidance so saved-but-not-selected rows do not invite no-op taps. Clarify batch and type_text tool descriptions to steer agents toward valid strict schemas and avoid wasted retry calls. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Safe batch stale switch cache
- Removed conditional check so runtime snapshot is always cleared after successful batch execution, preventing stale cache while preserving the optimization to skip post-action snapshot capture.
Or push these changes by commenting:
@cursor push 7d6a39031c
Preview (7d6a39031c)
diff --git a/src/mcp/tools/ui-automation/batch.ts b/src/mcp/tools/ui-automation/batch.ts
--- a/src/mcp/tools/ui-automation/batch.ts
+++ b/src/mcp/tools/ui-automation/batch.ts
@@ -204,9 +204,7 @@
try {
await executeAxeCommand(commandArgs, simulatorId, 'batch', executor, axeHelpers);
- if (!resolvedSteps.preserveSnapshot) {
- clearRuntimeSnapshot(simulatorId);
- }
+ clearRuntimeSnapshot(simulatorId);
log('info', `${LOG_PREFIX}/${toolName}: Success for ${simulatorId}`);
} catch (error) {
if (shouldInvalidateRuntimeSnapshotAfterActionError(error)) {You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 9a52e69. Configure here.
Keep the optimized same-screen batch path for inactive switches, but update the cached runtime snapshot after successful execution so those refs no longer look inactive. This prevents a follow-up batch from reusing stale switch state and undoing the settings change. Co-Authored-By: Codex <noreply@openai.com>
Keep within-element drag points for scroll containers, but use directional drag points for cell refs. Cells can expose swipeWithin for swipe gestures, but drag should start at the cell activation point and project in the requested direction. Co-Authored-By: Codex <noreply@openai.com>
Treat text view accessibility roles as text-entry controls so multiline editors receive tap and typeText actions instead of being normalized as static text. Co-Authored-By: Codex <noreply@openai.com>
Reject directional drag point calculations when viewport clamping would move the end point opposite the requested direction. This avoids reporting a valid drag for edge-adjacent elements when the generated stroke would go the wrong way. Co-Authored-By: Codex <noreply@openai.com>
Drop empty comma-separated entries before coercing numeric CLI array flags. This keeps trailing commas from becoming NaN values while still letting invalid non-empty numeric input fail validation normally. Co-Authored-By: Codex <noreply@openai.com>
Refresh the runtime snapshot after successful gesture actions so agents receive current refs and state instead of continuing from stale UI data. Co-Authored-By: Codex <noreply@openai.com>
Replace a silent early return with an explicit skipped snapshot case for CLI JSON swipe success coverage. The existing fixture still reflects the former coordinate-based swipe contract, so keep the gap visible until the runtime is intentionally re-baselined. Co-Authored-By: Codex <noreply@openai.com>
Collaborator
Author
|
Addressed the non-inline review findings:
I left the tap coordinate/id/label contract unchanged after re-auditing it against the PR design: the public tap tool is intentionally element-ref based here, with coordinate/selector details preserved internally after resolution rather than reintroduced as user inputs. |
…taps When a completion button such as Add or Save produced no screen change, snapshot rendering still suppressed the related incomplete status row. Pass the action context into the suppression decision so repeated no-op completion buttons do not hide the state that explains what remains unresolved. Co-Authored-By: Codex <noreply@openai.com>
Replace the nested scroll next-step ternary with an explicit if/else chain. This keeps behavior unchanged while making the sheet-drag versus swipe selection easier to review and maintain. Co-Authored-By: Codex <noreply@openai.com>
Apply Prettier wrapping to the no-op completion regression test so the branch is clean after the targeted fix. Co-Authored-By: Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add rs/1 runtime UI automation parity for simulator workflows.
This PR changes UI automation responses from "the action finished" into reusable next-action context. UI action tools can now return compact runtime captures with stable element refs, screen hashes, action targets, scroll targets, and app-agnostic next-step guidance. The goal is to let agents continue from the previous tool result instead of repeatedly calling
snapshot_uior falling back to screenshots for routine verification.Why this matters: the previous flow made agents rediscover the same UI after nearly every action. That inflated token use, encouraged extra screenshots/snapshots, and made scroll/tap decisions depend too much on agent guesswork. Runtime captures keep the structured-output envelope intact while giving the model enough foreground UI context to choose the next control directly.
The main implementation changes are:
scrollrows rather than invented refs.key_pressandkey_sequencethrough the daemon path so keyboard interactions share the same simulator/session state behavior as the other UI automation tools.data.capture.This is intentionally app-agnostic. The business rules do not special-case the Weather sample app or any specific UI. They reason from runtime UI data: visible/actionable controls, AX roles, identifiers when available, tree depth, containment, foreground surfaces, and nearby control relationships.
The companion contributor docs PR is: getsentry/xcodebuildmcp.com#16
Manual Claude comparison against the deterministic Weather UI task showed the intended improvement:
Local validation covered formatting, linting, type checking, build, focused UI automation tests, schema fixture validation, focused CLI JSON snapshots, and the full Vitest suite.