Add MAUI Android inner loop deploy measurement scenario by davidnguyen-tech · Pull Request #5165 · dotnet/performance

davidnguyen-tech · 2026-03-19T11:09:00Z

Summary

Adds a new scenario that measures MAUI Android developer inner loop performance:

first build+deploy+startup time measurement
10 incremental build+deploy+startup time measurements
parse build&deploy times from .binglogs.
parse startups times from the Android activity manager output

The scenario is the first one in the repo to build the app on Helix.
We send the app source code as the Helix payload because of the nature of this scenario.

The incremental change is simulated by:

one-line change in a .cs file
one-line change in a .xml file

Sequential incremental inner loop measurements are implemented by switching between the updated and the original versions of the files.

Test targets

Device	Queue
Pixel 8	`Windows.11.Amd64.Pixel.Perf`
Galaxy A16	`Windows.11.Amd64.Galaxy.Lowend.Perf`
Android 36 Emulator	`Ubuntu.2204.Amd64.Android.36`

AzDO Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2964307&view=results

davidnguyen-tech · 2026-03-20T22:55:37Z

/azp run

azure-pipelines · 2026-03-20T22:55:51Z

Azure Pipelines successfully started running 1 pipeline(s).

DrewScoggins · 2026-03-27T00:06:03Z

When this PR goes in, #5176, you will need to take the changes. I am changing how the auth works and it needs the update to upload.py to continue working.

Add a new C# parser that extracts build and deploy metrics from MSBuild binary logs (.binlog) for MAUI Android inner loop measurements. The parser captures: - Overall build duration (Publish Time) - Build task timings: Csc, XamlC, GenerateJavaStubs, D8, Javac, etc. - Build target timings: CoreCompile, _GenerateJavaStubs, _CompileToDalvik, etc. - Deploy task timings: FastDeploy, AndroidSignPackage, Aapt2Link - Deploy target timings: _Sign, _Upload, _DeployApk, _BuildApkFastDev Register the AndroidInnerLoop MetricType in Startup.cs so the parser can be selected via the measurement framework. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extend the shared test runner with a new ANDROIDINNERLOOP scenario type that orchestrates first deploy and incremental build+deploy+startup measurements for MAUI Android apps. Changes: - const.py: Add ANDROIDINNERLOOP constant and scenario name mapping - runner.py: Add argument parser and full ANDROIDINNERLOOP handler that performs first build+deploy, then N incremental iterations with source file toggling, binlog capture, startup time measurement via am start, and result upload to perflab - androidhelper.py: Add skip_install, screen_timeout_ms, skip_uninstall, and other parameters to AndroidHelper for inner loop reuse - startup.py: Fix copytree FileExistsError with dirs_exist_ok=True Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add the scenario-specific scripts for MAUI Android inner loop measurements: - pre.py: Bootstraps the .NET SDK, installs maui-android workload, restores NuGet packages, installs Android SDK dependencies (build tools, platform SDK, Java), and creates the MAUI test app with modified source files for incremental measurement - setup_helix.py: Helix-specific environment setup that discovers dotnet/SDK/Android/Java paths, installs workloads, and prepares the build environment on Helix agents - test.py: Entry point that invokes the shared test runner with ANDROIDINNERLOOP test type - post.py: Cleanup script that disables device animations, restores screen settings, and uninstalls the test APK using ADB Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add CI infrastructure to run inner loop measurements on Helix: - maui_scenarios_android_innerloop.proj: MSBuild project that defines Helix work items with per-platform PreCommands for environment setup, SDK discovery, workload installation, and test invocation - sdk-perf-jobs.yml: Add 6 inner loop job definitions covering Pixel 8, Galaxy A16, and Android 36 emulator queues, each with Mono and CoreCLR runtime configurations - build-machine-matrix.yml: Add ubuntu-x64-android-emulator build machine mapping to Ubuntu.2204.Amd64.Android.36 queue - run-performance-job.yml: Support androidinnerloop runtime flavor - run_performance_job.py: Extend maui_scenarios_android run_kind matching to include innerloop variant; copy binlogs to artifacts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add --screen-timeout-ms CLI argument (default 1800000 = 30 min) and ScreenTimeoutMs MSBuild property so the screen timeout can be tuned from the .proj file without code changes. Add LaunchState validation to AndroidHelper.measure_cold_startup(): if am start reports anything other than COLD (e.g. UNKNOWN when the screen is off), the method now throws with a clear diagnostic message suggesting to increase --screen-timeout-ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds a new Helix-driven scenario to measure MAUI Android “inner loop” performance by timing first build+deploy+startup and repeated incremental build+deploy+startup iterations, with binlog parsing for build/deploy breakdown and am start/logcat parsing for startup time.

Changes:

Introduces a new AndroidInnerLoop parser in the ScenarioMeasurement startup tool to extract target/task durations from .binlogs.
Adds a new androidinnerloop scenario flow in the Python runner, plus a dedicated mauiandroidinnerloop scenario directory (pre/setup/post scripts) and a new Helix project file.
Updates pipelines/job config to run the scenario on Pixel/Galaxy (Windows) and an Android emulator (Ubuntu), and fixes repeated trace directory uploads by allowing copytree into existing dirs.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/tools/ScenarioMeasurement/Util/Parsers/AndroidInnerLoopParser.cs	New binlog parser emitting build/deploy counters for the new scenario.
src/tools/ScenarioMeasurement/Startup/Startup.cs	Adds `AndroidInnerLoop` metric type wiring to the startup tool.
src/scenarios/shared/startup.py	Allows repeated trace uploads by using `copytree(..., dirs_exist_ok=True)`.
src/scenarios/shared/runner.py	Adds `androidinnerloop` subcommand and orchestrates build+deploy+startup iterations + aggregation/upload.
src/scenarios/shared/const.py	Adds `ANDROIDINNERLOOP` constant and scenario-name mapping.
src/scenarios/shared/androidhelper.py	Extends device setup options and adds cold-start measurement helper.
src/scenarios/mauiandroidinnerloop/test.py	New scenario entrypoint using shared runner.
src/scenarios/mauiandroidinnerloop/setup_helix.py	New Helix setup script (workloads, Android deps, adb readiness, restore).
src/scenarios/mauiandroidinnerloop/pre.py	Creates MAUI template payload + prepares file-edit toggles and NuGet config.
src/scenarios/mauiandroidinnerloop/post.py	Cleanup/uninstall/build-server shutdown for the new scenario.
scripts/run_performance_job.py	Adds new `run_kind` handling and binlog artifact copying for this scenario.
eng/pipelines/templates/run-performance-job.yml	Enables passing `--runtime-flavor` for the new run kind.
eng/pipelines/templates/build-machine-matrix.yml	Adds Ubuntu Android emulator queue to the build matrix (private builds).
eng/pipelines/sdk-perf-jobs.yml	Schedules the new scenario across Pixel/Galaxy/emulator for mono+coreclr Debug.
eng/performance/maui_scenarios_android_innerloop.proj	New Helix project defining payload prep + work items for device/emulator tracks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

kotlarmilos

Overall looks good. Reusing the existing constructs should reduce the amount of code and make it easier for review and maintenance

The default Google manifest in the .NET 11 preview Android workload doesn't yet include API 37, so InstallAndroidDependencies silently fails to install platforms;android-37.0 (NRE in ResolveAndroidTooling) while exiting 0, and the subsequent build dies with XA5207. GoogleV2 has the API 37 entry, so passing -p:AndroidManifestType=GoogleV2 unblocks the emulator scenario on Helix. TODO marker references the upstream tracking issue so this can be removed once the default manifest is updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

InstallAndroidDependencies sometimes exits 0 while skipping required SDK packages (dotnet/android#11319). Different manifests skip different packages, e.g. the default manifest dropped platforms;android-37.0 and GoogleV2 dropped platform-tools. Parse the warning text after the target succeeds, install each missing package via sdkmanager, and additionally re-install platform-tools unconditionally if adb is missing post-install. Verify adb is present and runnable before continuing; abort the script if it isn't. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Merge maui_scenarios_android_innerloop.proj into maui_scenarios_android.proj and select the inner-loop branch via a new $(_IsInnerLoop) property derived from the RunKind env var (set by run_performance_job.py). Pipeline entries that previously pointed at the inner-loop proj now reference the consolidated file; runKind is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The previous os.environ["RunKind"] in run_performance_job.py only affects that process, so the separate 'Send job to Helix' AzDO task did not see RunKind and the consolidated maui_scenarios_android.proj evaluated $(_IsInnerLoop) as empty, running the regular work items for inner-loop phases. Pipe runKind through the send-to-helix template as an env var so MSBuild promotes it to a property. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 17/17 changed files
Comments generated: 1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

kotlarmilos · 2026-05-20T13:17:40Z

The PR is ready for review, validated end-to-end on the https://dev.azure.com/dnceng/internal/_build/results?buildId=2979643.

Copilot

Copilot's findings

Files reviewed: 17/17 changed files
Comments generated: 2

- Restore edited source files in finally block so reused workspaces start clean - Bound adb shell poll in measure_startup_from_logcat with a 10s timeout so a hung adb does not consume the full polling window - Replace 'X | None' with Optional[X] in mauisharedpython for Python 3.9 compat Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 17/17 changed files
Comments generated: 2

…ards Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

matouskozak · 2026-05-20T14:52:20Z

+    <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'mono'">/p:UseMonoRuntime=true</_MSBuildArgs>
+    <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'coreclr'">/p:UseMonoRuntime=false</_MSBuildArgs>
+    <!-- Disable ReadyToRun so inner loop measures JIT behavior. -->
+    <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'coreclr'">$(_MSBuildArgs);/p:PublishReadyToRun=false;/p:PublishReadyToRunComposite=false</_MSBuildArgs>


Since we're settings these here instead of using https://github.com/dotnet/performance/pull/5165/changes#diff-652d0d84aa2897441204ca7f3c3e5f8b3dd75fb44edaa69c2095a610270297d3L18-R28, maybe we should create a separate .proj file for the inner loop? I don't see many things which are re-used among the two scenarios and it would remove the <PropertyGroup Condition="'$(_IsInnerLoop)' == 'true'"> checks

matouskozak

Overall looking good to me

davidnguyen-tech changed the title ~~Add MAUI Android inner loop deploy measurement scenario~~ [WIP] Add MAUI Android inner loop deploy measurement scenario Mar 20, 2026

davidnguyen-tech changed the title ~~[WIP] Add MAUI Android inner loop deploy measurement scenario~~ [WIP] Add MAUI Android Inner Loop scenario Mar 30, 2026

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch 2 times, most recently from 0418d92 to 54f7a75 Compare March 31, 2026 15:18

davidnguyen-tech changed the title ~~[WIP] Add MAUI Android Inner Loop scenario~~ Add MAUI Android inner loop deploy measurement scenario Mar 31, 2026

davidnguyen-tech and others added 4 commits April 1, 2026 11:36

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch from 73918d8 to ff881bb Compare April 1, 2026 09:36

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch from ff881bb to 746709a Compare April 1, 2026 09:37

davidnguyen-tech marked this pull request as ready for review April 1, 2026 21:24

davidnguyen-tech requested review from DrewScoggins, LoopedBard3, kotlarmilos, matouskozak, simonrozsival and vitek-karas April 1, 2026 21:25

LoopedBard3 requested a review from Copilot April 1, 2026 21:34

Copilot started reviewing on behalf of LoopedBard3 April 1, 2026 21:35 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Address review notes

986c7bf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

matouskozak reviewed Apr 2, 2026

View reviewed changes

Comment thread src/scenarios/mauiandroidinnerloop/pre.py Outdated

matouskozak reviewed Apr 2, 2026

View reviewed changes

Comment thread eng/pipelines/sdk-perf-jobs.yml

matouskozak reviewed Apr 2, 2026

View reviewed changes

Comment thread eng/performance/maui_scenarios_android_innerloop.proj Outdated

Address review notes 2

26ea3ea

kotlarmilos reviewed May 6, 2026

View reviewed changes

davidnguyen-tech mentioned this pull request May 11, 2026

[InstallAndroidDependencies] silently fails to install platforms;android-37.0 on fresh SDK dir (XARAT7001 NRE), exits 0 dotnet/android#11319

Closed

Copilot AI mentioned this pull request May 11, 2026

Fail InstallAndroidDependencies cleanly when ResolveAndroidTooling has no resolved SDK dotnet/android#11321

Merged

davidnguyen-tech and others added 2 commits May 11, 2026 19:28

Copilot AI review requested due to automatic review settings May 11, 2026 18:36

Copilot started reviewing on behalf of davidnguyen-tech May 11, 2026 18:37 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

Comment thread src/scenarios/shared/runner.py

Comment thread src/scenarios/shared/androidhelper.py

kotlarmilos and others added 2 commits May 15, 2026 15:13

Copilot AI review requested due to automatic review settings May 19, 2026 10:58

Copilot started reviewing on behalf of kotlarmilos May 19, 2026 10:59 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

Comment thread src/scenarios/shared/runner.py

Guard Helix traces cleanup on non-null upload dir

053f80c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

kotlarmilos requested review from LoopedBard3, jonathanpeppers, kotlarmilos and matouskozak May 20, 2026 13:16

Copilot AI review requested due to automatic review settings May 20, 2026 13:32

Copilot started reviewing on behalf of kotlarmilos May 20, 2026 13:33 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Comment thread eng/performance/maui_scenarios_android.proj

Comment thread src/tools/ScenarioMeasurement/Util/Parsers/AndroidInnerLoopParser.cs

kotlarmilos force-pushed the feature/measure-maui-android-deploy branch from 740a3c0 to 053f80c Compare May 20, 2026 13:42

Copilot AI review requested due to automatic review settings May 20, 2026 13:51

Copilot started reviewing on behalf of kotlarmilos May 20, 2026 13:52 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Comment thread src/scenarios/shared/androidhelper.py Outdated

Comment thread src/tools/ScenarioMeasurement/Util/Parsers/AndroidInnerLoopParser.cs

androidhelper: use grep -F so dots in package name aren't regex wildc…

7ec125c

…ards Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

matouskozak reviewed May 20, 2026

View reviewed changes

matouskozak approved these changes May 20, 2026

View reviewed changes

Conversation

davidnguyen-tech commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test targets

AzDO Build

Uh oh!

davidnguyen-tech commented Mar 20, 2026

Uh oh!

azure-pipelines Bot commented Mar 20, 2026

Uh oh!

DrewScoggins commented Mar 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kotlarmilos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

kotlarmilos commented May 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

matouskozak May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matouskozak left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

davidnguyen-tech commented Mar 19, 2026 •

edited

Loading

matouskozak May 20, 2026 •

edited

Loading