Internal - Refactor Agent Integration Tests to Remove Backend Dependency

[cleverchuk]

Summary

Replaces the existing smoke/e2e integration test suite with a new feature-tests module (testing/feature-tests) that uses the OTel AgentInstrumentationExtension to test agent behavior in-process. The old tests required a live SolarWinds backend, a k6 load driver, and sometimes a cloud environment just to assert correctness of agent internals. The new tests are self-contained: they run with the real agent loaded but intercept spans and metrics locally, removing all external dependencies from the core feature verification loop. Smoke tests whose coverage is now redundant are removed, and the Lambda CI workflow is also removed for the same reason. Some tests with external dependencies remain unchanged.

---

What Changed and Why

Replacing backend-dependent tests with in-process feature tests

The previous approach validated agent behavior by sending traffic through a full stack (k6 → app → SolarWinds backend) and then asserting on JSON result files. This made tests slow, and fragile. The new feature-tests module uses AgentInstrumentationExtension — the standard OTel test harness — which intercepts spans, metrics, and context directly in the test JVM. This lets each test assert on the exact OTel data structures the agent produces, without any network calls.

Seven feature areas are covered:

Test class	What it asserts
ContextPropagationTest	W3C traceparent and SolarWinds sw= tracestate header injection
SdkTracingTest	Manual span creation and attribute propagation via the OTel SDK tracer
TriggerTraceTest	X-Trace-Options trigger-trace header stamps the expected attributes on the root span
TransactionNamingTest	Custom transaction name attribute is set on the root span
TransactionFilteringTest	Spans matching a filtered transaction are dropped / not exported
MetricsGenerationTest	Request, response-time, and error metrics are emitted per span
ProfilingSpanProcessorTest	Profiling span processor emits the expected span events

Sampling setup: real sampler path instead of a bypass

The test extension previously installed a TestSampler that short-circuited sampling entirely, bypassing settings. The replacement uses TestUtils.initSettingsReader() to inject a TestSettingsReader with always-sample settings (rate = 1_000_000, flags (true, false, true, true, false)) directly into the production sampler. A new TestAgentListener (ordered last via order = Integer.MAX_VALUE) waits for AgentState.READY before injecting these settings, so the real startup sequence runs to completion before the test configuration is applied.

Co-located Spring MVC instrumentation tests

New HandlerNameInstrumentationTest classes are added inside the spring-webmvc-3.1 and spring-webmvc-6 javaagent submodules, keeping instrumentation-level tests alongside the instrumentation code they cover.

Cleanup

SmokeTest cases whose coverage moved to feature-tests (transaction filtering, JDBC, trigger trace, context propagation, transaction naming, metrics, profiling, SDK tracing) are deleted outright. What remains in SmokeTest are tests that are genuinely environment-specific and can't run in-process: proxy usage, log export to the backend, OTel metric export, and cloud metadata checks. SmokeTestV2, LambdaTest, and the corresponding k6 scenarios are deleted in full — they either duplicated the new tests or validated backend connectivity rather than agent correctness. The lambda-test.yml CI workflow and the top-level build reference to the removed module are also deleted.

Test services data

1. e-1712644058766987264
2. e-1712643928659124224
3. e-1742334541200846848
4. e-1777406072376840192

[Copilot]

Pull request overview

Refactors agent integration tests to remove the live SolarWinds backend dependency. A new in-process testing/feature-tests module uses OTel's AgentInstrumentationExtension plus a TestSettingsReader (injected via a new TestAgentListener) to assert agent behavior locally. Old backend-driven smoke tests, the Lambda smoke test, and the corresponding k6 scenarios / CI workflow are deleted; Spring MVC instrumentation gets co-located handler-name tests.

Changes:

• Adds testing/feature-tests with 7 in-process feature test classes; replaces TestSampler short-circuit with real-sampler-plus-TestSettingsReader setup via a new TestAgentListener.
• Adds HandlerNameInstrumentationTest in both spring-webmvc-3.1 and spring-webmvc-6 javaagent submodules and enables profiler/strict-context flags in the instrumentation test convention.
• Removes redundant smoke tests, the entire LambdaTest/SmokeTestV2 flows, related k6 functions, the lambda-test.yml workflow, and the cleanListedDependencies test finalizer.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
testing/feature-tests/build.gradle.kts	New module Gradle script; depends on shadowJar and Java 17
testing/feature-tests/src/test/.../ContextPropagationTest.java	Asserts W3C+sw tracestate injection
testing/feature-tests/src/test/.../SdkTracingTest.java	Manual SDK span / stacktrace / service.name asserts (two tests near-duplicate)
testing/feature-tests/src/test/.../TriggerTraceTest.java	Asserts X-Trace-Options attributes on root span
testing/feature-tests/src/test/.../TransactionNamingTest.java	Asserts SDK transaction-name attribute
testing/feature-tests/src/test/.../TransactionFilteringTest.java	URL-based filter via SettingsManager.initialize; tearDown does not reset config
testing/feature-tests/src/test/.../MetricsGenerationTest.java	Seven near-identical per-metric tests
testing/feature-tests/src/test/.../ProfilingSpanProcessorTest.java	Uses Thread.sleep; strict equality assertion on profile span count
testing/agent-test-extension/.../TestAutoConfigurationCustomizer.java	customize() no longer uses its parameter; duplicates listener setup
testing/agent-test-extension/.../TestAgentListener.java	Waits for AgentState.READY then installs test settings
testing/agent-test-extension/.../TestSampler.java	Deleted
testing/agent-test-extension/build.gradle.kts	Adds compileOnly deps for new listener
instrumentation/spring-webmvc/spring-webmvc-3.1/javaagent/...	New handler-name test + tests built against Spring 5.3.30 (not 3.1)
instrumentation/spring-webmvc/spring-webmvc-6/javaagent/...	New handler-name test (mirrors v3.1 version)
buildSrc/.../solarwinds.instrumentation-conventions.gradle.kts	Adds strict-context, profiler, and stacktrace JVM args globally
build.gradle.kts	Removes Test→cleanListedDependencies finalizer
settings.gradle.kts	Registers testing:feature-tests
smoke-tests/src/test/java/com/solarwinds/SmokeTest.java	Removes tests that moved in-process
smoke-tests/src/test/java/com/solarwinds/SmokeTestV2.java	Removes redundant tests
smoke-tests/src/test/java/com/solarwinds/LambdaTest.java	Deleted
smoke-tests/k6/basic.js	Removes large k6 verification helpers; tightens load options
.github/workflows/lambda-test.yml	Workflow deleted

Comments suppressed due to low confidence (3)

testing/feature-tests/src/test/java/com/solarwinds/opentelemetry/instrumentation/feature/test/ProfilingSpanProcessorTest.java:63

• The .as(...) description "sw.profile.spans attribute should be set 1" reads awkwardly. Consider rewording to e.g. "sw.profile.spans attribute should equal 1" so the failure message is clearer.

    assertThat(profileSpans).as("sw.profile.spans attribute should be set 1").isEqualTo(1);

testing/feature-tests/src/test/java/com/solarwinds/opentelemetry/instrumentation/feature/test/MetricsGenerationTest.java:93

• verifyTokenbucketExhaustionCountMetricIsReported asserts that the trace.service.tokenbucket_exhaustion_count metric is reported after a single successful span. Token-bucket exhaustion typically only happens when the sampler is rate-limited; with the always-sample settings used in this suite (rate = 1,000,000) this metric may or may not be emitted at all, making the test fragile. Please verify this metric is unconditionally emitted on every span flush; otherwise the test will become intermittently failing.

  @Test
  void verifyTokenbucketExhaustionCountMetricIsReported() {
    Tracer tracer = GlobalOpenTelemetry.get().getTracer("test");
    Span span =
        tracer.spanBuilder("GET /petclinic/api/pettypes").setSpanKind(SpanKind.SERVER).startSpan();
    span.end();

    testing.waitForTraces(1);
    await()
        .atMost(5, TimeUnit.SECONDS)
        .untilAsserted(
            () -> {
              List<MetricData> metrics = testing.metrics();
              assertThat(metrics)
                  .as("trace.service.tokenbucket_exhaustion_count metric should be reported")
                  .anyMatch(m -> m.getName().equals("trace.service.tokenbucket_exhaustion_count"));
            });
  }

testing/agent-test-extension/src/main/java/com/solarwinds/opentelemetry/extensions/TestSampler.java:1

• TestSampler was the only consumer of the SolarwindsSampler subclass extension point used by the test extension. With it deleted, sampling now goes through the real SolarwindsSampler. Please confirm there are no other tests or wiring referencing TestSampler (e.g. in resources/META-INF service files) that should also be cleaned up.