Fix Java in-process test lifecycle and parity

Prevent the Java in-process test profile from corrupting Surefire control
streams or poisoning later tests through the runtime's process-global LLM
provider registration. Preserve explicit subprocess and TCP transport choices,
sanitize process-only options before constructing in-process clients, and run
request-handler tests over their required isolated stdio runtime.

Fix the remaining test-contract issues by making fake socket RPC handler
registration atomic with reader startup, honoring the configured CLI
entrypoint when runtime.node is in a prebuilds directory, and isolating the
streaming model-cache scenario. Remove in-process skip annotations from tests
that already exercise an explicit subprocess transport.

The complete `mvn clean verify -Pinprocess` run now finishes successfully
without hangs, transport timeouts, provider-ownership failures, or Surefire
stream corruption.

File-by-file manifest:

- `java/sdk/pom.xml`: use Surefire's TCP fork channel for unit and integration
  tests so native runtime output cannot corrupt Maven's process-pipe protocol.
- `java/sdk/src/main/java/com/github/copilot/CopilotClient.java`: preserve
  explicitly selected TCP options when the default connection environment
  requests in-process transport.
- `java/sdk/src/main/java/com/github/copilot/JsonRpcClient.java`: add a socket
  construction hook that registers handlers before the reader thread starts.
- `java/sdk/src/main/java/com/github/copilot/ffi/NativeRuntimeLoader.java`:
  resolve the configured Copilot executable separately from runtime.node when
  the native library uses the package's prebuilds layout.
- `java/sdk/src/main/java/com/github/copilot/rpc/CopilotClientOptions.java`:
  allow `setCwd(null)` to clear a previously configured working directory.
- `java/sdk/src/test/java/com/github/copilot/ClientOptionsE2ETest.java`: run
  explicit fake-stdio option forwarding tests under the in-process profile.
- `java/sdk/src/test/java/com/github/copilot/ConfigCloneTest.java`: cover
  clearing a configured working directory.
- `java/sdk/src/test/java/com/github/copilot/CopilotClientTest.java`: remove
  obsolete in-process skips from explicit subprocess and TCP lifecycle tests.
- `java/sdk/src/test/java/com/github/copilot/CopilotClientTransportTest.java`:
  test explicit transport precedence, in-process option sanitization, and TCP
  token selection under the profile default.
- `java/sdk/src/test/java/com/github/copilot/CopilotRequestTestSupport.java`:
  explicitly select stdio for request-handler tests that register the
  process-global LLM inference provider.
- `java/sdk/src/test/java/com/github/copilot/E2ETestContext.java`: honor
  explicit transports, route request-handler clients to subprocess isolation,
  and clear environment, cwd, and CLI arguments before in-process client
  construction.
- `java/sdk/src/test/java/com/github/copilot/GitHubTelemetryTest.java`: register
  fake runtime RPC handlers before socket message processing begins.
- `java/sdk/src/test/java/com/github/copilot/MetadataApiTest.java`: run explicit
  stdio metadata tests instead of skipping them under the profile.
- `java/sdk/src/test/java/com/github/copilot/PerSessionAuthTest.java`: run the
  explicit subprocess unauthenticated case under the profile.
- `java/sdk/src/test/java/com/github/copilot/RpcServerMiscE2ETest.java`: run the
  explicit subprocess account lifecycle case under the profile.
- `java/sdk/src/test/java/com/github/copilot/StreamingFidelityTest.java`: give
  the gpt-5.4 reasoning/streaming scenario an isolated proxy and runtime model
  cache.
- `java/sdk/src/test/java/com/github/copilot/ffi/FfiRuntimeHostTest.java`: cover
  failed connection-open cleanup followed by successful sequential startup.
- `java/sdk/src/test/java/com/github/copilot/ffi/NativeRuntimeLoaderTest.java`:
  cover resolving a configured CLI beside a prebuilds runtime.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e03c4e94-97b0-41ad-9f4e-c01633dc0bf7

Author: edburns

1917-java-embed-rust-cli-runtime-remove-before-merge/post-agentic-01-test-parity-fix-remaining-tests-01.md

@@ -0,0 +1,172 @@
+# Prompt: make the Java InProcess test run clean
+
+You are working in `/home/edburns/workareas/copilot-sdk-01`, branch
+`edburns/review-copilot-pr-2272`. Read these files first:
+
+- `1917-java-embed-rust-cli-runtime-remove-before-merge/post-agentic-01-test-parity-fix-remaining-tests.md`
+- `java/20260807-0145-job-logs.txt`
+- the current git diff and the Java test/runtime/harness sources
+
+The target command is:
+
+```bash
+cd java
+mvn clean verify -Pinprocess
+```
+
+Make the implementation and test changes necessary for a genuinely clean,
+non-hanging run. Do not solve this by broadly skipping tests, increasing
+timeouts, weakening assertions, or hiding errors. Preserve the negative-test
+assertions; expected negative cases may be logged, but they must not be
+reported as test errors.
+
+## What the interrupted log establishes
+
+The run was interrupted after more than an hour; it has no `BUILD SUCCESS`.
+There are 88 errors in 20 suites. The failures are highly clustered:
+
+- `std/in stream corrupted` appears during `AskUserTest`.
+- `ByokBearerTokenProviderE2ETest` has the expected fake 404 in one negative
+  case, but the other two tests fail because
+  `llmInference.setProvider` says “Another client is already the LLM inference
+  provider.”
+- The same provider-ownership error breaks
+  `CopilotRequestCancelErrorE2ETest`, `CopilotRequestHandlerE2ETest`,
+  `SessionConfigE2ETest`, and other provider/handler tests.
+- `CompactionTest`, `CopilotSessionTest`, `ErrorHandlingTest`,
+  `EventFidelityTest`, `ExecutorWiringTest`, `HooksTest`, `McpAndAgentsTest`,
+  `ModeHandlersTest`, `MultiProviderRegistryE2ETest`, `PermissionsTest`,
+  `PreMcpToolCallHookTest`, `RpcSessionStateExtrasE2ETest`,
+  `SessionConfigE2ETest`, and `SessionEventsE2ETest` contain repeated
+  approximately 60-second `sendAndWait`/future timeouts.
+- `GitHubTelemetryTest` fails immediately because an InProcess connection
+  receives `Method not found: connect` and `Method not found: ping`; determine
+  whether this test must explicitly use the subprocess/socket transport or
+  whether the InProcess endpoint is missing required handlers.
+- `RpcServerE2ETest` has a 30-second RPC timeout and
+  `RpcSessionStateExtrasE2ETest` has a 60-second timeout.
+- `PerSessionAuthTest` has one skipped test and a negative 401 “Bad
+  credentials” trace. The test itself is not an error.
+- `ClientOptionsE2ETest` skips all three tests. Other suites also report
+  intentional-looking skips: `CopilotClientTest` (14),
+  `CopilotClientTransportTest` (4), `MetadataApiTest` (3),
+  `RpcServerMiscE2ETest` (1), and `CompactionTest` (1).
+- Many stack traces in `CreateSessionReKeyEntryTest`, `JsonRpcClientTest`,
+  `LifecycleEventManagerTest`, `RpcHandlerDispatcherTest`, and
+  `SessionHandlerTest` are deliberately generated negative-test traces and
+  are followed by passing summaries. Do not misclassify them as failures.
+
+## Priority 1: stop stream corruption and fix InProcess ownership/lifecycle
+
+Investigate `std/in stream corrupted` first. Trace every process and stream
+created by the InProcess FFI path, `host_start`, the bundled `copilot`
+entrypoint, `NativeRuntimeLoader`, `InProcessRuntimeConnection`, `CapiProxy`,
+and Surefire. Identify which native/child process is writing bytes to the
+Surefire-controlled stdout/stdin protocol. Ensure child stdout/stderr are
+consumed or redirected in the same way as the supported transport and that
+the FFI receive/send streams are not closed or reused by another client.
+Do not merely suppress Surefire output.
+
+Then fix the “Another client is already the LLM inference provider” root
+cause. Determine whether clients, native hosts, provider registrations, or
+`InProcessEnvGuard` instances survive test teardown. Verify the close path on
+both successful and failed `start()`, failed `createSession()`, and failed
+requests. Ensure a failed startup cannot leave a provider registered and that
+each test context closes its client/proxy/runtime deterministically. If the
+InProcess runtime is process-global, serialize or otherwise coordinate provider
+ownership rather than allowing overlapping providers. Add focused regression
+coverage for failed-start cleanup and sequential client startup.
+
+The earlier context notes that `E2ETestContext.applyContextOptions()` must
+clear InProcess-incompatible `cwd` and `cliArgs` in addition to `environment`.
+Implement that carefully, and verify the actual setter semantics:
+`setEnvironment(null)` clears to an empty map, while `setCwd(null)` and
+`setCliArgs(null)` must be checked rather than assumed. Add or update tests so
+the options are truly absent according to constructor validation.
+
+## Priority 2: isolate and repair the common timeout
+
+After Priority 1, run small, serial selectors, not the full suite:
+
+```bash
+cd java
+COPILOT_SDK_DEFAULT_CONNECTION=inprocess mvn test -pl sdk \
+  -Dtest="AskUserTest,ByokBearerTokenProviderE2ETest,CopilotSessionTest" \
+  -DfailIfNoTests=false
+```
+
+Use a bounded shell timeout while debugging so a regression cannot consume an
+hour. For any remaining timeout, capture a thread dump and inspect the
+corresponding Surefire report plus replay-proxy output. Follow one request
+from Java JSON-RPC send, through the FFI callback/`QueueInputStream`, into the
+replay proxy, and back to the Java reader. Confirm that:
+
+1. `host_start` returns a valid handle and the child `copilot` entrypoint is
+   reachable.
+2. The request reaches the proxy with the expected snapshot.
+3. Every response/event is framed correctly and enqueued to the receive
+   stream.
+4. stream completion/EOF and client close wake blocked readers.
+5. callbacks do not depend on a thread or executor that has already shut down.
+
+Use `StreamingFidelityTest.testShouldEmitStreamingDeltasWithReasoningEffortConfigured`
+as the minimal streaming reproducer, but also test one ordinary
+`CopilotSessionTest` request. Do not patch each timed-out suite individually;
+the repeated 60-second failures indicate a shared transport or lifecycle
+defect. Once the common path works, rerun representative handler, hook,
+permission, event, session-config, MCP, and RPC-server selectors and only
+then the complete profile.
+
+`GitHubTelemetryTest` is a separate transport-contract issue: inspect its
+test setup and the supported connection mode. If it intentionally uses a
+minimal fake RPC peer that only supports telemetry, make it explicitly select
+that transport so the global InProcess profile cannot route it to a runtime
+without `connect`/`ping`. If InProcess is intended, implement the missing
+protocol surface and add focused coverage.
+
+## Priority 3: remove unjustified skips
+
+Audit every skipped test in the log and the associated assumptions. For each:
+
+- make it run under InProcess when the behavior is transport-independent;
+- explicitly force subprocess/socket transport when the test is specifically
+  validating subprocess-only options or protocol behavior; or
+- change the test setup so the same public behavior is exercised through
+  InProcess.
+
+Do not add a profile-wide exclusion and do not convert skipped tests to
+passing assertions. In particular, investigate all three
+`ClientOptionsE2ETest` skips, the `PerSessionAuthTest` skip, and the skips in
+`CopilotClientTest`, `CopilotClientTransportTest`, `MetadataApiTest`,
+`RpcServerMiscE2ETest`, and `CompactionTest`. The final profile run should
+have zero skips unless a test is demonstrably impossible on the platform and
+the repository’s existing policy explicitly permits it; document any
+remaining exception in the test source.
+
+## Priority 4: make expected negative output intentional
+
+Do not alter assertions for negative tests. After all tests pass, reduce noisy
+expected stack-trace logging only where the repository’s logging conventions
+support it: distinguish expected test-triggered failures from unexpected
+transport failures, and avoid logging full stack traces at warning/error for
+the expected path if that can be done without hiding real failures. The
+`fake byok endpoint`, `401 Bad credentials`, `session.resume` not-found,
+handler exceptions, malformed JSON, socket-close, and re-key traces must
+remain asserted and diagnosable.
+
+## Validation and completion criteria
+
+Use the repository’s normal Java bootstrap and Maven logging conventions.
+Format Java changes with `mvn spotless:apply` from `java`. Run focused tests
+after each root-cause fix, then:
+
+```bash
+cd java
+mvn clean verify -Pinprocess
+```
+
+The task is complete only when this command terminates normally with
+`BUILD SUCCESS`, all test suites report zero failures and zero errors, no
+test hangs or 60-second transport timeouts occur, no Surefire stream
+corruption occurs, and the skip count is zero or each explicitly justified
+platform exception is documented and approved by the existing test policy.

java/sdk/pom.xml

@@ -243,6 +243,7 @@
                     </execution>
                 </executions>
                 <configuration>
+                    <forkNode implementation="org.apache.maven.plugin.surefire.extensions.SurefireForkNodeFactory" />
                     <systemPropertyVariables>
                         <project.build.directory>${project.build.directory}</project.build.directory>
                         <project.build.finalName>${project.build.finalName}</project.build.finalName>
@@ -264,6 +265,7 @@
                 <artifactId>maven-surefire-plugin</artifactId>
                 <configuration>
                     <runOrder>alphabetical</runOrder>
+                    <forkNode implementation="org.apache.maven.plugin.surefire.extensions.SurefireForkNodeFactory" />
                     <!-- Inject JaCoCo agent + any JDK-version-specific flags -->
                     <argLine>${testExecutionAgentArgs} ${surefire.jvm.args} --add-opens com.github.copilot.java/com.github.copilot.e2e=ALL-UNNAMED</argLine>
                     <!--

java/sdk/src/main/java/com/github/copilot/CopilotClient.java

@@ -269,6 +269,9 @@ static RuntimeConnection resolveDefaultConnection(CopilotClientOptions options,
                 if (options.getPort() != 0) {
                     return inferConnectionFromOptions(options);
                 }
+                if (!options.isUseStdio() || options.getTcpConnectionToken() != null) {
+                    return inferConnectionFromOptions(options);
+                }
                 return RuntimeConnection.forInProcess();
             }
             if (!"stdio".equalsIgnoreCase(envValue)) {

java/sdk/src/main/java/com/github/copilot/JsonRpcClient.java

@@ -18,6 +18,7 @@
 import java.util.concurrent.Executors;
 import java.util.concurrent.atomic.AtomicLong;
 import java.util.function.BiConsumer;
+import java.util.function.Consumer;
 import java.util.logging.Level;
 import java.util.logging.Logger;
 
@@ -59,6 +60,11 @@ private JsonRpcClient(InputStream inputStream, OutputStream outputStream, Socket
 
     private JsonRpcClient(InputStream inputStream, OutputStream outputStream, Socket socket, Process process,
             boolean ownsStreams) {
+        this(inputStream, outputStream, socket, process, ownsStreams, null);
+    }
+
+    private JsonRpcClient(InputStream inputStream, OutputStream outputStream, Socket socket, Process process,
+            boolean ownsStreams, Consumer<JsonRpcClient> initializer) {
         this.inputStream = inputStream;
         this.outputStream = outputStream;
         this.socket = socket;
@@ -69,6 +75,9 @@ private JsonRpcClient(InputStream inputStream, OutputStream outputStream, Socket
             t.setDaemon(true);
             return t;
         });
+        if (initializer != null) {
+            initializer.accept(this);
+        }
         startReader();
     }
 
@@ -100,6 +109,10 @@ public static JsonRpcClient fromSocket(Socket socket) throws IOException {
         return new JsonRpcClient(socket.getInputStream(), socket.getOutputStream(), socket, null);
     }
 
+    static JsonRpcClient fromSocket(Socket socket, Consumer<JsonRpcClient> initializer) throws IOException {
+        return new JsonRpcClient(socket.getInputStream(), socket.getOutputStream(), socket, null, false, initializer);
+    }
+
     /**
      * Creates a JSON-RPC client over arbitrary input/output streams. The client
      * takes ownership of the streams and closes them when {@link #close()} is

java/sdk/src/main/java/com/github/copilot/ffi/NativeRuntimeLoader.java

@@ -135,7 +135,19 @@ public static Path resolve() throws IOException {
      *             if the CLI executable cannot be located
      */
     public static Path resolveEntrypoint() throws IOException {
-        Path runtimePath = resolve();
+        String configuredCli = System.getenv(COPILOT_CLI_PATH_ENV);
+        return resolveEntrypoint(configuredCli, resolve());
+    }
+
+    static Path resolveEntrypoint(String configuredCli, Path runtimePath) throws IOException {
+        if (configuredCli != null && !configuredCli.isBlank()) {
+            Path configuredPath = Path.of(configuredCli).toAbsolutePath().normalize();
+            if (resolveFromCliPath(configuredCli) != null && Files.isRegularFile(configuredPath)
+                    && Files.size(configuredPath) > 0) {
+                return configuredPath;
+            }
+        }
+
         Path parent = runtimePath.getParent();
         String cliName = isWindows() ? CLI_FILENAME_WINDOWS : CLI_FILENAME;
         Path cliPath = parent.resolve(cliName);

java/sdk/src/main/java/com/github/copilot/rpc/CopilotClientOptions.java

@@ -282,13 +282,11 @@ public String getCwd() {
      * Sets the working directory for the CLI process.
      *
      * @param cwd
-     *            the working directory path (must not be {@code null} or empty)
+     *            the working directory path, or {@code null} to clear
      * @return this options instance for method chaining
-     * @throws IllegalArgumentException
-     *             if {@code cwd} is {@code null} or empty
      */
     public CopilotClientOptions setCwd(String cwd) {
-        this.cwd = Objects.requireNonNull(cwd, "cwd must not be null");
+        this.cwd = cwd;
         return this;
     }
 

java/sdk/src/test/java/com/github/copilot/ClientOptionsE2ETest.java

@@ -15,8 +15,6 @@
 
 import org.junit.jupiter.api.Test;
 
-import com.github.copilot.e2e.SkipInProcess;
-
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.github.copilot.generated.rpc.SessionLimitsConfig;
@@ -31,7 +29,6 @@ class ClientOptionsE2ETest {
     private static final ObjectMapper MAPPER = new ObjectMapper();
 
     @Test
-    @SkipInProcess("Exercises direct CLI argument and working-directory forwarding to a spawned stdio subprocess")
     void testShouldForwardAdvancedSessionCreationOptionsToTheCli() throws Exception {
         try (var fake = FakeStdioCli.create()) {
             var workDir = fake.path("create-work");
@@ -98,7 +95,6 @@ void testShouldForwardAdvancedSessionCreationOptionsToTheCli() throws Exception
     }
 
     @Test
-    @SkipInProcess("Exercises direct CLI argument and working-directory forwarding to a spawned stdio subprocess")
     void testShouldForwardSingularProviderConfigurationOnSessionCreation() throws Exception {
         try (var fake = FakeStdioCli.create()) {
             try (var client = fake.createClient()) {
@@ -127,7 +123,6 @@ void testShouldForwardSingularProviderConfigurationOnSessionCreation() throws Ex
     }
 
     @Test
-    @SkipInProcess("Exercises direct CLI argument and working-directory forwarding to a spawned stdio subprocess")
     void testShouldForwardAdvancedSessionResumeOptionsToTheCli() throws Exception {
         try (var fake = FakeStdioCli.create()) {
             var workDir = fake.path("resume-work");

java/sdk/src/test/java/com/github/copilot/ConfigCloneTest.java

@@ -377,6 +377,15 @@ void copilotClientOptionsSetEnvironmentNullClearsExisting() {
         assertTrue(env == null || env.isEmpty());
     }
 
+    @Test
+    void copilotClientOptionsSetCwdNullClearsExisting() {
+        CopilotClientOptions opts = new CopilotClientOptions().setCwd("/tmp");
+
+        opts.setCwd(null);
+
+        assertNull(opts.getCwd());
+    }
+
     @Test
     @SuppressWarnings("deprecation")
     void copilotClientOptionsDeprecatedGithubToken() {