Skip to content

Local E2E Multi-Stack Runbook

Use this runbook when running Playwright E2E tests from more than one SyRF worktree. Each worktree receives its own host ports, Compose project, containers, pid files, logs, Lambda build artifacts, runtime web configuration, and browser auth-cookie names. The port resolver uses lsof plus Python 3's standard-library fcntl locking. lsof is available by default on macOS and can be installed from the standard package repositories on Linux; the existing Python 3 prerequisite supplies the same kernel-backed lock on both platforms without a separate flock command.

Reuse and state isolation

Reuse cached Docker image layers; recreate stateful containers and their data for independent acceptance runs. Keeping an iterative stack running is useful for debugging, but it does not prove isolation: MongoDB, SQL Server, RabbitMQ, LocalStack and browser sessions can retain state. Browser acceptance runs must also create fresh contexts and regenerate their authentication storage.

Run bash e2e/scripts/test-resource-isolation.sh from a PR worktree to reproduce the image-reuse check. It recreates that worktree's owned E2E stack three times, seeding canaries in MongoDB, SQL Server, a durable RabbitMQ queue and an S3 bucket. It checks that canaries are absent after normal teardown and after a simulated partial teardown, while image IDs remain unchanged. Each cycle verifies that owned containers and the network were removed; an EXIT trap also cleans up on failure. Logs and container/image identities remain in the printed temporary directory. This is a local real-store check, separate from CI cancellation-hook verification and from the full browser smoke suite.

Dedicated CI recovery

The fixed CI stack runs on pomegranate-01. Each Compose service, network, volume and LocalStack Lambda executor carries org.syrf.e2e.owner (repository, run, attempt and job) and org.syrf.e2e.listener. These labels also connect the workflow to the installed root-owned completion hook's listener-scoped fallback. The previous workflow omitted those labels, so that hook could not recognize containers left by a failed or cancelled bring-up.

After checking the listener prerequisites and checking out the trusted head, ci-resources.py recover inventories all fixed-stack candidates and checks each prior owner against GitHub's run-attempt status. Only completed attempts from the same repository qualify. Active owners, missing ownership, foreign network attachments and volumes used by foreign containers stop recovery before any deletion, with all blockers listed. Legacy resources without owner labels need separate provenance investigation; a matching name is insufficient.

Normal teardown checks the current owner before removal, retries partial Docker failures and verifies absence. The completion hook handles hard cancellation. Unrelated PR label events have a distinct concurrency group so they cannot cancel an active suite. The additional cleanup step has time reserved in the test budget. The job needs actions: read to prove prior attempts completed; it gains no write permission. No product feature flag is needed for this CI-only repair.

Validate recovery with python3 e2e/scripts/test-ci-resources.py and the workflow contract with bash .github/scripts/test-e2e-concurrency.sh. A green trigger job with a skipped E2E job is not test evidence: record the executed smoke count, report artifacts and cleanup result, followed by a second clean-start run.

Prerequisites

  • Docker
  • .NET 10 SDK
  • Node.js 22 and pnpm
  • Python 3 (also used for atomic reservation retirement and Lambda tooling)
  • lsof
  • Chromium for Playwright: cd e2e && pnpm exec playwright install chromium

Start a stack

From the worktree root:

bash e2e/run-local.sh --spec infrastructure-isolation --iterative

The first run builds the required services. Do not use --skip-build until the worktree has current binaries. --iterative leaves this worktree's Docker infrastructure running for subsequent test runs; the host .NET processes still stop when the command exits.

Direct Playwright commands use the same resolver automatically, but they do not start the API, PM, or Quartz host processes. Use them only while those services are already running for this worktree's resolved allocation:

cd e2e
pnpm exec playwright test tests/infrastructure-isolation.spec.ts --project=@full

Do not run docker compose with the file's fallback defaults. That bypasses the reservation and worktree ownership environment. For infrastructure-only setup, use the ownership-aware script:

bash e2e/scripts/setup.sh

Inspect the allocation

source e2e/scripts/ports.sh
e2e_print_ports

For a pr<N>.<slug> worktree, the preferred allocation is:

slot   = PR number mod 41
offset = slot * 137
port   = service base port + offset

There are 41 slots. The 137-port step prevents ports from different services and stacks overlapping, while keeping the highest possible assigned port below Linux's default ephemeral range. Offset 0 preserves the established CI ports.

Every Compose container is labelled with org.syrf.e2e.worktree. A resolver atomically creates /tmp/syrf-e2e-port-reservations/offset-<offset> before it checks ports or starts Compose. The reservation records the canonical worktree, a stable owner token, and the owning process identity. A second worktree that maps to the same preferred slot therefore advances before either stack binds a port; it cannot observe and accept the same free slot.

A reservation is reusable only by the same resolver lifecycle or by the worktree whose labelled stack is already live. Before allocating, the resolver scans every slot for that worktree's existing stack, so a prior fallback remains stable even if the preferred slot later becomes free. A dead owner is reclaimed only after confirming that it has no matching container and no listener remains in the slot. Release and stale reclamation atomically rename the directory to an inode-specific hidden tombstone; a delayed old owner therefore cannot delete a new reservation created at the same offset. External listeners make the slot unavailable and the resolver releases its new reservation before trying the next slot. An explicit E2E_PORT_OFFSET=<number> must be one of the 41 supported offsets (slot * 137) and fails if that slot is occupied rather than sharing another stack. Stopped deterministic containers also occupy a slot unless their worktree ownership label matches the current checkout.

CI exports E2E_PORT_OFFSET=0 for the whole E2E job because later workflow steps intentionally use the established default ports and container names. If offset 0 is unavailable, CI fails instead of silently moving to another slot.

Browser session isolation

Browser cookies are scoped by hostname, not port. run-local.sh therefore sets BffAuth:SessionCookieName to syrf-session-$E2E_STACK and BffAuth:PkceCookieName to syrf-pkce-$E2E_STACK. The API uses those same configured names for session authentication and for the complete OIDC PKCE issuance, callback, and deletion lifecycle. Two localhost stacks can consequently remain signed in and complete simultaneous interactive logins in one browser profile without overwriting each other's auth cookies.

Artifacts and cleanup

Generated process files are scoped by the resolved $E2E_STACK:

  • /tmp/$E2E_STACK-{api,pm,quartz}.{pid,log}
  • /tmp/$E2E_STACK-lambda-publish
  • /tmp/$E2E_STACK-lambda.zip

Normal teardown is ownership-aware and must be run from the same worktree. It also enumerates the exact stack network before removing any LocalStack Lambda executor that Docker Compose does not manage. The reservation is released only after the matching labelled stack is gone and both the worktree and owner token match:

source e2e/scripts/ports.sh
bash e2e/scripts/teardown.sh

Do not stop containers or processes by an unscoped name. If diagnosing a stale container, inspect its org.syrf.e2e.worktree label before taking action.

Verify isolation logic

bash e2e/scripts/ports.test.sh

The test covers preferred-slot selection, labelled-stack and prior-fallback reuse, fallback when a slot belongs elsewhere, concurrent same-slot starts, stale-owner recovery, ownership-checked release, external listener fallback, explicit-offset refusal, the cross-stack collision invariant, and the ephemeral-port ceiling.

The Pomegranate CI setup accepts the Compose file’s stack-scoped LocalStack identity and the deployment script’s existing bucket-versioning and notifier metadata setup. It only injects missing legacy setup, so native initialization is not duplicated before browser tests.

CI performance work

The measured performance plan separates runtime build reduction from multi-stack CI. Local multi-stack support does not by itself make two shards in one writable checkout safe, or add concurrent runner capacity.