Add e2e test infrastructure with Playwright + node-pty by ryancbahan · Pull Request #6899 · Shopify/cli

ryancbahan · 2026-02-26T23:45:14Z

WHY are these changes introduced?

The CLI team manually runs a ~30-step QA checklist before every release. This is time-consuming and error-prone. The Cucumber suite tests CLI commands non-interactively but cannot handle interactive CLI prompts, long-running dev servers, auth, or browser verification. We need e2e test infra that automates QA and provides regression testing for CLI features.

WHAT is this pull request doing?

Introduces a new packages/e2e/ package with Playwright + node-pty infrastructure for automated end-to-end testing.

Directory structure:

packages/e2e/
  setup/          → Playwright fixture definitions (DI wiring)
    env.ts          worker-scoped XDG isolation & env var management
    cli.ts          CLI process management (execa + node-pty)
  helpers/        → Pure utility functions
    strip-ansi.ts   ESM wrapper for strip-ansi
  tests/          → Test specs
    smoke.spec.ts       validates execa path (shopify version)
    smoke-pty.spec.ts   validates PTY path (shopify version via node-pty)

Key design decisions:

setup/ to distinguish Playwright fixture definitions from static test data
cli.ts provides both exec() (non-interactive via execa) and spawn() (interactive via node-pty with output matching, key sending, process lifecycle)
Worker-scoped env isolation with XDG directory sandboxing
helpers/ contains stateless utility functions with no Playwright dependency

Subsequent PRs in this stack add OAuth login, app scaffold/deploy/dev tests, CI integration, and TOML config regression tests.

How to test your changes?

pnpm test:e2e

Measuring impact

n/a - this is test infrastructure

ryancbahan · 2026-02-26T23:45:28Z

Add e2e tests for app init, deploy, and dev server #6900 : 2 dependent PRs (#6901 , #6903 )
Add e2e test infrastructure with Playwright + node-pty #6899 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-02-27T20:30:06Z

Coverage report

St.^❔	Category	Percentage	Covered / Total
🟡	Statements	78.83%	14475/18363
🟡	Branches	73.14%	7200/9844
🟡	Functions	79.06%	3693/4671
🟡	Lines	79.17%	13675/17274

Test suite run success

3791 tests passing in 1448 suites.

Report generated by 🧪jest coverage report action from 6e1ad36

ryancbahan · 2026-02-27T20:35:23Z

packages/e2e/.env.example

@@ -0,0 +1,19 @@
+# Required: Client ID of the primary test app (must be in the genghis account's org)
+# CI secret: E2E_CLIENT_ID
+SHOPIFY_FLAG_CLIENT_ID=


For initial E2E tests, we won't create new apps but instead use existing ones. We should 100% evolve this, but we need a starting point to monitor and observe stability of this job.

packages/e2e/setup/cli.ts

binks-code-reviewer · 2026-02-27T21:15:55Z

🤖 Code Review · #projects-dev-ai for questions
React with 👍/👎 or reply — all feedback helps improve the agent.

✅ Complete - No issues

📋 History

✅ 1 findings → ✅ 1 findings → ✅ No issues

isaacroldan · 2026-03-05T11:53:15Z

packages/e2e/tests/smoke-pty.spec.ts

+
+test.describe('PTY smoke test', () => {
+  test('shopify version runs via PTY', async ({cli}) => {
+    const proc = await cli.spawn(['version'])


I wasn't able to execute this test initially, i was getting a Error: posix_spawnp failed.
Apparently (and according to claude) pnpm strips the execution permission of binaries installed through it, so the spawn-helper in node-pty didn't have enough permissions. Fixing that this worked, but probably something to account for.

how did you fix? i didn't run into this problem

Claude found the spawn-helper binary inside node_modules and suggested to add execution permission (chmod +x), after that everything worked. Not sure why it happened though, i've installed binaries other times and didn't have this problem. Maybe because I ran pnpm i inside the e2e folder? :thinking_face:

I ran pnpm iagain from the root and it worked fine this time, so maybe we can ignore this

isaacroldan · 2026-03-05T11:53:41Z

packages/e2e/helpers/wait-for-port.ts

+/**
+ * Polls until a TCP connection to host:port succeeds, or timeout is reached.
+ */
+export async function waitForPort(port: number, host = '127.0.0.1', timeoutMs = 30_000): Promise<void> {


Am I wrong or this isn't used anywhere?

isaacroldan · 2026-03-05T11:55:11Z

packages/e2e/helpers/file-edit.ts

+/**
+ * Replaces text in a file. Useful for modifying source files to trigger hot reload.
+ */
+export function replaceInFile(filePath: string, search: string | RegExp, replacement: string): void {


Same for these functions, they are not used right?

packages/e2e/playwright.config.ts

Rename the fixtures directory to setup/ to clearly distinguish Playwright fixture definitions from static test data. Rename cli-process.ts to cli.ts for brevity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ryancbahan mentioned this pull request Feb 27, 2026

Add e2e tests for app init, deploy, and dev server #6900

Open

ryancbahan changed the title ~~add new section for e2e testing~~ Add new section for e2e testing Feb 27, 2026

This was referenced Feb 27, 2026

add tests for deploy #6901

Closed

add app dev tests #6902

Closed

Add e2e tests to PR CI workflow #6903

Open

ryancbahan force-pushed the e2e-testing branch from ad0ad81 to a55826d Compare February 27, 2026 19:41

ryancbahan mentioned this pull request Feb 27, 2026

Add cleanup script for e2e test org #6907

Open

ryancbahan force-pushed the e2e-testing branch from a55826d to c19eb9e Compare February 27, 2026 20:23

ryancbahan marked this pull request as ready for review February 27, 2026 20:23

ryancbahan requested a review from a team as a code owner February 27, 2026 20:23

ryancbahan commented Feb 27, 2026

View reviewed changes

ryancbahan requested review from dmerand and jantnovi February 27, 2026 21:15

binks-code-reviewer bot reviewed Feb 27, 2026

View reviewed changes

packages/e2e/setup/cli.ts Show resolved Hide resolved

ryancbahan requested a review from amcaplan March 2, 2026 14:30

isaacroldan reviewed Mar 5, 2026

View reviewed changes

add e2e section

074270e

ryancbahan force-pushed the e2e-testing branch from c96d6b9 to 074270e Compare March 5, 2026 16:35

binks-code-reviewer bot reviewed Mar 5, 2026

View reviewed changes

packages/e2e/playwright.config.ts Show resolved Hide resolved

ryancbahan mentioned this pull request Mar 5, 2026

Add TOML config regression tests #6940

Open

ryancbahan and others added 2 commits March 5, 2026 11:20

Remove unused e2e helpers (wait-for-port, file-edit)

5b17def

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ryancbahan changed the title ~~Add new section for e2e testing~~ Add e2e test infrastructure with Playwright + node-pty Mar 5, 2026

retrigger CI

6e1ad36

ryancbahan requested a review from isaacroldan March 5, 2026 19:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e test infrastructure with Playwright + node-pty#6899

Add e2e test infrastructure with Playwright + node-pty#6899
ryancbahan wants to merge 4 commits intomainfrom
e2e-testing

ryancbahan commented Feb 26, 2026 •

edited

Loading

Uh oh!

ryancbahan commented Feb 26, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 27, 2026 •

edited

Loading

Uh oh!

ryancbahan Feb 27, 2026

Uh oh!

Uh oh!

binks-code-reviewer bot commented Feb 27, 2026 •

edited

Loading

Uh oh!

isaacroldan Mar 5, 2026

Uh oh!

ryancbahan Mar 5, 2026

Uh oh!

isaacroldan Mar 6, 2026

Uh oh!

isaacroldan Mar 5, 2026

Uh oh!

isaacroldan Mar 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ryancbahan commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHY are these changes introduced?

WHAT is this pull request doing?

How to test your changes?

Measuring impact

Uh oh!

ryancbahan commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Test suite run success

Uh oh!

ryancbahan Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

binks-code-reviewer bot commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

isaacroldan Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

ryancbahan Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

isaacroldan Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

isaacroldan Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

isaacroldan Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ryancbahan commented Feb 26, 2026 •

edited

Loading

ryancbahan commented Feb 26, 2026 •

edited

Loading

github-actions bot commented Feb 27, 2026 •

edited

Loading

binks-code-reviewer bot commented Feb 27, 2026 •

edited

Loading