Your first green run.
Five checkpoints take you from an empty machine to a real browser test with a report you can open and share.
Add the TestPilot command
Install the public launcher globally so the testpilot command is available from any project.
npm install --global @macevoorhees/testpilotThe public launcher runs on the current Node LTS generation.
macOS Apple silicon/Intel, Windows x64, or glibc Linux arm64/x64.
TestPilot brings .NET, Playwright, and its console. Alpine/musl is not supported.
Use --global during installation. A local npm install may succeed but leave a bare testpilot command unavailable in your shell.
A test that explains itself.
Click any line to see what TestPilot understands, how it carries context forward, and what evidence the line must produce.
new test named "…"Starts a testThe header is unindented. One file can contain multiple tests.
two leading spacesCreates a stepEvery indented line is timed, executed, and reported independently.
@tags recipe, serialAdds metadataTags apply to one test. Only serial changes built-in scheduling.
@viewport 1920x1200Changes one viewportOptional test metadata from 640×480 through 7680×4320.
# a helpful commentDocuments intentComments and blank lines are ignored by execution.
.pilot or .testpilotNames the fileOrdinary unmatched steps use Computer Use automatically.
Reuse the same value without model memory.
Use a named saved value for generated numbers, strings, and booleans—or a visible run-ID placeholder when the identifier should remain obvious in the file.
- ✓ Isolated to one test
- ✓ Generated and owned by the framework
- ✓ Sensitive values are rejected
new test named "Create a unique recipe"
create a recipe named "AUTO_RECIPE_<final 8 characters of the run id>"
search for "AUTO_RECIPE_<final 8 characters of the run id>"
verify that exact recipe is visibleSay which visible dialog or section owns the field, ask to replace its entire value and verify the exact whole value, then name the exact Save control to click once. Every test may change data when its authored steps require it.
Teach the common routes once.
Action aliases keep specs natural while stable guidance or protected browser recipes handle workflows the team repeats.
Add product-specific guidance while Computer Use still reads and controls the visible browser. The executor line is optional because computer is the default.
new action named "open analytics"
aliases "go to analytics", "open the analytics module"
use executor "computer"
guidance:
Open Analytics from the visible application navigation.
Confirm the page heading before marking the step complete.Use a deterministic recipe for credential-bearing flows. Required values resolve locally and are not included in the model prompt.
new action named "login"
aliases "login using the application credentials"
use executor "browser"
requires "APP_EMAIL"
requires "APP_PASSWORD"
browser:
go to the environment start page
fill "input[name='email']" with credential "APP_EMAIL"
fill "input[name='password']" with credential "APP_PASSWORD"
click "button[type='submit']"
wait until the page URL does not contain "login"go to the environment start pagefill "selector" with credential "VARIABLE"fill "selector" with text "value"click text "Exact label"click "selector"wait 1000 millisecondswait until the page URL does not contain "login"verify "selector" is visibletestpilot actions list
testpilot actions validatetestpilot credentials list
testpilot credentials set APP_EMAILOne suite. Clear boundaries.
Each project owns its model key, reusable actions, test files, output, and strict list of environments.
my-suite/ ├── actions/ ├── specs/ │ └── example.pilot ├── .env.local ├── .env.example ├── .gitignore ├── README.md └── testpilot.config.json
Project API key and local variables. Ignored by source control.
Natural-language tests that teammates can review.
Aliases, guidance, and protected browser recipes.
Model, deadline, directories, and HTTPS environments.
{
"name": "My TestPilot suite",
"specsDirectory": "specs",
"actionsDirectory": "actions",
"outputDirectory": "output",
"openAi": {
"apiKeyVariable": "TEST_PILOT_AI_API_KEY",
"model": "gpt-5.6-luna",
"maxTurnsPerStep": 30
},
"execution": {
"stepTimeoutSeconds": 120
},
"safety": {
"allowModelCode": false
},
"environments": {
"example": {
"baseUrl": "https://example.com/",
"allowedHosts": ["example.com"]
}
}
}openAi.maxTurnsPerStep defaults to 30. Change it to any whole number from 1 through 200. The limit resets for every authored step, so it is not a cap on the whole test.
The selected base host must be allowlisted, and top-level navigation outside it is blocked.
Override with TESTPILOT_MODEL only when the chosen model supports the built-in computer tool.
allowModelCode stays off in new projects. Enable it only on disposable, least-privilege runners; screenshot Computer Use remains the fallback.
Every limit resets for the next authored step.
execution.stepTimeoutSeconds limits one natural-language line—not the whole test.
openAi.maxTurnsPerStep is configurable from 1 through 200 and stops a fast no-progress loop.
openAi.requestTimeoutSeconds limits one OpenAI request.
browser.operationTimeoutSeconds limits one Playwright operation.
Run exactly what you mean.
Target one file, a folder, or the entire configured suite. Build the command visually, then paste it into your project terminal.
Choose the run. Copy the command.
testpilot run specs --env exampleNothing runs from this page. Paste the command into a terminal inside your test project.
Parallelizes normal tests, then runs serial tests one at a time.
Uses the local worker pool for normal tests, then preserves serial tests.
Explicit concurrencyRuns every selected test in stable order with one active browser test.
Debugging and tight memoryTests and individual steps show running, passed, failed, or skipped. Horizontal dividers keep runs readable.
Toggle model and browser activity while the run continues. --details starts it visible.
Failed steps and tests are not automatically rerun. Only transient API requests retry—initial request plus two attempts by default.
A failed or timed-out step skips the rest of that test while unrelated parallel tests continue.
Parallelism at the right layer.
Workers share one machine. Shards split the suite across CI containers. Combine both deliberately instead of making every container fight for memory.
--workers 4Up to four isolated browser contexts inside the current machine or container.
--shard 2/4One deterministic slice. CircleCI node index and total are detected automatically.
Give each container multiple files but run them serially with --workers 1.
One license pair across every shard.
Pre-provision an activation token and matching stable installation ID. Do not activate parallel containers independently with the raw license key.
TESTPILOT_ACTIVATION_TOKEN=tpa_...
TESTPILOT_INSTALLATION_ID=circleci-my-project
TEST_PILOT_AI_API_KEY=your-project-keytestpilot merge shard-a/run.json shard-b/run.json \
--out aggregate-report \
--expect-tests 20Built-in multi-shard selection rejects a set that contains a serial test. For mixed suites, distribute whole spec files in the CI workflow and run each assigned file set locally with one worker.
Every result brings receipts.
Each run writes one portable bundle for people, CI systems, and fast visual diagnosis.
Expandable tests, per-step status, timings, details, and screenshots.
The structured source used to merge evidence from multiple shards.
Machine-readable test results for CircleCI and other CI interfaces.
A screenshot after each step by default, with failure evidence always attempted.
6 of 6 steps passed
✓ passed verify final sales equals the saved "sales amount" 4.2s
The complete command deck.
Search every TestPilot command by its name, outcome, or the problem you are trying to solve.
Install TestPilot
Install the public launcher globally and make the testpilot command available in your shell.
npm install --global @macevoorhees/testpilotActivate this installation
Enter a license key through a hidden prompt and download the verified private runtime.
testpilot activateRemove local activation
Remove the saved activation state from this computer.
testpilot deactivateCreate a test project
Scaffold a standalone suite. Omit the directory to initialize the current folder.
testpilot init [directory]Check readiness
Inspect configuration, browser availability, and declared credential presence without changing them.
testpilot doctor [--project|-p directory]Install the compatible browser
Explicitly install TestPilot's pinned Chromium build when automatic readiness needs recovery.
testpilot install-browserValidate the suite
Check config, reusable actions, and selected .pilot files without running the browser.
testpilot validate [paths...] [--project|-p directory]List environments
Show the project's strict environment and host allowlist.
testpilot environments|envs [--project|-p directory]List reusable actions
Show action names, aliases, executors, and required variables in the selected suite.
testpilot actions list [--project|-p directory]Validate reusable actions
Check every .action file and its browser recipe without executing it.
testpilot actions validate [--project|-p directory]Check credential presence
Report whether declared variables are available without printing their values.
testpilot credentials list [--project|-p directory]Save a local credential
Prompt privately and save one suite-scoped credential in macOS Keychain.
testpilot credentials set VARIABLE [--project|-p directory]Convert a text file
Turn pasted natural-language lines into a .pilot test and reuse any matching action aliases.
testpilot convert scenario.txt --name "My test"Convert piped text
Read plain text from standard input and choose the destination file explicitly.
testpilot convert - --name "My test" --out specs/my-test.pilotRun a folder
Discover and run every selected test under a directory using automatic scheduling.
testpilot run specs --env exampleWatch one file run
Open a visible browser and run only the selected .pilot file.
testpilot run specs/checkout.pilot --env example --headedRun locally in parallel
Use up to four isolated browser contexts on this machine.
testpilot run specs --env example --mode parallel --workers 4Run everything serially
Execute each selected test in stable order with one active test at a time.
testpilot run specs --env example --mode serialRun one CI shard
Select a deterministic one-based slice of the suite. CircleCI node variables are detected automatically.
testpilot run specs --env example --shard 2/4Filter by tags
Include or exclude organizational tags. Repeat either option to add more filters.
testpilot run specs --env example --include-tag checkout --exclude-tag slowPreview the selection
Validate discovery and emit reports without opening a browser or calling OpenAI.
testpilot run specs --env example --dry-runPrint detailed activity
Keep model and browser activity visible continuously. Locally, Ctrl+O toggles the same panel.
testpilot run specs --env example --detailsMerge CI results
Combine shard JSON into one portable HTML, JSON, and JUnit bundle and reject partial aggregation.
testpilot merge shard-a/run.json shard-b/run.json --out aggregate-report --expect-tests 20Open terminal help
Show the general command list or focused usage for one command.
testpilot help [command]Show the installed version
Print the active TestPilot runtime version.
testpilot versionTEST_PILOT_AI_API_KEYOpenAI project key owned by the consumer suite
TESTPILOT_MODELOptional computer-capable model override
TESTPILOT_ENVDefault selected environment
TESTPILOT_EXECUTION_MODEDefault auto, serial, or parallel mode
TESTPILOT_OUTPUT_DIRECTORYAlternate report root, useful per CI shard
TESTPILOT_LOG_LEVEL=debugShort diagnostic details for unexpected failures
CI=trueConservative worker defaults and noninteractive output
CIRCLE_NODE_INDEX / TOTALAutomatic deterministic CircleCI shard selection
When the runway is not clear.
Start with the symptom. Each answer points to the smallest safe check or command.
01testpilot: command not found+
Install the launcher globally: npm install --global @macevoorhees/testpilot. A local npm install does not normally add a bare command to your shell path.
02The browser is not ready+
Run testpilot doctor first. A normal init or run checks Chrome automatically; testpilot install-browser explicitly installs the pinned Chromium build when recovery is needed.
03The project or environment cannot be found+
Run inside the suite, or add --project path/to/suite. Use testpilot environments to see the exact names accepted by --env.
04The OpenAI key is missing+
Add the suite's key to its own ignored .env.local as TEST_PILOT_AI_API_KEY. TestPilot never borrows a key from another project.
05A step seems stuck+
The default hard deadline is 120 seconds for each authored step, not the whole test. Adjust execution.stepTimeoutSeconds from 1–3600 only when the workflow genuinely needs more time.
06A failed test did not retry+
That is intentional. TestPilot retries transient API requests, not a failed step or entire test. Read the HTML report and screenshot, fix the product or test intent, then run it again deliberately.
07CircleCI containers cannot activate+
Give every parallel job the same pre-provisioned TESTPILOT_ACTIVATION_TOKEN and matching stable TESTPILOT_INSTALLATION_ID. Do not rotate one raw license key independently in each job.