Run OttoTester from your CI

OttoTester’s Pipeline Integration is the path to take when your CI workflow should explicitly invoke OttoTester — instead of OttoTester listening for deploys via the GitHub App. Your pipeline is the actor; OttoTester is the callee. The call goes to the public API, authenticated with a personal access token; OttoTester runs what your snippet tells it to run, against the URL your snippet tells it to test.

This is the right shape when:

Your deploy platform doesn’t post deployment_status events to GitHub, or you’re not on GitHub at all.
You want OttoTester to gate the next stage of a release — after a preview deploy, gate the merge to main; after a staging deploy, gate the promotion to production; after a production deploy, fail the pipeline if tests regressed so your rollback or alerting kicks in.
You want to wire OttoTester into pipeline logic that doesn’t reduce to “fire on every deploy” — for example, only run the full regression on the release branch, or chain an OttoTester run after a database migration step.

For the alternative — OttoTester listening for GitHub deployment events and firing runs automatically — see the GitHub App half of Connect to CI/CD. The two paths coexist; teams use either or both per app.

Set it up in four steps

The setup runs end-to-end in about five minutes.

1. Generate a personal access token

The pipeline authenticates with a personal access token (PAT) you generate from Settings → Account → Access tokens. Give it a name (CI: production-deploys or similar) and scope it to the workspace that holds the variant you want to test. OttoTester shows the full token once — copy it then.

A token acts as you: it carries whatever permissions your account has on that workspace. If your role changes, every token you hold changes with it. See API tokens for the full lifecycle (name, expiry, last-used, revoke).

2. Store the token as a CI secret

Add the token to your CI provider’s secrets store under the name OTTOTESTER_API_TOKEN. That name matches what the snippet OttoTester generates expects — keep it consistent so the snippet works without edits.

GitHub Actions — repository or organization secret.
GitLab CI — CI/CD variable, masked.
Jenkins, CircleCI, others — their equivalent secret store.

Never commit the token. Give each integration its own token so a leak revokes cleanly without disturbing the rest.

3. Build the snippet in OttoTester

Open the application that owns the variant you want to test. Click the Integrations tab, find the Pipeline Integration card, and click the variant you want to wire. A drawer slides in.

In the drawer:

Pick a suite. This is the Replay template the snippet will run — the named selection of tests plus run settings (browser, headless, retries) the operator saved earlier. The drawer pre-fills the variant’s first template; switch it if you have several.
Pick a target-URL strategy. Three options; pick the one that matches how your pipeline knows the URL.
Pick an output format. The mode tabs — GitHub Actions, curl, GitLab CI — render the same call in three CI flavors. The drawer’s code block updates live as you change suite, URL strategy, or mode.

Copy the snippet.

4. Paste it into your repo

Paste the snippet into the file the drawer’s filename header suggests — .github/workflows/ottotester.yml, .gitlab-ci.yml, or a shell script. Commit, push, and the next time the workflow runs the OttoTester step fires.

The three target-URL strategies

The URL OttoTester tests against is the one piece of data that’s different on every run — yesterday’s preview deploy lives at a different URL than today’s. The drawer offers three patterns; the right pick depends on how your pipeline knows the URL.

From an env var passed by your pipeline

Best when each run’s URL is unique — preview deploys, ephemeral environments, branch-named URLs. Your pipeline writes the URL into a variable (DEPLOY_URL) before the OttoTester step runs, and the snippet picks it up.

The snippet expands $DEPLOY_URL at call time, so the JSON body sent to OttoTester carries the real URL. Your job is to make sure the variable is set before the OttoTester step — usually one line in the step before it (the deploy step’s output, an environment write, a ${{ steps.deploy.outputs.url }} reference).

Hardcode a URL

Best for fixed environments — a staging URL that never moves, a long-lived QA environment. Type the URL into the drawer, and the snippet literal-strings it. Change it any time by editing the snippet.

Use the variant’s configured URL

Best when this variant exists specifically for one environment — the variant’s own configured URL is the URL to test. The snippet omits target_url entirely; OttoTester uses the variant’s configured URL on the run.

This is the simplest snippet and the right default for a variant set up as “production” with prod’s URL baked in.

Sync vs async — wait or poll

The snippet defaults to sync mode (wait_for_completion: true) — the API holds the connection open until the run finishes and returns the pass/fail summary in one response. The pipeline step waits, then exits red if any test failed (because fail_on_test_failure: true).

Sync is the right default for short runs (under fifteen minutes) where the lowest-latency feedback matters. Past that, your CI provider’s idle-connection timeout may close the connection before OttoTester is done; the server’s own hold timeout caps at fifteen minutes by default (operator-tunable). When the server’s timeout fires before the run finishes, the response is a 504 carrying the run id — you can fall back to polling by run id.

For long runs, switch to async mode: edit the snippet to set wait_for_completion: false. The call returns the run id immediately; your pipeline polls the API for the terminal status. The vendored composite action (see below) handles polling for you when invoked in async mode.

Use the vendored GitHub Action

If you’re on GitHub Actions, there’s a vendored composite action that wraps the raw curl call with the build summary, conventional exit codes, and 429 handling. Reference it directly:

- name: Run OttoTester suite
  # Pin to a release tag or a full commit SHA in production (e.g.
  # @v1 or @abc123…) so a future change on the default branch
  # can't surprise your pipeline. `@main` is fine while you're
  # trying it out.
  uses: BPS-Consulting/ottotester/.github/actions/run-tests@main
  with:
    # Cloud customers: https://app.ottotester.ai.
    # Self-hosted: your OttoTester instance's base URL, no trailing slash.
    host: ${{ secrets.OTTOTESTER_HOST }}
    api-key: ${{ secrets.OTTOTESTER_API_TOKEN }}
    workspace-id: <your-workspace-id>
    template-id: <your-template-id>
    target-url: ${{ steps.deploy.outputs.url }}     # optional
    wait-for-completion: 'false'                    # poll mode for long runs

OTTOTESTER_HOST is a separate secret from the token — add the base URL to your CI provider’s secrets store alongside OTTOTESTER_API_TOKEN. Cloud customers use https://app.ottotester.ai; self-hosted deployments use whichever URL your instance runs at.

What the action adds on top of the raw call:

A per-job build summary — pass / fail / skipped / total counts, cost in cents, and a per-test breakdown with error messages on failed. Surfaces at the top of the workflow’s run page.
Conventional exit codes — 0 for pass, 1 for fail, 2 for errored / cancelled / rate-limited, 124 for timeout. Downstream steps can continue-on-error: false and trust the exit code.
Polling for long runs — set wait-for-completion: 'false' and the action polls the run’s status URL until terminal, with a configurable interval. Skip the snippet’s sync-mode hold timeout entirely.
Outputs for downstream steps — run-id, status, passed, failed, skipped, total, cost-cents, report-url. Wire them into a if: condition, a Slack notify, a build badge.

The drawer’s GitHub Actions tab currently emits the raw curl form for compatibility — it works today, and the eventual swap to uses: reference is a few-line edit you can do whenever you want the polish.

Troubleshooting

`401 Unauthorized`

The token is missing, malformed, or revoked. Check the secret is set in your CI environment and the request is sending Authorization: Bearer <token> (not just <token>). If the token was deleted from Settings → Account → Access tokens, the next call gets a 401 — generate a new one and update the secret.

`403 Forbidden`

The token is valid but doesn’t have access to the workspace named in the request. PATs are scoped at creation time; if the token doesn’t list the target workspace in its scope, every call to that workspace returns 403. Either generate a new token scoped to the right workspace, or — if your account doesn’t have access either — ask a workspace admin to add you.

`429 Too Many Requests`

Two different limits can return 429 on this endpoint; the response body names which one. Both include a Retry-After header (in seconds) — wait at least that long before retrying.

Per-token request rate — your token has fired too many calls in the current minute. The default cap is 30 requests per minute per token; that’s plenty for a normal CI workflow that fires a handful of runs a day, but a workflow that retries aggressively on every transient failure can trip it. Back off the retry rate, or split into multiple tokens scoped to different workspaces.
Concurrent runs in flight — your workspace’s plan caps how many runs can be active at once, and your call would push you over. This isn’t a request-rate problem; it’s a parallelism problem. Either reduce how many pipelines invoke OttoTester in parallel, serialize the calls, or upgrade the plan if the cap is consistently the bottleneck.

The right mitigation depends on which limit you hit — back-off helps the first, reducing parallel runs (or upgrading) helps the second. Check the response body before changing anything.

`504 Gateway Timeout` (sync mode)

Sync mode held the connection open until the server’s hold timeout fired (fifteen minutes by default) and the run still hadn’t finished. The response includes the run id. Fall back to polling the run by id, or switch the snippet to wait_for_completion: false and use async mode going forward.

The run started, but tests run against the wrong URL

Check the snippet’s target_url value. In env-var mode, the issue is usually that DEPLOY_URL isn’t set when the OttoTester step runs — either the deploy step didn’t export it, or it’s named something else. Add an echo "$DEPLOY_URL" to the step right before the OttoTester step to confirm the value the snippet will see. In variant-default mode, the URL comes from the variant’s configured URL in OttoTester — open the variant and check the Base URL field.

What the API call looks like

For reference (and for non-GitHub / non-GitLab CIs), the call the snippet makes is:

POST /api/v1/pipelines/runs
Authorization: Bearer <your-token>
Content-Type: application/json

{
  "workspace_id": "<workspace-id>",
  "template_id": "<template-id>",
  "target_url": "https://your-deploy-url",
  "wait_for_completion": true,
  "fail_on_test_failure": true
}

The response is the run summary — status, per-test counts, cost, and the report URL. The drawer hides all of this behind the snippet builder; this is the underlying shape if you need to wire it by hand into a CI that isn’t on the mode-tab list.

Workflow → Connect to CI/CD, Schedule recurring runs, Read a run report
Reference → API tokens
Concept → Variants, Test suites, Replay templates