Connect to CI/CD

OttoTester connects to your CI/CD pipeline in two ways — triggering runs, and filing defects.

Picking between the two trigger paths

There are two ways to fire an OttoTester run from your CI/CD. They aren’t mutually exclusive — you can use either or both — but for a given app most teams pick one. The difference comes down to who tells OttoTester to start the run:

	GitHub App (webhook)	Pipeline Integration (API)
The short version	OttoTester listens to your repo’s deploys	Your CI calls OttoTester directly
What kicks the run off	Your deploy platform reported a successful deploy to GitHub	A step in your CI workflow ran
Where you configure the trigger	In OttoTester (a trigger rule on the App Integrations tab)	In your own repo (a YAML step in your CI workflow)
When tests run	After every successful deploy your platform reports	Whenever your CI workflow says to — push, PR, schedule, manual click, or after a preview/staging deploy to gate the next stage
Best for	Hands-off post-deploy validation, especially with a modern PaaS that reports deploys to GitHub automatically	Custom timing, gating a stage promotion (e.g., preview → main, staging → prod), or wiring OttoTester into non-GitHub CI
Setup effort	Install the OttoTester GitHub App + add trigger rules in the OttoTester UI	Generate an API token + paste a snippet into your workflow YAML

The one question that usually settles it: do you want OttoTester to fire automatically on every successful deploy, or do you want your CI workflow to invoke it explicitly? The first answer points at the GitHub App; the second points at Pipeline Integration.

Either way, OttoTester isn’t making temporal decisions on its own — it’s responding to something your infrastructure did. The GitHub App path is reactive (OttoTester sees an event your deploy platform reported to GitHub and matches it against rules you wrote); the Pipeline Integration path is imperative (your CI explicitly calls the API).

Trigger a run from your pipeline (Pipeline Integration)

You can start an OttoTester run as a step in your build or release pipeline and have the pipeline wait for the result — so a regression caught by the run gates the next stage of your release.

It works through the OttoTester API. Your pipeline:

1. Starts a run — one request, naming the variant and the test suite to run.
2. Waits for it to finish — asking the API to hold the connection until the run terminates, or polling for the result.
3. Reads pass or fail — and exits accordingly, so a failed test turns the build red.

Any CI system that can make an HTTP request can do this — GitHub Actions, GitLab CI, Jenkins, CircleCI, and others. See Run OttoTester from your CI for the step-by-step walkthrough: generate a token, paste the snippet OttoTester builds, and the pipeline is wired.

The GitHub App — three steps, three scopes

If your team uses GitHub, install the OttoTester GitHub App. Once it’s installed, GitHub starts streaming your repo’s events to OttoTester. OttoTester picks the successful deployment events out of that stream, matches them against trigger rules you’ve written, and fires runs against the URL the deploy actually went to. Results post back as native PR Check Runs.

Two layers of “registration” worth understanding:

• Subscribing to the event stream — happens automatically when you install the GitHub App on your repo. GitHub starts sending OttoTester everything happening in that repo (push, PR, deployment_status, check_run actions). OttoTester can’t pick and choose at this layer; it gets the firehose.
• Filtering the stream into runs — happens when you write a trigger rule in the OttoTester UI. The rule says “when a successful deploy with environment label Production lands on this repo, run the regression suite.” Without the rule, OttoTester sees the event but has nothing to do with it.

So OttoTester doesn’t decide when to fire on its own — it sees what GitHub sends, applies your rules, runs what matches. Setup runs in three steps, each in the scope it belongs to:

1. Connect on the workspace. A workspace admin opens Settings → Workspace → Integrations and installs the OttoTester GitHub App on the GitHub org. This is the connection — secrets, the App installation, the list of repos OttoTester can see. There’s exactly one GitHub connection per workspace; a team with two GitHub orgs creates a second workspace.

2. Bind on the application. A tester or admin opens the application → Integrations tab and picks which of the workspace’s GitHub repos this app tracks. This is the binding — set once, almost never changes.

3. Add trigger rules per variant. On the application’s Integrations tab, the GitHub card lists every variant. Click a variant to open its trigger-rules drawer and add a rule — “when a deploy to Preview is reported as successful on this repo, run this variant’s smoke-suite replay template.” Rules live per-variant because each variant has its own templates and its own URL; the rules you write for one variant only fire runs against that variant.

The same shape applies to Slack — connect on the workspace, pick a default channel on the application, override per variant from the Slack card’s variant drawer where the variant needs its own channel.

How OttoTester picks up deploys

OttoTester triggers off GitHub’s deployment_status event. GitHub fires this event automatically when a deployment platform reports a successful deploy back to the repository — and the event carries the URL of the deploy, which is what OttoTester targets the test run against.

That matters: the URL OttoTester tests is the URL the deploy actually went to, not the variant’s static default URL. So a PR’s preview deploy gets tested against the preview, and a production deploy gets tested against production — automatically, without you wiring the URLs by hand.

How OttoTester picks up deploys depends on your setup. Three customer flavors cover almost everyone:

If you deploy via a modern PaaS

If you deploy via a hosted PaaS connected to your GitHub repo — Vercel, Netlify, Cloudflare Pages, AWS Amplify, Render, Fly.io, Railway — you’re done as soon as you install the OttoTester GitHub App. The PaaS reports each successful deploy back to GitHub automatically. OttoTester picks the event up and runs against the deployed URL.

Add a trigger rule on the variant with the Replay template you want to run. Leave Environment blank to fire on every successful deploy, or set it to Preview or Production to differentiate (more on that below).

If you deploy via GitHub Actions using Environments

If your deploy step is a GitHub Actions workflow that uses GitHub’s Environments feature, deployment events fire automatically. Any job with environment: <name> in its YAML — for example:

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: ./deploy.sh

— triggers a deployment_status event when the job finishes successfully. OttoTester picks it up. No further configuration.

If you deploy via bare GitHub Actions steps

If your GitHub Actions workflow deploys without using the environment: keyword — for example, a raw kubectl apply or terraform apply step — GitHub doesn’t know a deployment happened, so no deployment_status event fires.

The one-line fix: stick environment: <name> on the job that does the deploy. Any name works — production, staging, preview, whatever matches how you think about the deploy. With that single line, GitHub starts firing deployment events and OttoTester picks them up.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production   # ← this line is the whole change
    steps:
      - run: kubectl apply -f deployment.yaml

Using the Environment filter

The trigger rule’s optional Environment field matches the deployment’s environment label — exactly. Leave it blank to fire on every successful deploy, or set it to a specific value to scope the rule.

The most common pattern: different test rigor per stage. Add two rules on the same variant:

• Environment Preview → run a Smoke template. Fast, on every PR’s preview deploy.
• Environment Production → run a Regression template. Thorough, only on production deploys.

That way a noisy PR doesn’t kick off a 30-minute regression suite, but a production deploy still gets the full validation. Same variant, same repo, two rules.

Match exactly what your deploy platform reports as the environment label — Vercel uses Preview and Production, Netlify uses production and deploy-preview, GitHub Environments uses whatever name you put in the workflow YAML. Check a recent deployment in GitHub’s Deployments view if you’re not sure.

Authenticate with an API token

If you wire OttoTester into a CI system using the Pipeline Integration path (GitHub Actions invoking the API directly, GitLab CI, Jenkins, a plain curl step), the pipeline authenticates with an API token. Generate one, store it as a secret in your CI system, and pass it on each request. The token carries your permissions, so it can only reach workspaces you can. See Run OttoTester from your CI for the full snippet-builder walkthrough.

Auto-file defects

OttoTester can open a defect in your issue tracker when a test fails — so a regression caught by a scheduled or pipeline run lands in your team’s workflow without anyone copying it across by hand.

Connect an issue tracker — GitHub, GitLab, Bitbucket, or Jira — to an application. Once it’s connected, a failing test can file an issue automatically, with the details of what broke.

Related

• Workflow → Run OttoTester from your CI, Schedule recurring runs, Read a run report
• Reference → API tokens
• Concept → Test suites, Variants, Applications