Skip to content
OttoTester docs

Test runs and reports

What a run is

Section titled “What a run is”

A test run is one execution against a single variant. Three things can start one:

  • The in-app runner — you pick a Test Authoring Template and run it.
  • A scheduled run — a suite set to run on a cadence.
  • CI/CD — an external pipeline calling POST /api/v1/runs.

However it starts, a run produces a report.

Run lifecycle — the phases

Section titled “Run lifecycle — the phases”

A run moves through phases as the agent pipeline works:

  1. Planning — the planner explores and picks test cases.
  2. Generating — the generator writes Playwright scripts.
  3. Executing — the scripts run.
  4. Healing — failed scripts get a repair pass.
  5. Analyzing — results are scored and summarized.

Which phases actually run depends on the template — a resume run skips planning, for example. Healing is a phase; it isn’t a per-test status.

Run status vs test status

Section titled “Run status vs test status”

Two different things are both called “status”:

  • The run is pending, running, completed, failed, or cancelled. That’s the run as a whole.
  • Each test is passed, failed, skipped, running, or pending. That’s one test inside the run.

A run can be completed while individual tests failed — “completed” means the run finished, not that everything passed.

The report

Section titled “The report”

The report is what you read after a run. For each test it shows the status, the script that ran, and a link to its Playwright HTML report — Playwright’s own detailed record, with each step, screenshots, network activity, and a trace you can step through when a test fails. Open it for the specific test you’re investigating.

Dashboards

Section titled “Dashboards”

The variant dashboard rolls up runs into trends. Two views answer specific questions:

Newly failing

Section titled “Newly failing”

The newly failing view answers “what just broke?” — tests that were passing in earlier runs and started failing in the latest one. It cuts through a noisy report: if nothing newly failed, the regression you’re worried about isn’t here.

Compare variants

Section titled “Compare variants”

Pick two variants of the same application and the compare view shows their latest runs side by side — pass rates, common failures, tests one variant has and the other doesn’t. The usual use case: “did this change break staging in a way it doesn’t break dev?”

Section titled “Related”