Release notes

Every OttoTester release, newest first. Each one starts with a sentence on what it gives you, then lists what is new and what was fixed.

If you use OttoTester at ottotester.ai, you are always on the newest release. If you run your own copy, ./ottotester update in your install folder moves it to the newest release, and ./ottotester update v1.2.3 moves it to a particular one.

v1.1.11 (October 29, 2026)

Setup now picks a suggested model for each AI agent, and warns you when your OttoCloud email still needs confirming.

New

Setup sets each AI agent to the model suggested for it when your key can use that model, instead of one model for every agent. You can change them later in Global Admin, AI Models, Assignments.
A Release notes page lists what changed in each release.

Fixed

Setup warns you when your OttoCloud email isn’t confirmed yet, and lets you confirm it before going on.
A document that fails to read because OttoCloud refused to run AI now says why, for example a spending limit or an unconfirmed email.

v1.1.10 (October 28, 2026)

A new install opens straight on Create account, and setup’s question about sharing scores is clearer.

New

A new install opens on Create account instead of Sign in, so the first person can set it up right away.
Setup’s question about sharing anonymous scores now says more plainly what is never sent.

v1.1.9 (October 27, 2026)

Self-hosted installs start faster and more reliably, because the built-in embedding model now comes inside the release.

New

The built-in embedding model ships inside the release, so a new install no longer downloads it on first start.
Setup can be pointed at a different OttoCloud address, for trying an install without touching your real account.

Fixed

The sign-in and sign-up pages show every sign-in provider’s logo again.
Setup sends OttoCloud users to the right pages for their key and their billing.
The built-in embedding service recovers on its own after an interrupted download.
A fresh install no longer points at an old OttoCloud address.

v1.1.8 (October 26, 2026)

Setup asks about sharing anonymous scores last, once your provider and model are set.

New

The question about sharing anonymous scores moved to the end of setup.

v1.1.7 (October 26, 2026)

Install and update OttoTester with one command, import requirements and defects from GitHub Issues, and see coverage numbers that agree everywhere.

New

Install from the published release with one command, and update with ./ottotester update. If the new version doesn’t start, the update puts the previous one back.
./ottotester demo starts a demo app to try OttoTester on.
An older version refuses to start on data a newer version has already updated, so going back a version can’t damage your data.
Connect GitHub Issues as an issue tracker. Import issues as requirements or defects by their labels, narrow them with a visual filter, and open an issue for a failed test.
The Assignments page recommends a model for each agent, and the model catalog offers only models your install can actually use.
Mark each document you upload as requirements, defects or reference. Each part of a requirements or defects document counts toward coverage, and a person confirms the requirements OttoTester finds before they count.
After OttoTester reads a document, a popup shows what it found.
Coverage uses the same words and the same numbers on every screen, and the coverage trend ends with a live point for today.
The live run card shows which test is running.
An install can run without an email server. Invitations, password resets and notifications still work.
Setup explains the anonymous scores your install shares, and how to turn sharing off.
The Tests page shows why a test was fixed.
New run templates pause between steps and between tests unless you change them.

Fixed

The first account on a new install becomes its global admin.
A failed planning call now fails the run and says why, instead of ending it as completed.
Claude models through OpenRouter work with thinking turned on.
A fresh install has prices for newer models from the start, and setup suggests a model your key can run.
Export Tests downloads only the selected variant.
A Jira filter can no longer reach projects outside the ones the app chose.
A test is no longer saved as finished when its steps were never seen working, and editing a test’s steps replaces its old script.
Reading a document again keeps your tests connected to the proposed tests they came from.
One unreadable reply from a model no longer throws away the whole run.
Proposing tests from a document works with OpenAI models.
Two people who sign up in the same second with the same name get separate organizations.
The browser tab icon is easy to see on light and dark tab bars.

v1.1.6 (September 29, 2026)

Running your own copy gets much easier: one command to a working install, a built-in embedding model, OttoCloud as a model supplier, and storage and email beyond AWS.

New

One install command that ends on a working install, with step-by-step setup in the terminal.
A demo app ships with OttoTester, so your first run can reach a passing test.
A built-in embedding model is the default, so it needs no key of its own.
OttoCloud can supply your models. OttoTester checks your account may spend before it sends a request.
Run reports can be stored on any S3-compatible storage, not only AWS S3.
Email can go through any mail server, not only AWS SES.
Run a suite now or on a schedule from one Run Suite dialog.
Read a run’s results with a workspace API token, and copy a test plan between variants through the API.
The deduper, which spots duplicate tests, can have its own model on the Assignments page.
Advanced admin settings carry a caution note and a one-click reset to the recommended values.
OpenRouter costs are recorded as OpenRouter charged them, not estimated.

Fixed

The embedding service no longer restarts over and over on machines without a graphics card.
Adding an app works for sites that build their pages in the browser.
Filter labels in the Tests sidebar are no longer cut off.
Setup checks your Node version and offers to set up pnpm for you.

v1.1.5 (August 26, 2026)

The Tests page is reorganized around how you work, proposed tests get sharper, and you can sign in with your own Microsoft Entra.

New

Sign in to OttoTester with your organization’s own Microsoft Entra.
The Tests page groups its actions by what they act on. Run just the test you’re looking at, pick specific tests, or run and fix a failing test, all from one place.
Start a run from the Runs page and watch it there. The current run’s results show in place, with past runs folded away.
Run settings are kept in three separate lists: manual runs, scheduled runs and CI runs.
Proposed tests appear as soon as they are written and are graded after. Each document chooses which kinds of tests it suggests, and steps a browser can’t check are left out.
Edit a proposed test’s priority and type right on its card.
See when a test was proposed, created and last changed, and what it cost across every stage.
Download your tests as readable Markdown files in a zip.
Grant access by picking a person by name or email.
Models from countries you exclude no longer appear in the model catalog at all.

Fixed

A test that passes is no longer left marked as blocked.
A sign-in helper can no longer report success when nobody is signed in, and one that times out names its role and time limit.
A space pasted into a credential is caught when you enter it.
Create Tests makes as many tests as it says it will.
Documents added from the Proposed tab update on their own.
Running a test from the Tests menu shows its result.
Workspace and team members show names, not ids.
Version history shows changes from before a test’s file was renamed.
Quality reports keep what was actually spent, instead of repricing old runs at today’s rates.

v1.1.4 (August 8, 2026)

Downloaded tests and their history stay complete, even after a failed run or a server replacement.

Fixed

Each app’s test history survives the server being replaced.
Downloads and history stay complete after a failed run.

v1.1.3 (August 8, 2026)

Download an app’s tests as a runnable Playwright project, and get help from a new support page.

New

Download an application’s tests as a zip that runs as a Playwright project.
A new support page, with a FAQ in the docs.
Runs started from CI add a readable summary to the pull request.

v1.1.2 (August 5, 2026)

The trial countdown banner goes away when it should.

Fixed

The trial countdown banner no longer stays on screen for good.

v1.1.1 (August 5, 2026)

Keep out models from countries you exclude, compare models by cost band, and sign-up records your plan and consent properly.

New

Leave out models from vendors in countries you exclude, both in the catalog and when a run starts.
Cost bands are one click away, and Premium now means the priciest tenth of models.
Global admins can assign a plan to an organization.

Fixed

Sign-up records your plan and your acceptance of the terms, and refuses an invalid plan in the sign-up link.
No verification email is sent when email verification is turned off.
The Users page shows each person’s last sign-in and sign-in provider.
Models OpenRouter no longer lists, or can’t price, are dropped.
Long model replies are no longer cut short, and run details report only what was measured.

v1.1.0 (August 2, 2026)

Choosing a model gets easier with a cost and quality guide, and signing in to the app you test is far more reliable.

New

Pick a model with help: cost bands from Low to Premium, a “Help me choose” slider, and guidance on what each agent needs.
The model catalog can be filtered and sorted, shows when each model was added, and its rates are easier to read.
The Model Leaderboard plots quality against cost.
Each plan, script and fix records how long it took, not just what it cost.
Jira issues stay in sync, with a visual filter for which ones to import.
OttoTester records how it reached your app’s sign-in, step by step.

Fixed

OttoTester finds your app’s sign-in address on its own, follows redirects to it, handles Login.gov when it skips the one-time code, and picks the right role when a variant has several.
When sign-in fails, the message says plainly what to do.
A test the planner labels as signed out is no longer blocked for good.
Default test time limits are long enough for a real sign-in.
Every test written gets graded, so a run’s quality score covers all of it.
An AI call can no longer hang forever, and the default time limit per call is longer.
Model names are readable in the picker again, and the header fits narrow screens.

v1.0.9 (July 27, 2026)

A curated model picker you can filter, and runs that can reset the app you test before they start.

New

A curated model picker you can filter, with a badge for models that read images and a price you can override.
Model selection shows blended rates and a cost-to-quality slider for each agent.
A run can reset the app you test before it starts.

Fixed

A grader that replies without a score is asked again, instead of the score being lost.

v1.0.8 (July 24, 2026)

Import Jira stories as proposed tests, and see requirements coverage on the dashboard.

New

Connect Jira to a workspace, browse its issues, and import a story as proposed tests, attachments included.
Requirements coverage on the dashboard, with a list of gaps, and production defects tied to coverage.
Proposed tests now live on the Tests page, under Proposed.
Add a reference document by pasting a link to it.
Input variations: try the same steps with different kinds of input.
The public sites ask for cookie consent.

Fixed

After a new release, the app recovers on its own from an out-of-date page.
The Analytics model filter works on the quality tabs.
Proposed tests are no longer all thrown away when a document’s summary runs long.
The run log no longer shows raw cost figures.
Pages no longer jump around on phones.

v1.0.7 (July 15, 2026)

Clearer, more accurate wording on the website and the sign-in page.

v1.0.6 (July 14, 2026)

Global admins can set each plan’s AI credits and give an organization a one-time top-up.

New

On hosted OttoTester, global admins can set each plan’s AI credit allowance.
Global admins can give an organization a one-time credit top-up.

v1.0.5 (July 13, 2026)

New sign-ups get free AI credits to try OttoTester before paying.

New

On hosted OttoTester, new sign-ups get 1,500 free AI credits.

Fixed

The features page is simpler and easier to read.

v1.0.4 (July 13, 2026)

AI credits arrive on hosted OttoTester, and you can delete your own account.

New

Plans include an allowance of AI credits, with a warning when you run low. You can buy more, and bought credits last 12 months.
See AI use per person on the Org Usage tab.
Delete your own account. It is held for 30 days before it is removed.
A public features page.

Fixed

Run details show AI use in credits, not raw cost, and workspace members no longer see raw cost.
Unlimited plan limits read “Unlimited”.
Contact sales and Request a demo go somewhere useful.
The test writer no longer gives up when a test data value appears in what it clicks or checks.

v1.0.3 (July 7, 2026)

Sign up and pay on your own, keep test data in one place, see test coverage on the dashboard, and replay sign-ins instead of retyping them.

New

On hosted OttoTester, sign up on your own with a 14-day trial, manage billing, and change plan without starting over.
Test Data keeps one set per test, with columns that take a new value on each run, and a test can run against every row.
Test coverage on the variant dashboard, over time, with a coverage count on proposed tests.
Pick proposed tests when starting a generation run, with help writing the prompt.
OttoTester saves your app’s sign-in and replays it on each run, instead of filling in the form again.
A ready-made sign-in helper for Okta.
Console and network errors from your app can be captured and shown in the run, the PDF report and the run-completed email.
Analytics can filter runs by run type, call type and model, and opens a run’s details in a side panel.
Global admins can set models for each organization and each plan.
Duplicate tests are spotted by meaning, not only by name.
Run templates can slow down and pause between tests.
Global Admin has a side menu, and Models is now called AI Models.

Fixed

One-time sign-in codes are no longer reused, which caused sign-ins to fail now and then.
A working sign-in helper is no longer reset, which blocked every test that needed it.
A brief bad reply from a model no longer throws away the whole run.
A run that can’t load its tests now fails instead of reporting zero tests.
Replay runs no longer run zero tests.
Analytics is shown only to global admins with Global Admin mode on.

v1.0.2 (June 12, 2026)

OttoTester saves the sign-in address it finds, and long AI calls can be stopped.

New

After a successful sign-in, OttoTester saves the sign-in address it found.
The full OpenRouter model catalog is available.

Fixed

AI calls have a time limit, and stop when a run is cancelled.

v1.0.1 (June 11, 2026)

Sign-in is shared between planning and writing tests, scheduled jobs get their own screen, and CI can start runs.

New

Planning and test writing share one sign-in, and the Login.gov sign-in helper is chosen automatically.
A redesigned sign-in card for credentials, showing the sign-in address, with a way to stop it being used automatically.
Scheduled jobs have a catalog with details and settings, and global admins are told when a job fails.
Models get scheduled health checks and a Check now button.
Start runs from your CI pipeline through the runs API or the GitHub Action, set up from the Integrations tab.
Edit a variant’s trigger rules in a side panel.
Test Data has a catalog, a table editor and CSV upload.
Resend the verification email after signing up.

Fixed

Credentials are back, as a tab under Test Data.
Times follow your time zone setting.
Resume Incomplete Work skips tests that already pass.
The app picker switches to an app you just created.
New run templates start with sensible time limits.

v1.0.0 (June 5, 2026)

The first full release. Runs can start when GitHub reports a new deployment.

New

Start a run when GitHub reports a new deployment of your app.

Earlier releases

Releases before v1.0.0 were early previews and have no notes here.