What your copy sends us

If you run your own copy of OttoTester, it sends us anonymous scores after each test run and each set of proposed tests. This page lists exactly what is sent, what is never sent, and how to check or stop it.

If you use OttoTester at ottotester.ai, this page does not apply to you. Our privacy policy covers that service.

Why we ask

OttoTester’s AI agents plan, write and fix tests, and each one runs on a model you choose. Your copy grades every agent’s work. Pooled across many copies, those grades show which models do each job well. That’s more than any single copy can learn by itself, and it’s how we keep model recommendations honest for all community members.

What is sent

After a run, your copy sends one short record for each agent that worked on it: the planner, the writer and the fixer. A run with many tests may split one agent’s scores across a few records. A record holds:

Which agent it was, and the model it ran on and the model that graded its work. Each model name comes with a yes or no saying whether it means “whatever version is latest”. The grader also carries the version number of its grading instructions.
Scores. For each test the agent wrote or fixed, or once for the whole plan from the planner: a grade from 0 to 100 and the parts behind it (0 to 4 each), listed below.
Mistake counts. How many times the agent made specific known mistakes, listed below.
Results. How many tests passed, failed, were written and were run.
Effort. Tokens used, cost in cents, time taken and how many steps it took.
A rough description of your app. Which families of web framework it uses (React, Angular, Vue, Svelte, Next.js, jQuery or other), whether it runs mostly in the browser, how people sign in (none, a form, single sign-on or other), and a range for how many tests it has, such as “10 to 50.”
When the run finished, and the run’s id. The id is a random code that your copy also uses for that run. It means nothing to us. It groups one run’s records.
Record details. A random id for the record itself, so a retry is counted once; whether this record carries the run’s totals or only more scores; and the version number of the record format.

Scores and mistakes, by agent

Agent	Parts of the grade	Mistakes counted
Planner	Whether the plan covers its goal; how good its checks are; whether each step can really be carried out	Tests with no checks; result steps with no check; checks that can never fail; two tests with the same name; a missing sign-in step for a role
Writer	Whether the test checks the right behavior; how strong its checks are; how well it avoids random failures; how it finds things on the page; how easy it is to maintain	Empty checks; web-address checks loose enough to pass on the wrong page; checks that skip themselves without failing
Fixer	Whether the fix keeps the test’s purpose; whether the fix is right; whether it keeps the original checks; how small the change is	Adding any of the writer’s three mistakes; removing or skipping a test; how many checks the fix added or took away, which is below zero when it took some away

After proposed tests are graded

When OttoTester proposes tests from a document or an issue, it suggests them in two steps: a first draft, then a cleaned-up version. Your copy grades both. It then sends one record for each step, holding:

Which step it was, and the model it ran on and the model that graded it, as above.
Scores. For each proposed test: its kind (a normal case, a failure case, an edge case or other), a grade from 0 to 100, and its two parts (0 to 4 each): how faithfully it follows the document, and whether it can become a test that runs. On the first draft, whether the test made it into the cleaned-up version.
Effort. Tokens used and cost in cents, or blank when your copy has no price for that model.
When the tests were proposed, and the proposal’s id. The id is a random code your copy also uses. It groups the two records.
Record details, as above.

The tests’ names and the document itself are never sent.

What is never sent

Your prompts, page content, test steps or test code.
Test names, app names, web addresses or file paths.
Your organization, workspace or app ids, or anything else that names you.
An account, a password or a key. Sending needs no account with us.
Anything that identifies your machine.

The graders’ written notes about your app stay on your copy. Only the numbers are sent.

See exactly what left

Each record we accept is written in full to a log. If you installed with the one-line install command, it is data/telemetry/sent.log in your install folder, which is ~/ottotester unless you chose another. The installer shows the exact path when it asks about sharing. Otherwise it is telemetry/sent.log in the folder set by OTTOTESTER_STATE_DIR, or in the folder OttoTester runs from if that isn’t set. Open it at any time to read what was sent.

The log keeps about 8 MB. When it fills up, it is renamed to sent.log.1 and a new one starts, so the oldest records are eventually dropped. If the folder can’t be written to, records are still sent and a warning appears in the server log.

To see what would be sent without sending anything, set:

TELEMETRY_DRY_RUN=true

Each record is then printed to the server log and saved to the same file, and nothing is sent.

Turn it off

Sending is on unless you turn it off. The installer asks you once, at the end of setup. To change your answer later, set this in your .env file and restart:

TELEMETRY_ENABLED=false

If the value is misspelled, or a similar name like DISABLE_TELEMETRY is used, your copy treats it as off. A typo will never leave sending on after you meant to stop it.

With sending off, your copy also stops asking us which models to recommend. The Assignments page still marks a recommended model for each agent, using the picks that came with your version of OttoTester. Picks based on community results need sending on.

To keep the records yourself, point them at your own server instead. Both kinds of record go to that one address:

TELEMETRY_URL=https://your-server.example.com/records

Who receives it

Records go to OttoCloud, run by Business Performance Systems, LLC, at cloud.ottotester.ai.

Kept indefinitely. Records are pooled with those from every other copy to improve model recommendations for everyone in the community. The longer the history, the better those recommendations get.
IP addresses are not logged. Your copy’s address is used only to stop any one sender flooding the service. It is held in memory for about a minute and never written down.
Never sold. We do not sell these records or share them with anyone to sell.