Skip to main content
Version: 0.1.147

Test runs and reports

The scenarios you agreed are also the tests. A BDD run executes them against the running application and reports, scenario by scenario, what passed.

Where runs come from​

A run is submitted from wherever the project's test runner wrote it, rather than from one fixed path the platform insists on. Your project's runner lives where it lives — at the repository root, one level down, or in a shared test folder for a project combining several repositories — and the platform finds it. A runner may also change into a test folder of its own and run Cucumber there with that folder's config. The platform then reads the suite from that folder too, so a scenario the config leaves out, such as one tagged to be skipped, is not expected in a full run.

For a codebase you provided, Repave can write that runner for you. In Project settings, the General card has Generate BDD Config: it starts an agent that writes or repairs the project's BDD test runner, at the path in BDD Test Script, and the support files it needs, then proves it by running the suite once before committing them. A project Repave scaffolded already has its runner, so the control is not shown there. In Repave IDE, /repave:bdd-config does the same from your own session. /repave:bdd-parallel then makes that runner run several scenarios at once, so a daily test run can use parallel workers.

Runs can also be produced outside the platform. Test evidence generated locally can be pushed and replayed, which is the usual route for a developer who ran the suite on their own machine.

An uploaded run updates each scenario's test status, so the specification itself carries the result. A scenario is not just described in the product; it is green or red in it.

Daily tests​

A feature's own scenarios run when it is implemented. Daily tests runs the whole suite, every feature at once, so a change one feature makes that breaks another is found the next morning rather than whenever someone next looks.

Turn it on in Project settings, on the Implementation tab. The suite then runs once a day at the time you set. Run now, on the Daily tests page in the project's sidebar, runs it at any time. A run can run several scenarios at once and has its own memory limit; both are in the same settings.

Before the suite starts, a run brings the project's branch up to date with your repository, so it tests what is merged there, including changes merged outside Repave. If the branch cannot be brought up to date, because it has diverged from the repository, the repository cannot be reached, or uncommitted changes are in the way, the run is Could not run and says why, rather than testing old code.

A run only tests. When scenarios fail, an agent fixes them if the project fixes failures automatically (off by default, in the same settings card); otherwise the run says Failed, and Fix failures on the run's page starts the same agent when you choose. A suite that could not start is fixed the same way: when the project's test runner fails before any scenario runs (an app that crashes on startup, a setup hook that fails, an install the project's code broke), the run is Failed, its page says The suite could not start. with the error, and View runner log opens the runner's whole log. The agent reads that log, fixes the cause, runs the suite, and fixes what still fails. Since the run has no results of its own, the fix is checked against the last run whose suite did start: a scenario that passed there must still pass. Fix failures is offered on the latest run only, since an older run's failures were found on older code, and it can also start the fix again after a fix that failed.

The page lists every run, newest first, with how many scenarios passed and what happened next:

  • Passed — every scenario passed.
  • After fix, in the Trigger column — the suite a fix ran to check itself, listed as a run of its own. It names the run it fixed, which links back to it, and it starts no fix: anything it shows still failing is on the run it fixed, with the agent's reason.
  • Failed — scenarios failed and no fix has been started. Open the run and press Fix failures to start one.
  • Fixing…, then Fixed — scenarios failed, and an agent fixed them. The fix is merged into the project's branch and pushed, the way features merge in your project.
  • Fix in review — in a project that merges through pull requests, the fix is a pull request from the fix's own branch, waiting for your review, with a title and description an agent wrote from the change. A later run whose failures that pull request already fixes says Waiting on PR and starts no new agent. A later run with new failures starts a new fix, whose pull request says which earlier one it replaces; close the earlier one once the new one merges.
  • Fix failed — the suite is still red, and you are notified as for any failed job. Either the agent could not make the suite pass in 5 attempts and nothing was merged, or it fixed what it could, merged that, and left failing a scenario it believes is out of date. The run shows what was fixed, and what still fails with the agent's reason for each. The next run tries again if the project fixes failures automatically; Fix failures on the run tries again now.
  • Skipped — a scheduled run did not start, and the run says why: the previous run was still going, there is no BDD test script, or the modernized repository does not exist yet.
  • Could not run — the suite did not start, and the run shows the runner's error; or it ran but its runner left its report empty, and the run says so and names the runner's log; or it ran out of memory, and the run says which limit it hit and the setting that raises it, for example Out of memory: the suite container hit its 16 GiB limit. Raise Daily test run memory in Project Settings. A run cut off by memory decided nothing, so it is not Failed and no fix starts, even in a project that fixes failures automatically.

When the fix agent believes a scenario, not the code, is out of date, it asks you, as any agent asks a question: the question appears in your notifications and in the agent's conversation, with its evidence, and you choose Fix the code (the scenario is right) or Leave it failing (you will change the scenario). While the question waits, the run stays Fixing…, and a scheduled run in the meantime is Skipped. A project that does not let agents ask (Agents can ask you and hear from you, in Project settings) gets no question: the agent leaves the scenario failing and says why.

Open a run to see each scenario's result. A fixed run shows its own results, and View the run after the fix opens the run the fix checked itself with. Scenarios that share a name in one feature file, such as the examples of an outline that differ only in an unquoted value, are listed apart and numbered by their place in the file — Search orders by amount #1, #2, #3 — so each keeps its own result.

Download PDF, at the top of a run's page, saves the run as a test report you can file with a change request or hand to someone without a Repave account. It holds a summary (commit, branch, when it ran, the result and each feature's counts) and every feature, scenario and step with its status. Under a failing step it prints the error, and any data table, doc string or screenshot the step had. For a run that was fixed, the report shows only the result after the fix, and its commit is the one the fix was merged as. The first run and what the fix changed stay on the run's page. The report is laid out in your organization's Test report template, if an administrator has uploaded one (see Documents), and its pages are numbered. Anyone who can open the run can download it. The report is made when you press the button, so it always uses the current template. It is offered once a run has finished, and not for a run that did not start.

To send the report outside your organization, choose Download with a password… from the menu beside Download PDF and type a password. The file then asks for it before it shows a page, in any PDF reader; once open, it can be printed and copied as usual. Repave does not keep the password, so note it down and give it to the recipient separately. Use up to 32 letters, digits or symbols (fewer for accented or non-Latin characters), and no full-width letters or digits: some PDF readers could not open the file with such a password, so Repave will not make one.

The fix agent may change application code and step definitions. It never changes a .feature file: the scenarios are the specification. A fix is only merged when no scenario that passed in the run fails after it, whatever the agent reports. Only one run goes at a time, its fix included, so Run now is disabled while one is going. Runs are kept for 90 days.

The fix's branch​

Every fix works on a branch of its own, named work/daily-tests-fix- and a few characters of the run, in a worktree that is also listed in Repave IDE. The Fix card on the run's page shows it with the same actions Repave IDE offers: Open in Repave IDE, Push, Create PR (or Merge to the base branch, in a project that merges directly), and Pull under ⋯. Create PR has an agent write the pull request's title and description.

A fix that does not land keeps what it changed on its branch, so you can push it, open a pull request from it, or carry on in Repave IDE. While the fix is still working, the actions are disabled and say so. A fix that landed, or a branch someone deleted, is named on the card in plain text.

When the next fix starts, an earlier fix's branch that nobody pushed, opened a pull request from or committed to is removed. A branch you used stays until it merges or you delete it.

Reading a report​

Reports collect what a run found; the Report Catalog is the inventory across runs.

A failing scenario is worth reading in this order:

  1. The scenario — is the expectation right? A scenario can fail because the implementation is wrong, or because the scenario describes something nobody agreed to.
  2. The trace — what the browser actually did, step by step, with the page as it was rendered at each step. UI implementation is verified against rendered trace frames, so the frames are the evidence rather than a summary of it.
  3. The code, last.

Traces are viewable in Repave IDE too, under the Repave sidebar's BDD Runs view.

Repave keeps each scenario's latest trace, plus any trace a bug report was filed from or recorded as its fix. When a run records a newer trace for a scenario, the older one is deleted. If the newest trace is unreadable, the last readable one is kept as well. Resetting a feature deletes its traces, except the ones bug reports point to.

Test Replay​

Test Replay re-runs a scenario so you can watch it. It stays scoped to the Gherkin view you are in, so you are replaying the scenarios in front of you rather than the whole suite.

While replaying, you can pin problems directly to what you see, and report several pinned problems from one replay as one submission — see Bug reports.

Unit tests​

For a provided modernized codebase, unit tests run the way your project already runs them, using the configuration detected when the project was set up. Detection reports why it will not run rather than silently skipping. Whether unit tests, coverage thresholds and a false-positive review gate an implementation is a project setting — see Project settings.

The coverage threshold is checked twice once the unit tests pass: on the whole project, and on the lines the feature changed — every line it adds or changes against the branch it started from, wherever a coverage report lists it. Comments, blank lines, tests and files your coverage configuration leaves out do not count. If the changed lines fall short, the agent is sent back with the files and line ranges no test ran, and writes tests for them. A change with fewer than 20 coverable lines is judged on the whole project only.

When the agent's next try changes nothing and the check still fails, Repave stops and shows you the failure and the agent's last reply instead of trying again. A run that cannot finish, for example because the test container runs out of memory, fails with the reason; the agent is not asked to fix it.

A branch that builds on unfinished work is not checked during Write tests or implementation: the whole suite would also judge the code it builds on, which its agent does not own. Its agent runs its own tests and is told to leave that code alone. The job log says "Unit tests not checked by Repave yet: this branch builds on Orders, which has not landed. They are checked before this branch lands." Before the branch lands, once that work has merged, Create PR, Merge and module delivery run the same check, even when the project skips the full test suite before work lands. If it fails, the agent fixes it there, changing tests only, with up to 5 tries; otherwise nothing lands and the reason is shown.

Surfaces your operator may have enabled​

Off unless your installation turns them on:

  • Cucumber Test Runner and Unit Test Runner — running suites from the browser.
  • UAT Tools — user-acceptance environments per feature, with test data committed as seed files, a database baseline restored from a server-side backup, and several environments able to run at once. UAT sessions support the same pinned problem reporting as Test Replay. A feature can also be run in development mode and verified from the feature's page — see Seeing the implementation.
  • Characterization Tests — capturing existing legacy behaviour as tests.

Next: Bug reports.