Testing and acceptance
Devs, Testers, and BAs accept a task in one of two ways: run it automatically with ph bandum verify, or mark it manually with ph bandum test. This page covers both, and what to do when a task is returned. The task must be in the status: review state. See The life of a task.
Swipe sideways to see the whole diagram.
Where are the test cases?
Each task has a testcases.md file that lists the test cases (TCs). Each TC links to at least one acceptance criterion (AC), and each AC must have at least one TC. The file lives in the PR branch, depending on the task level.
| Task level | Directory in the PR branch |
|---|---|
| S, M | docs/acceptance/GH-<number>/ |
| L | docs/features/GH-<number>-<name>/ |
The directory holds testcases.md, acceptance.md (the results of each test run), and evidence/ (screenshots). You only append to the end of acceptance.md and never edit earlier runs. If anyone edits them, ph bandum check reports acceptance_edited. The full format is in Bandum format.
Automated testing
The project declares how to run in .bandum.json at the repo root, with setup, run, test, and report. The verify command checks out the PR branch into a worktree, runs those steps, reads the JUnit results by TC id, and writes them to acceptance.md:
ph bandum verify GH-12Phillip matches tests to TCs by name. The test name (or classname) must contain the TC id as a standalone token, for example TC-1 (AC-1.1) log in with correct password. A TC with no test, or whose test is skipped, counts as failed. The command exits with code 0 on pass and 2 when the task is returned, so it works in CI. Add --quiet for less log output.
Manual testing
You mark manually when a TC needs a person to look at the result. There are three ways.
The first is the Bandum web app. Open the task page and go to the Manual testing section. Mark each TC Pass or Fail. A Fail needs a description of the bug. Then click Approve if every TC passes, or Send back. The detailed steps for BAs are in Get started for BAs, POs, and PMs. If the project declares stagingUrl, the task page also has an Open staging button.
The second is to let the test command open the web app for you. The command checks out the PR branch, runs the app, and opens the app and the task page so you can mark:
ph bandum test GH-12The third is to work in the terminal, with no web app:
ph bandum test GH-12 --cli # asks about each TC
ph bandum test GH-12 --pass TC-1,TC-2 --fail "TC-3=Account is not locked" --note "Tried on Chrome"With the last two ways, Phillip writes acceptance.md, commits, pushes to the PR branch, reviews the PR, and sets the status: approved or status: rejected label. Add --evidence <file> to attach a screenshot. The result always carries the GitHub or GitLab account name of the tester, and the person who opened the PR can't approve their own PR.
What to do when a task is returned
A task is returned when at least one TC fails. The dev then handles it in three steps.
Read the comments from the latest test run:
bashph bandum pull GH-12Have the agent fix it according to the comments, test cases, and ACs:
bashph bandum fix GH-12To pass extra arguments to Claude Code, put them after
--. Seeph bandum.Run the tests again, then move the task back:
bashph bandum verify GH-12 ph bandum review GH-12
Bandum and Ryoma aggregate the first-pass approval rate and the number of returns by project. These are flow metrics. Don't use them to evaluate individuals.
Where to look when something goes wrong
| You see | What to do |
|---|---|
| The Approve or Send back button is disabled | A TC is still unmarked, or a TC failed without a bug description. Approve is enabled only when every TC passes |
| "The PR branch just changed" (new commit, or someone else recorded a result) | Click Reload and test the new version |
The repo has no .bandum.json | The project hasn't declared how to run the app. Use --cli or ask the dev |
| The task has no test cases | Ask the dev to add testcases.md, then move the task back to testing |