Skip to content

Testing and acceptance ​

Devs, Testers, and BAs accept a task in one of two ways: run it automatically with ph bandum verify, or mark it manually with ph bandum test. This page covers both, and what to do when a task is returned. The task must be in the status: review state. See The life of a task.

Two acceptance-testing paths converge on one decision. The automated path runs ph bandum verify and reads JUnit results by testcase code. The manual path runs ph bandum test, then you mark each testcase Pass or Fail (Đạt or Không đạt) on the web or with a CLI flag. Both write results to acceptance.md, then the task is approved (Duyệt) or sent back (Trả lại).AutomatedManualAuto checkph bandum verifyRead JUnit resultsmatch tests by TC codeManual testingph bandum testMark each TCPass or Fail,on web or with --cliRecord resultsacceptance.md,commit and pushApprovestatus: approvedSend backstatus: rejected

Swipe sideways to see the whole diagram.

Both paths write to acceptance.md under the tester's account name. The person who opened the PR does not approve their own PR.

Where are the test cases? ​

Each task has a testcases.md file that lists the test cases (TCs). Each TC links to at least one acceptance criterion (AC), and each AC must have at least one TC. The file lives in the PR branch, depending on the task level.

Task levelDirectory in the PR branch
S, Mdocs/acceptance/GH-<number>/
Ldocs/features/GH-<number>-<name>/

The directory holds testcases.md, acceptance.md (the results of each test run), and evidence/ (screenshots). You only append to the end of acceptance.md and never edit earlier runs. If anyone edits them, ph bandum check reports acceptance_edited. The full format is in Bandum format.

Automated testing ​

The project declares how to run in .bandum.json at the repo root, with setup, run, test, and report. The verify command checks out the PR branch into a worktree, runs those steps, reads the JUnit results by TC id, and writes them to acceptance.md:

bash
ph bandum verify GH-12

Phillip matches tests to TCs by name. The test name (or classname) must contain the TC id as a standalone token, for example TC-1 (AC-1.1) log in with correct password. A TC with no test, or whose test is skipped, counts as failed. The command exits with code 0 on pass and 2 when the task is returned, so it works in CI. Add --quiet for less log output.

Manual testing ​

You mark manually when a TC needs a person to look at the result. There are three ways.

The first is the Bandum web app. Open the task page and go to the Manual testing section. Mark each TC Pass or Fail. A Fail needs a description of the bug. Then click Approve if every TC passes, or Send back. The detailed steps for BAs are in Get started for BAs, POs, and PMs. If the project declares stagingUrl, the task page also has an Open staging button.

The second is to let the test command open the web app for you. The command checks out the PR branch, runs the app, and opens the app and the task page so you can mark:

bash
ph bandum test GH-12

The third is to work in the terminal, with no web app:

bash
ph bandum test GH-12 --cli                                   # asks about each TC
ph bandum test GH-12 --pass TC-1,TC-2 --fail "TC-3=Account is not locked" --note "Tried on Chrome"

With the last two ways, Phillip writes acceptance.md, commits, pushes to the PR branch, reviews the PR, and sets the status: approved or status: rejected label. Add --evidence <file> to attach a screenshot. The result always carries the GitHub or GitLab account name of the tester, and the person who opened the PR can't approve their own PR.

The task page keeps every test run, so you can see right away which test case failed last time.

What to do when a task is returned ​

A task is returned when at least one TC fails. The dev then handles it in three steps.

  1. Read the comments from the latest test run:

    bash
    ph bandum pull GH-12
  2. Have the agent fix it according to the comments, test cases, and ACs:

    bash
    ph bandum fix GH-12

    To pass extra arguments to Claude Code, put them after --. See ph bandum.

  3. Run the tests again, then move the task back:

    bash
    ph bandum verify GH-12
    ph bandum review GH-12

Bandum and Ryoma aggregate the first-pass approval rate and the number of returns by project. These are flow metrics. Don't use them to evaluate individuals.

Where to look when something goes wrong ​

You seeWhat to do
The Approve or Send back button is disabledA TC is still unmarked, or a TC failed without a bug description. Approve is enabled only when every TC passes
"The PR branch just changed" (new commit, or someone else recorded a result)Click Reload and test the new version
The repo has no .bandum.jsonThe project hasn't declared how to run the app. Use --cli or ask the dev
The task has no test casesAsk the dev to add testcases.md, then move the task back to testing

Read more ​