Skip to content

Measurement ​

Ryoma measures a project's work, not people. Read this page to learn where the numbers come from, how to read them, and how to use them to improve the process without turning them into a scoring tool.

Don't use metrics to evaluate individuals

Metrics are computed per project and per flow of work. Per-person metrics exist only for cost quotas, and only that person and administrators can see them. Don't rank people against each other, or projects against each other.

Four principles for reading metrics ​

Compare a project only with itself over time. Each project has its own characteristics, so putting two projects side by side tells you nothing.

Always read speed together with quality. Put flow time next to the first-pass acceptance rate and the number of returns, because work that is fast but keeps getting returned isn't fast.

The effect of AI is correlation, not causation. When a comparison group has fewer than 10 tasks, Ryoma draws no conclusion.

Ryoma also states data coverage clearly: how many tasks have a work-type label, and how much AI cost can be attached to tasks. When coverage is low, the report says so directly. For measurement frameworks, the platform uses only two: the Flow Framework and the AI measurement framework. It doesn't use DORA metrics and doesn't run experience surveys.

Where the metrics come from ​

Ryoma reads tasks from issues that follow the Bandum workflow, that is, the GH-<number> tasks. The work type comes from the label type: feature, type: bug, type: debt, or type: risk, and a task with no label counts as unknown. The time in each status comes from the history of the status: … labels, together with when the issue was closed or reopened. AI cost and usage come from Claude Code telemetry through Phillip, and are attached to a task through the issue ID in the branch name.

The telemetry data flow. Claude Code sends OTLP through Phillip to the Collector. The Collector attaches an identity from the token, removes sensitive fields such as email, then keeps only the fields on the allowlist. The filtered data is written to ClickHouse, which internal Grafana and the API then read. Source code, prompt content, replies and tool output never leave your machine.Claude Codevia Phillip,OTLP/HTTPCollectorAttach identityfrom the verified tokenRemove sensitive fieldsemail, account ID, bodyAllowlistonly fields Ryoma readsClickHousecounters andmetadataGrafanainternal,row isolationAPI Ryomareads metricsby permissionNever leaves your machineYour source codePrompt content and repliesTool outputDropped by the CollectorEmail and account IDTool parameters and log contentAny field not on the allowlist

Swipe sideways to see the whole diagram.

The Collector runs these steps in order and rejects the whole batch if one step fails, so unfiltered data never reaches ClickHouse.

So for accurate metrics, add every work-type label and name branches with the task ID, for example feature/GH-12-login. Bandum statuses map to the flow as follows.

StatusIn the flow
TodoNot started, not counted in flow time
In progressActive: a dev or agent is working
ReviewWait: waiting for acceptance testing
RejectedWait: returned, waiting to be picked up for fixes
DoneDone

Flow Framework metrics ​

Flow velocity is the number of items done in a period, counted per week and by work type. Flow time is the time from the first entry into In progress to Done. Ryoma shows the median and p85 rather than the mean, because this distribution is skewed. Flow efficiency is Active time divided by the total of Active and Wait, computed over finished items. Flow load is the number of items in Active or Wait, averaged per day over the period. Flow distribution shows the share of each work type among the items done in the period.

Alongside these are the first-pass acceptance rate, the number of returns, test waiting time, and the number of risk tasks.

Here's a quick way to read them. Low flow efficiency means tasks wait more than they are worked on, and the bottleneck is usually in testing or review, rarely in coding. Rising flow load with flat velocity means the team has too much work in progress at once. Flow distribution shows how much effort the team spends on features, and how much on bug fixes and technical debt.

The AI measurement framework ​

The framework has three groups.

The Utilization group shows how much AI is used: the number of AI users each week (distinct people who have AI sessions in the project's repo), the share of done items with at least one session attached to a task ID, the share of repos that use a harness, and cost coverage, which is the part of AI cost that can be attached to tasks out of the repo's total AI cost.

The Impact group compares items with AI against items without AI, for the same work type and the same S/M/L size when the sample is large enough, on flow time, flow efficiency, first-pass rate, and number of returns. Ryoma also plots the project's own monthly trend.

The Cost group computes cost per done item by work type, and rework cost, which is the AI cost of sessions after the first return.

AI cost is only an estimate at converted API prices. It isn't the amount actually paid under a subscription plan. Read it as a relative measure.

Where to see the metrics ​

WhereWho sees itContent
Tara Portal, ProjectsEvery role of the organization (PO, PM, BA included)Org-wide numbers in the project list, and Flow, AI, tasks and at-risk tasks on each project page, over a time range you choose. Guardrails counts are under Settings → Guardrails policy. See Metrics on the Projects pages
Tara Portal, ProjectsEvery role of the organizationOne project's page with its Flow and AI sections
Weekly digestPMVelocity, flow time, WIP, and rework cost compared with the previous week
ph ryomaYouYour AI usage against your quota
ph insightsEvery roleThe same numbers as the Projects pages, in the terminal

For details about who can see what, see Who can see what.

How quotas work ​

Quotas are computed on converted cost, per week and per month, for each person and each project. Ryoma warns at 80% and 100%, but doesn't block work. Quotas are the only reason per-person metrics exist.

Next steps ​