Measure the delivery system

Begin with the decision you need to make

A team wants to know whether AI improves delivery. Counting generated lines answers a different question. Define the useful outcome and the quality conditions before selecting a metric.

For a fictional export service, the desired outcome is reliable delivery of accepted changes with less total effort. Record preparation, implementation, review, correction, and waiting. Include changes that failed or were abandoned.

Use one service with a clear boundary. Combining an experimental website and a critical payment service can produce a number that explains neither. Describe the context before comparing periods or teams.

Use current definitions

DORA’s current delivery model contains five metrics. Their scope is delivery performance, not the value of every feature or an individual’s contribution. DORA metric definitions.

A dashboard may use another definition. Read it before interpreting the result. Taiga’s current deployment documentation describes four reported metrics derived from provider deployment records. Its recovery measure uses a subsequent successful deployment. That is not a complete record of every production incident. Taiga definitions.

Inspect a fictional change sequence

Suppose a service makes twelve deployments in a month. Eight deliver planned changes. Four repair problems from earlier releases. The count is twelve, but the composition matters.

In the next month, the team makes ten deployments: nine planned changes and one repair. Fewer deployments can coexist with more useful work. These figures illustrate interpretation; they are not a performance benchmark.

Also inspect the distribution. One long review wait can disappear inside an average. A recovery measure from a single failure is weak evidence for future reliability. Report the number of observations and material exceptions.

Pair flow with consequences

Use service signals to check whether delivery changes affect users. A faster pipeline is not sufficient if exports fail more often. Use an appropriate SLO or another clearly defined outcome measure. SLO guidance.

Review effort and rework help explain the result. If AI shortens implementation but produces large diffs, review may become the constraint. If environments take days to obtain, faster coding may have little effect on elapsed delivery time.

Choose one improvement that addresses the observed constraint. For example, provide a supported test environment or reduce the size of a change. Define a quality balancing metric so the team can detect an apparent speed gain caused by weaker checks.

Keep measurement useful

Avoid individual rankings based on PR counts or generated code. These measures can reward splitting work artificially, avoiding difficult maintenance, or shifting review effort to colleagues.

Review the result with the people responsible for the complete service. Record what changed in the tool, work mix, team, and environment. Treat a before-and-after comparison as evidence with limitations, not automatic proof of causation.

The purpose is a better next decision. A small, trustworthy measurement that leads to a verified improvement is more useful than a large dashboard without agreed meaning.

Practice with ten changes

This separate fictional dataset records ten planned changes. All times are UTC on the date shown. An empty correction field means no correction was recorded in this dataset.

Compare the time from code ready to review start, then from acceptance to release. Identify the longest visible wait. Investigate its cause before calling it avoidable. These timestamps do not measure active effort or identify when an incident began. A correction release alone cannot establish failed deployment recovery time.

Download the fictional dataset (CSV)