Article 3 of 5
Test Run
The execution of chosen cases against one version, at one moment.
A test case says what should happen. A test run records what actually happened — on this build, on this environment, today. That is why results live on the run and not on the case.
Run the same cases tomorrow and you get a new run. The old one stays as evidence of what you knew when you shipped.
What it is made of
- Name
- Usually the purpose plus the build: “Regression Run #4”, “Smoke — Release 1.4”.
- Selection
- Which cases are included. A smoke run is a handful; a regression run is most of them.
- Results
- One status per case, plus a comment when something went wrong.
Result statuses
- Passed
- The actual result matched the expected result. Every step of it.
- Failed
- The product behaved differently than expected. This is a bug in the product.
- Blocked
- You could not check it: the environment was down, a dependency was broken, the feature was not deployed. Nothing is known about this case.
- Skipped
- You deliberately did not run it — out of scope for this run, or not worth the time.
- Not Run
- Not reached yet. The run is unfinished.
Rules that save you later
- Failed and Blocked are not the same. Failed says the product is wrong; Blocked says you learned nothing. Marking blocked cases as failed invents bugs that do not exist.
- Never edit a finished run to make the numbers look better. Fix the product and start a new run.