What it takes to prepare a case: an internal measurement
What it takes to prepare a case: an internal measurement
Finding. In an internal test of Calavion’s case-preparation tooling, four short files from a fictional company (35 lines in total) became an internally reviewed interactive case, using a fictional company role, with 11 applicant actions (3 record reads, 3 messages and 5 other actions, including finishing the case) and 5 possible endings. Of the 9 statements in the “what we understood” sheet, 3 (one third) did not come from the company’s material: 2 were Calavion assumptions and 1 was something Calavion looked for and could not find. That is the part the company had to review.
What we measured
We counted the contents of two prepared cases and recorded what the preparation and review steps surfaced. We did not measure human time (see Limitations).
The input
| File | Lines |
|---|---|
| Chat export | 7 |
| Account and box records | 16 |
| Policy | 7 |
| Outcome note (what happened; what the company would have wanted) | 5 |
| Total | 35 |
The prepared cases
| Element | Fictional Northwind pause case (prepared from the four files) | Cancellation case (the live sample environment) |
|---|---|---|
| Actors (people and systems) | 4 | 5 |
| Records (entities) | 4 | 5 |
| Applicant actions (record reads, messages and other actions) | 11 | 13 |
| Reactions to applicant actions | 1 | 3 |
| Delayed events | 1 | 2 |
| Evidence rules | 3 | 5 |
| Company policies encoded | 3 | 2 |
| Possible endings | 4 rule-defined (plus “unresolved”, 5 in all) | 3 rule-defined (plus default) |
| Session limit | 20 minutes | 30 minutes |
What the preparation and review surfaced (Northwind case)
| Measure | Result |
|---|---|
| Statements in the “what we understood” sheet | 9: 3 from the records, 3 company rules, 2 Calavion assumptions, 1 not found |
| Share of the sheet not drawn from the company’s material | 3 of 9 (33%) |
| Automated preparation checks on the package | 11 proved, 0 errors |
| Items flagged as needing company review after the checks | 14 |
| Items the company had to answer on its review screen | 4 (two assumptions, one gap, one proposed answer) |
| Criteria drafted | Of the 6 essential and prohibited conditions, 3 are marked critical (1 essential, both prohibited). One ending is also flagged critical, so 4 flags in all. The 5 endings are classed separately as acceptable, partial or unacceptable |
| Deliberately broken test packages submitted to the checks | 2, both refused; nothing staged |
| Sessions run end to end (scripted test personas) | 3 routes, 3 different findings |
| Replay events cited by findings that resolved to a real moment | 26 of 26 |
What we can and cannot conclude
We can say: a small amount of company material can contain enough to build a working case; one important judgment task in preparation is deciding what to assume where the material is silent; and the tooling makes those assumptions visible to the company before approval, rather than burying them.
We cannot say: that any real company’s material would behave this way, that preparation is fast, or that the tooling is a live customer feature (it is not deployed).
Limitations
- Fictional material, one author. The case was invented by Calavion, and the company-side steps were performed by Calavion staff playing the company. Real companies will be messier, and the share of assumptions will differ.
- One prepared case. Two cases are not a sample. The cancellation case was built earlier by a different route and is shown for scale only.
- No human time. The test run records machine time (about 54 seconds end to end, almost all of it running the three sessions). It excludes the time a person needs to read the files and write the case, sheet and criteria. That time is the interesting number, and we do not have it. Do not quote the 54 seconds as a preparation time.
- Scripted personas. The three sessions were not people. They show that different routes produce different findings; they say nothing about how real applicants behave.
- Counts are structural. More actions do not mean a better case.
Reproducibility
The figures come from the case package, the “what we understood” sheet, the criteria file and the operator log generated by Calavion’s automated test (companyCaseProof) on every run. The counts of the cancellation case come from its package definition. We can supply these files with the fictional material to researchers and journalists on request.
What comes next
Study 2 on our research plan measures human preparation time on real engagements. See also the method.