Research plan
Research plan
Calavion intends to study four broad areas in order, through seven proposed studies: what it takes to prepare a case, how long managers take to review the evidence, how hiring teams behave when they have it, and, much later, outcomes. Each study has a different kind of evidence and supports different claims. We publish the plan so readers can hold us to it, and we label every finding by type.
How we label evidence
| Label | Meaning | May be used to claim | May not be used to claim |
|---|---|---|---|
| Fictional demonstration | A made-up case run with scripted or internal users | That the mechanics work as described | Anything about real hiring, applicants or outcomes |
| Internal measurement | A measurement of Calavion’s own materials or tooling | What the tooling does or what preparation contains | That customers, managers or applicants behave a certain way |
| Customer finding | Data from a real engagement, with the company’s consent | What happened in that engagement | General effects, unless the sample supports it |
| Outcome study | Follow-up of hires or decisions against a defined outcome | Only what its design supports | Causation without a suitable comparison |
The studies
| # | Question | Evidence type | Data needed | Status |
|---|---|---|---|---|
| 1 | What does preparing a case contain, and where does Calavion have to assume? | Internal measurement | Prepared cases; the “what we understood” sheet; review items | First measurement published: one fictional case and the live cancellation case. Human preparation time not yet measured. |
| 2 | How much human time does preparing a case take, from first file to company approval? | Internal measurement, then customer finding | Time logs from the person preparing and the company reviewing, across at least 5 cases | Not started. Needs real engagements |
| 3 | How long does it take a manager to review a Replay and decide? How does that compare with their current step? | Customer finding | Reviewer time per applicant; company’s current process time; number of applicants per role | Waiting for a first design partner |
| 4 | Do reviewers agree on what the record shows (observation) more than on what it means (interpretation)? | Customer finding | At least two independent reviewers per Replay; their separate notes | Waiting for a design partner |
| 5 | How do hiring teams use the evidence in the interview and the decision? What do they cite? | Customer finding (qualitative) | Interview recordings or notes with consent; decision rationales | Waiting for a design partner |
| 6 | Do applicants find the exercise fair and relevant? | Customer finding | Applicant survey after the exercise; completion and drop-out | Waiting for a design partner; applicant-facing consent must be final first |
| 7 | Is performance in the environment related to later job outcomes? | Outcome study | Defined outcome measures, adequate sample, comparison group, months of follow-up | Long-term. No claim will be made until a design supports one |
Design principles
- Consent and privacy first. Customer findings use only data the company and applicants have agreed to share, with identifying details removed.
- Pre-specified measures. We write down what we will measure before we look.
- Report the null. If a study shows no benefit or a cost, we publish that.
- Observation and interpretation stay separate in our own analysis, as in our method.
- No fairness claims without a fairness study. Fairness and adverse-impact analysis needs a large enough sample and appropriate expertise. We will say plainly when we cannot do it, and we will seek independent review.
What we will not claim until the data exists
- That the method improves hire quality, retention or speed.
- That it reduces bias, or is fairer than other methods.
- That performance in an environment predicts job performance.
For what existing research says about related methods, see What research says about seeing applicants perform real work. If you are an employer interested in being a design partner, email us.