What research says about seeing applicants perform real work

Calavion Team · Published 25 September 2026 · Updated 25 September 2026 · Pillar: Designing fair opportunities

This is an evidence review of research on hiring methods that ask applicants to perform work or work-like tasks: work-sample tests, job simulations, realistic job previews and, as a comparison, structured interviews. It covers job relevance, reliability, predictive validity, adverse impact, applicant reactions and practical constraints. It does not evaluate Calavion, and it says so plainly in its own section below.

Terms used in this review

• Work-sample test: OPM describes these as requiring applicants to perform tasks or work activities that mirror the tasks employees perform on the job.
• Job simulation: a broader family that includes recreated work settings. Lower-fidelity forms include situational judgment tests; higher-fidelity forms include assessment-center exercises (Lievens & Patterson, 2011).
• Validity (in the tables below): the estimated correlation between a selection method’s scores and later job performance. Higher is better, and .00 means no relationship.
• Realistic job preview: giving applicants an accurate picture of the job, including its unattractive parts, before they accept it.

Predictive validity: moderate, and lower than once thought

Schmidt and Hunter (1998) summarized 85 years of selection research and reported a validity of .54 for work-sample tests. Roth, Bobko and McFarland (2005) re-examined that literature, found the older narrative review behind it to be plagued by methodological problems, and estimated an observed mean correlation of .26 between work-sample tests and job-performance measures, rising to .33 after correcting for unreliability in performance ratings. They described that as about one third lower than previously thought.

Sackett, Zhang, Berry and Lievens (2022) then argued that many earlier meta-analyses overcorrected for restriction of range, so that the validity of many selection procedures had been substantially overestimated. Their revised estimates were lower by .10 to .20 points for most procedures, and structured interviews became the top-ranked procedure. For work samples they kept Roth et al.’s .33 and applied no range-restriction correction, noting that 53 of the 54 underlying studies were concurrent rather than predictive.

Estimated validity for predicting job performance, and Black–White score differences

Scroll sideways to see all columns.

Source: Sackett et al. (2022), Table 3. The lower credibility value indicates how low validity could plausibly be in a setting at the bottom end of the observed range. d is the standardized difference in average scores between Black and White groups in the studies compiled. Only a subset of the article’s rows is shown.

Two cautions apply. First, the work-sample estimate rests almost entirely on concurrent studies, where current employees are tested, which may not match how applicants perform. Second, Roth et al. also found work-sample scores correlated .32 with general cognitive ability tests, so a work sample may partly measure the same thing as an ability test.

Job relevance

OPM states that work-sample tasks show a high degree of content validity, meaning the tasks resemble the work, and that applicants often perceive work samples as fair. OPM also notes that these tests can be costly to develop in both time and money, and recommends them where the measured competencies are highly critical and the number of applicants to test is limited. Job relevance is an argument about resemblance to the job. It is not the same as demonstrated prediction, and the table above shows why the two should be kept apart.

Simulations of different fidelity

Lievens and Patterson (2011) studied 196 applicants in an advanced-level, high-stakes selection setting. A knowledge test, a lower-fidelity situational judgment test and a higher-fidelity assessment center were all significantly related to job performance. The situational judgment test and the assessment center each added predictive value beyond the knowledge test, and the assessment center added value beyond the situational judgment test. This is a single study in one setting.

Arthur, Day, McNelly and Edens (2003) meta-analyzed 34 articles on assessment-center dimensions and reported estimated true validities from .25 to .39 across six dimension categories, such as communication and problem solving.

Structured interviews as the comparison point

Interview research shows how much design matters. Huffcutt and Arthur (1994) found that interview structure was a major moderator of validity across 114 entry-level validity coefficients, with a ceiling beyond which more structure added essentially nothing. Levashina and colleagues (2014) reviewed the structured-interview literature and concluded that much is known but many questions remain open. Sackett et al. (2022) report structured interviews at .42 with a wide spread between studies (lower credibility value .18), which means a structured interview is not guaranteed to outperform other methods in a given setting.

Reliability

The reliability evidence located for this review concerns interviews. Conway, Jako and Goodman (1995) meta-analyzed 111 interrater reliability coefficients and found that interview structure (standardizing questions, standardizing how responses are evaluated, and how ratings are combined) moderated reliability, with multiple ratings proving useful when combined mechanically, and they estimated upper limits of validity of .67 for highly structured and .34 for unstructured interviews. The sources reviewed here do not give a single reliability figure for work-sample or simulation scores. That figure depends on how performance is observed and rated, and it was not established in this review.

Adverse impact and group score differences

The evidence here conflicts, and the conflict is worth stating rather than smoothing over.

• OPM says generally little or no performance differences are found between men and women or applicants of different races on work samples, depending on the competencies measured. The page shows no date and the text reviewed cites no figures.
• Roth, Bobko, McFarland and Buster (2008) noted that the widely used Black–White difference for work samples, d = .38, was plagued by problems such as range restriction. In job-applicant samples they found a markedly larger overall difference, d = .73. The difference depended on what was rated: work-sample ratings of cognitive and job-knowledge skills averaged d = .80, while ratings of social skills ranged from .21 to .27.
• Bobko and Roth (2013) reached a similar conclusion across predictors: some, such as work samples, can show differences similar to paper-and-pencil cognitive tests.
• Sackett et al. (2022) list d = .67 for work-sample tests and d = .23 for structured interviews (table above).
• Ployhart and Holtz (2008) reviewed 16 strategies intended to reduce subgroup differences and adverse impact while keeping validity.

Under the US Uniform Guidelines (29 CFR 1607.4), a selection rate for a group that is less than four-fifths (80 percent) of the rate for the group with the highest rate will generally be regarded by federal enforcement agencies as evidence of adverse impact. That rule concerns selection rates, not average score differences, so the d values above do not by themselves tell any employer what its own selection rates will be.

Applicant reactions

Hausknecht, Day and Thomas (2004) meta-analyzed 86 independent samples (N = 48,750) and found that interviews and work samples were perceived more favorably than cognitive ability tests, which were perceived more favorably than personality inventories, honesty tests, biodata and graphology. Anderson, Salgado and Hülsheger (2010) synthesized 38 samples from 17 countries and reported a similar ordering, with work samples and interviews most preferred, and found that favorability was structurally similar across countries. They also found positive correlations between favorability ratings and validity, which is a correlation across methods and not evidence that liking a method makes it predictive.

Realistic job previews

A meta-analytic path analysis by Earnest, Allen and Landis (2011), covering 52 studies and about 17,000 participants, found that enhanced perceptions of organizational honesty were the primary mechanism by which realistic job previews influenced voluntary turnover. They concluded that the most effective design may be an oral or written preview delivered after hiring and designed to signal organizational honesty. Phillips (1998) meta-analyzed 40 earlier studies; its abstract was not accessible for this review, so no finding from it is reported here. The evidence base concerns previews of jobs, and it does not evaluate assessments as previews.

Practical constraints

• Development cost. OPM says work samples may be costly to develop in time and money and need updating as job tasks change.
• Applicant burden. The sources reviewed contain no reliable figures for how long applicants spend on work samples or simulations, or how time burden affects who completes them.
• Criterion measurement. Validity estimates depend on how job performance was measured, often by supervisor ratings, and on corrections for their unreliability (Roth et al. used .60 for ratings and .80 for objective measures).
• Local evidence. A meta-analytic average does not show how a specific exercise performs for a specific employer.

Where the evidence is thin or in conflict

• No peer-reviewed study specific to take-home assignments was located in this review’s search.
• No source reviewed evaluates interactive environments whose state changes in response to the applicant’s actions, delayed consequences, or replay-style records of applicant work.
• Estimates of validity changed materially between 1998 and 2022 as methods were revisited, so current figures may also change.
• Group-difference estimates depend on sample type and on the constructs rated, and sources disagree.
• Most work-sample validity data are concurrent, not predictive.

What this research does and does not establish about Calavion

Calavion is a real-work hiring platform that turns work a company has handled into an approved work environment and gives the employer a record of the applicant’s work, called Replay. Calavion does not currently produce automatic scores, applicant rankings or hiring decisions. Employers inspect the work and make the decision.

• What it does establish: asking applicants to do work-like tasks is an established family of methods, and its validity is moderate, not high; job resemblance, applicant reactions and group differences all need separate attention.
• What it does not establish: that a prepared environment built from an employer’s own work, with delayed consequences and a review record, predicts performance, treats applicant groups equitably, or improves any outcome. Those are open questions for Calavion.

Sources

Links go to the original publication record or official page. Access date for all sources: 25 September 2026.

1. Anderson, N., Salgado, J. F., & Hülsheger, U. R. (2010). Applicant reactions in selection: Comprehensive meta-analysis into reaction generalization versus situational specificity. International Journal of Selection and Assessment, 18(3), 291–304. doi:10.1111/j.1468-2389.2010.00512.x
2. Arthur, W., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56(1), 125–153. doi:10.1111/j.1744-6570.2003.tb00146.x
3. Bobko, P., & Roth, P. L. (2013). Reviewing, categorizing, and analyzing the literature on Black–White mean differences for predictors of job performance. Personnel Psychology, 66(1), 91–126. doi:10.1111/peps.12007
4. Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565–579. doi:10.1037/0021-9010.80.5.565
5. Earnest, D. R., Allen, D. G., & Landis, R. S. (2011). Mechanisms linking realistic job previews with turnover: A meta-analytic path analysis. Personnel Psychology, 64(4), 865–897. doi:10.1111/j.1744-6570.2011.01230.x
6. Hausknecht, J. P., Day, D. V., & Thomas, S. C. (2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57(3), 639–683. doi:10.1111/j.1744-6570.2004.00003.x
7. Huffcutt, A. I., & Arthur, W. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79(2), 184–190. doi:10.1037/0021-9010.79.2.184
8. Levashina, J., Hartwell, C. J., Morgeson, F. P., & Campion, M. A. (2014). The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology, 67(1), 241–293. doi:10.1111/peps.12052
9. Lievens, F., & Patterson, F. (2011). The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. Journal of Applied Psychology, 96(5), 927–940. doi:10.1037/a0023496
10. Phillips, J. M. (1998). Effects of realistic job previews on multiple organizational outcomes: A meta-analysis. Academy of Management Journal, 41(6), 673–690. doi:10.2307/256964 (abstract not accessible for this review)
11. Ployhart, R. E., & Holtz, B. C. (2008). The diversity–validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology, 61(1), 153–172. doi:10.1111/j.1744-6570.2008.00109.x
12. Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity: Updating and integrating some classic literature. Personnel Psychology, 58(4), 1009–1037. doi:10.1111/j.1744-6570.2005.00714.x
13. Roth, P. L., Bobko, P., McFarland, L. A., & Buster, M. (2008). Work sample tests in personnel selection: A meta-analysis of Black–White differences in overall and exercise scores. Personnel Psychology, 61(3), 637–661. doi:10.1111/j.1744-6570.2008.00125.x
14. Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. doi:10.1037/apl0000994
15. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274. doi:10.1037/0033-2909.124.2.262
16. US Office of Personnel Management. Work samples and simulations. opm.gov (page shows no publication date).
17. Uniform Guidelines on Employee Selection Procedures, 29 CFR 1607.4(D). Text read via Cornell Legal Information Institute; official version at eCFR.

Related Calavion resources

Work samples, job simulations, take-home assignments and real-work hiring compares the formats. Methodology and Definitions explain how Calavion uses its terms. Replay shows what an employer receives, and Product describes the platform. Calavion prepares each environment with the company: bring your work.