The Criterion Problem: Actual vs Conceptual vs Composite Performance Measures

Understand the criterion problem: why measuring performance is harder than it seems. Learn about actual, conceptual, and composite performance measures and their limitations.
The Criterion Problem: Actual vs Conceptual vs Composite Performance Measures
"The criterion problem is fundamentally a problem of values. What we choose to measure and how we choose to measure it reflects what we believe is important—and those beliefs may not align with what actually matters for organizational success." — James T. Austin & Peter Villanova, Journal of Applied Psychology (1992)
Executive Summary
What if the performance metrics you trust are missing critical dimensions of success—or measuring things that have nothing to do with actual job performance? One of the most persistent and underappreciated challenges in organizational psychology is what researchers call the criterion problem: the fundamental difficulty of identifying, defining, and measuring job performance.
The gap between the ultimate criterion (what we theoretically want to measure), the conceptual criterion (how we define it), and the actual criterion (what we actually measure) creates systematic errors that distort organizational decisions.
Understanding this problem—and the empirical evidence about what works—is essential for building fair, effective performance management systems.
The Criterion Problem: Why Measurement Is Harder Than It Looks
The criterion problem, identified decades ago by researcher Flanagan (1956) and comprehensively reviewed by Austin and Villanova (1992), asks a deceptively simple question: How do we know if someone is performing well at their job?
The answer is far more complex than it appears.
The Three Levels of Criteria
Ultimate Criterion: What We Want to Measure. The ultimate criterion is the theoretical construct—what we ideally want to assess. For a salesperson, the ultimate criterion might be "success in generating revenue while maintaining customer relationships and contributing to team effectiveness." The ultimate criterion is rarely if ever directly measurable. It's abstract, multidimensional, and context-dependent.
Conceptual Criterion: How We Define It. The conceptual criterion is how we operationalize the ultimate criterion in specific contexts. For the salesperson, we might define it as: "Individual performance on selling tasks + willingness to help colleagues + adaptability to changing market conditions." The conceptual criterion is still primarily a theoretical construct, but it's more specific and measurable.
Actual Criterion: What We Actually Measure. The actual criterion is what we measure in practice: sales figures, supervisor ratings, customer satisfaction scores, or attendance records. This is where systematic problems emerge. The actual criterion rarely, if ever, perfectly captures the conceptual criterion. The gap between conceptual and actual criteria creates two distinct problems: criterion deficiency and criterion contamination.
Criterion Deficiency: When Measures Miss Critical Dimensions
Criterion deficiency occurs when the actual measure fails to capture important components of the conceptual criterion. Important aspects of performance are simply omitted.
Classic Example: Imagine you're hiring an administrative assistant and you administer a work sample test involving organizing files, scheduling meetings, and managing correspondence. The test seems comprehensive and objective. However, the work sample doesn't assess the applicant's ability to type on a computer because the test uses paper-based filing systems. Yet in the actual job, computer typing is essential. Your measure is deficient—it misses a critical performance dimension.
Unlock the Full Research
Create a free account or sign in to access the complete methodology, deep-dive analysis, and key implementation frameworks.
Don't have an account? It's free to join.
Organization Learning Labs
Research & Insights Division



Comments