Situational judgment tests (SJT) are commonly used employee selection tools that present job applicants with realistic work situations. Job applicants are required to choose from several response options that range in their effectiveness, as rated by subject matter experts. While SJTs tend to predict future job performance rather well, there is still debate as to what SJTs actually measure (e.g., cognitive ability, personality, job knowledge, or experience) and how they operate in different contexts (e.g., job applicants vs. incumbents).
JOB INCUMBENTS VERSUS JOB APPLICANTS
To further investigate the validity of SJTs, researchers (MacKenzie et al., 2010) addressed a few major issues common in SJT research. For example, much of the existing research on SJTs relies on samples of job incumbents. However, SJTs are most frequently used in new employee selection and the authors speculate that the knowledge, skills, and abilities of incumbents may differ significantly from that of job applicants. In other words, SJTs may operate differently for job incumbents and applicants.
The researchers collected SJT data from applicant and incumbent samples from six different organizations, arriving at a total sample of over 40,000 people. Because the SJTs were tailored to the individual organizations using them, six different SJTs were employed. However, all six were similarly designed (each situation was accompanied by five response options and participants chose the best and worst responses).
For all six organizations, job incumbents tended to have higher SJT scores than applicants. More importantly, the findings suggest that SJTs do function differently for job applicants and incumbents. For all six organizations, cognitive ability was more strongly related to SJT scores for job incumbents than for applicants.
IMPLICATIONS FOR ORGANIZATIONS
These findings suggest that practitioners should take care when interpreting the results of SJT research conducted on job incumbents, especially if the SJTs are intended for use with new employees in selection settings. Nevertheless, SJTs enhance our ability to make predictions about future job performance and are even known to reduce adverse impact against racial minorities, which is a common by-product of using pure cognitive ability tests in selection.
MacKenzie, W.I., Ployhart, R.E., Weekley, J.A., & Ehlers, C. (2010). Contextual effects on SJT responses: An examination of construct validity and mean differences across applicant and incumbent contexts. Human Performance, 23(1), 1-21.