We did our in-basket research
a valid in-basket for the assessment of management performance
We love in-basket research. On the whole, in-basket HighLight has been appraised as an efficient, reliable and valid instrument for the assessment of management skills. Moreover, the test has shown above-average acceptance with participants and users.
Since 2011, more than 3000 executives from different industries have been put through the online in-basket HighLight to assess leadership potential or as part of employee selection. This huge amount of results provides informative and thorough statistical analyses in order to ensure exact standardization and validation of the test exercises.
The current reference group comprises the results of 1110 candidates.
Our in-basket research shows that reliability for the eighteen scales following is between α = .90 and α = .98 (Cronbach’s Alpha). The reliability of the in-basket HighLight was confirmed, the overall consistency across the 18 criteria was high (.93). Furthermore, the evaluation of participant feedback shows above-average acceptance of the test, as well as very high comprehensibility of its criteria.
Alpha values above .90 are classified as ‘excellent’.
Evaluation of participant feedback
Participants like the Highlight in-basket. Half of the participants report that Highlight in-basket is realistic. Another 38% says: it ‘s quite realistic.
Several experts agree that the test items measure the most important aspects of the theoretical constructs (e.g. proactive behavior, quality assurance) exhaustively.
Human Resources Consultants and Human Resources Developments experts confirmed our selection of competencies in a survey. None of the competencies in the HighLight in-basket exercise was found irrelevant or inappropriate in the study.
Experts were asked explicitly if competencies were missing, but they did not add any response. In sum, these expert agreements can be considered as representations for a high content validity.
A comparison for the test results of sub-groups (e.g. business managers, middle management, experts, employees without leadership experience) showed differences between the groups, which were expected. We found a significant difference in the scores for ‘business managers’ and the ‘standard group’ regarding the criteria for ‘analytical and logical reasoning’, ‘goal economic behaviour’ and ‘goal efficiency’. These group differences can be considered as evidence for construct validity of the in-basket exercise HighLight.
Additionally, no significant differences were found between the test results of female and male candidates.
Highlight is a Situational Judgement Tests (SJT), Research shows that SJT are very good predictors of future effectiveness and performance. There is a clear reason for it: SJT and in baskets are simulations. They are worksamples with realistic tasks and working situations. Even when intelligence tests are part of the assessment center, these SJT still add incremental validity.
Operators and participants report that the HighLight in-basket results correspond to a high extent with observations from assessment centres or self-evaluation of test-takers. These ratings apply notably to the criteria as follows:
- Economic approach
- Management of resources
- Analytical and logical reasoning
- Activities concerning customers and employees
- Research behaviour
- Monitoring behaviour
- Willingless to delegate