Situational judgment test design

Our first situational judgement test case study is for our client BUPA.

Situational judgment test design

In 2015 Bupa commissioned a partnership between Rob Williams Assessment Ltd and Work Brighter Ltd to produce an online situational judgement test and a realistic job preview for their Care Assistant role.

BUPA Situational Judgement Test design

Care Assistant applicants would answer a representative range of care home-based scenarios. Each scenario would be addressing Bupa-specific value(s) and high quality approach to residential care. Whilst, also representing a typical situation a trainee Care Assistant was highly likely to encounter in their first few days of employment.

Aims and Challenges for situational judgment test design

The first priority was a reliable and face valid situational judgement test. The realistic job preview (RJP) would then be an adapted, non-test version of the situational judgement test (SJT) used to demonstrate to potential applicants the nature of the role.

In summary, the four main project aims/challenges were to:

  1. Improve the current recruitment process for care assistants using a sift to remove those applicants who did not demonstrate Bupa’s caring and resilient approach to care.
  2. Use realistic job scenarios to capture the key values and behaviours for effective performance as a Care Assistant in both residential and/or nursing Bupa care homes.
  3. Using an advanced situational judgement Best and Worst answer format to determine which of these key values and behaviours could be measured both effectively and consistently.
  4. A final version with up to 20 of the best situations based upon analysis of the trial situational judgement questionnaire results.

The Key Steps to a Situational Judgement Test Solution

In consultation with Bupa, an assessment tool was produced via the following stages:

  • The Care Assistant job description and BUPA values documentation were analysed to determine which competencies/values were measurable using an SJT scenario-based approach to assessment.
  • Role analysis sessions based on the collection of critical incidents were conducted. A representative sample of experienced Care Assistants was involved across a range of BUPA care homes in northern, southern and central England.
  • Visionary interviews were held with Care Home Managers at each Bupa care home involved in this job analysis stage.
  • Interview notes were analysed and scenario writing completed for each competency/value. Each scenario was based on the job analysis output, hence described a situation relevant to the job.

Key Steps Part II

  • Two item writers used their own item-writing style to create a broad range of scenarios. Each item writer then reviewed the other’s items for ease of understanding, social desirability and suitability for assessing Care Assistant skills (using an SJT). A consensus was reached as to the final set of representative trial item scenarios.
  • Produced a trial (Subject Matter Expert) version for completion and review by a small cohort of Care Home Managers. Their comments were particularly useful for editing scenarios and ensuring the realism of the trial scenarios.
  • An edited and improved trial (Care Assistant) version was then produced.
  • Arrangements were made to conduct a paper trial of 70 care assistants working around the UK. For each scenario, trial participants were also asked to rate how realistic the scenario was and to add any comments they had.
  • BUPA trial participants were asked to rate the effectiveness of each action associated with each situation using a rating scale from Totally Ineffective to Highly Effective.
  • Convening a key stakeholder panel meeting with several Care Home Managers and recruitment personnel to review each scenario’s trial statistics.
  • The consistency of statistical analysis was presented to these Bupa subject matter experts (SMEs) for discussion and final scenario selection.

 The Beneficial Outcomes

  • Best practice was followed throughout the design process in SJT design.
  • An SJT was produced which successfully incorporated a range of care home-specific scenarios.

Situational Judgement Test Design –  Selection Sift

With the UK headed for a triple-dip recession, graduates face an increasingly challenging job market. According to the latest figures, 40% of graduates have failed to find graduate-level positions two years after finishing their degree. To help them select the very best candidates from an abundance of quality. Understanding these tests and how they are used will give applicants a competitive edge in today’s job market.

Candidates may also have to take a personality test as part of the recruitment process. There is a vast array of personality tests, which pose questions about a candidate’s behaviour and personal preferences. A typical question may ask whether you prefer attending parties or staying home with a good book. These personality tests help employers to determine whether a candidate has the right profile for the role.  The best strategy for completing a personality test is to answer honestly, as these tests are carefully designed.

Situational judgement tests (SJTs) have also become prevalent in graduate recruitment. These tests presents scenarios to applicants and asks them to select the best and the worst thing to do next. The scenarios are set within the context of the recruiting organisation, so the questions are usually perceived favourably by candidates. SJTs are very popular in the United States due to their excellent record of fairness across different ethnic groups.  

Another trend is offering company-specific tests. With a bespoke psychometric test, organisations can design the content of their test(s) to match their own industry sector.