Situational test design

The focus of this page is situational test design case studies and situational test design examples.

Situational test design

In 2015 we produced an online situational judgement test and a realistic job preview for BUPA’s Care Assistant role.

BUPA Situational Judgement Test design

Each Care Assistant SJT scenario addressed Bupa-specific value(s) and high quality approach to residential care. Whilst, also showing a trainee Care Assistant was highly likely to encounter in their first few days of employment.

Aims and Challenges for situational judgment test design

The first priority was a reliable and face valid situational judgement test. The realistic job preview (RJP) would then be an adapted, non-test version of the situational judgement test (SJT) used to demonstrate to potential applicants the nature of the role.

In summary, the four main project aims/challenges were to:

  1. Improve the current recruitment process for care assistants using a sift to remove those applicants who did not demonstrate Bupa’s caring and resilient approach to care.
  2. Use realistic job scenarios to capture the key values and behaviours for effective performance as a Care Assistant in both residential and/or nursing Bupa care homes.
  3. Using an advanced situational judgement Best and Worst answer format to determine which of these key values and behaviours could be measured both effectively and consistently.
  4. A final version with up to 20 of the best situations based upon analysis of the trial situational judgement questionnaire results.

The Key Steps to a Situational Judgement Test Solution

In consultation with Bupa, an assessment tool was produced via the following stages:

  • The Care Assistant job description and BUPA values documentation were analysed.
  • Role analysis sessions based on the collection of critical incidents.
  • Visionary interviews were held with Care Home Managers.
  • Interview notes were analysed and scenario writing completed for each competency/value.
  • Each scenario was based on the job analysis output.

Key Steps Part II

  • Two item writers used their own item-writing style to create a broad range of scenarios. Each item writer then reviewed the other’s items for ease of understanding, social desirability and suitability for assessing Care Assistant skills (using an SJT). A consensus was reached as to the final set of representative trial item scenarios.
  • Produced a trial (Subject Matter Expert) version for completion and review by a small cohort of Care Home Managers. Their comments were particularly useful for editing scenarios and ensuring the realism of the trial scenarios.
  • An edited and improved trial (Care Assistant) version was then produced.
  • Arrangements were made to conduct a paper trial of 70 care assistants working around the UK. For each scenario, trial participants were also asked to rate how realistic the scenario was and to add any comments they had.
  • BUPA trial participants were asked to rate the effectiveness of each action associated with each situation using a rating scale from Totally Ineffective to Highly Effective.
  • Convening a key stakeholder panel meeting with several Care Home Managers and recruitment personnel to review each scenario’s trial statistics.
  • The consistency of statistical analysis was presented to these Bupa subject matter experts (SMEs) for discussion and final scenario selection.

 The Beneficial Outcomes

  • Best practice was followed throughout the design process in SJT design.
  • An SJT was produced which successfully incorporated a range of care home-specific scenarios.
  • SJT’s also make excellent company-specific tests. With a bespoke psychometric test, organisations can design the content of their test(s) to match their own industry sector.

Situational Test Design

                  I.           Match to job competencies

You can buy off-the-shelf SJTs which cater to your industry or which are specific to a function such as sales. However, you then lose that one bit of magic that really makes an SJT special. This is that each question can be crafted to assess critical role behaviours. Having role competencies at your disposal, better yet behavioural indicators is helpful in scenario design. Although…

                II.           Conduct a range of job analyses

You should run critical incidents role analysis to capture all the critical behaviours: defined as those that lead to successful or unsuccessful workplace outcomes. For example, a happy or an unhappy customer interaction. With, or without, a set of up-to-date competencies for your role, you must schedule as many role analysis sessions at the start of your design process as time (and money!) will allow.

If there are national – or international – role differences then it’s vital, particularly in the US, to document how your job analysis plan captures representative sample(s) for each role being assessed. Logistical pros and cons will certainly dictate the balance of role analysis formats you adopt. I usually recommend a blend of face-to-face strategic interviews with senior stakeholders; a telephone-based approach for job incumbents, but running separate focus groups for their line and regional managers.

            III.           Probe interviewees (for a range of options)

A few people in your sample will provide most of the information you need. Simply must include the obligatory Pareto Law quote here and say it’s exactly 80% of what you need. Take your time and probe these individuals carefully, going beyond the critical scenarios to explore possible correct and incorrect responses to that situation. Ideally also what a less experienced colleague may do in that situation. Then voila, your interviewee has written the SJT question for you!

             IV.           Follow the “Rules of Item Writing”

More seriously there’s lots of “rules” to writing clear and concise SJT scenarios and answer options. Too many to cover here in detail. Suffice to say that for each broken item writing rule, your SJT’s validity will suffer that little bit more.

               V.           Score both answers

When designing the scoring key/format, my preference is usually to score both the Most and the Least answers chosen. This has always worked well in terms of variance and final item selection.

              VI.           Include an SME Trial

A subject-matter expert (SME) review is vital. It must be done pre-trial so that any unsuitable scenarios are discredited before any valuable trial time is wasted on them. Do schedule in your own editorial time since you need to prepare a finessed Role Incumbent Trial version next.

Aim for 15+ subject matter experts from across the business who have different selection, managing and developing responsibilities for the role being assessed. You can sell this process to senior managers as “the real role experts”, thereby enhancing their engagement in your entire SJT design project!

           VII.            Check scenario realism

Your trial phase gives you the opportunity to get respondents to rate how realistic the scenario/answer options are. This is invaluable at the trial data analysis phase.

         VIII.           Maximise trial numbers

Aim for 300 current role incumbents to trial your SJT. Do extend the trial period if necessary to achieve this. However, such a high number is needed to ensure that you have sufficient numbers within each subgroup (for your equal opportunities analysis).

             IX.            Validate

To set accurate cut-off scores, collect criterion validity evidence. You have two samples. Firstly, your trial group. secondly, your validation sample. Focus on those trial participants for whom you also have job performance data.

                X.           Updated norm group

My final, key lesson is to use an enhanced norm group as soon as you can, ideally from the start. Get sufficient numbers of “live applicants”. This is then the best norm group to use. Any cut-off/equal opportunities analysis can be adjusted accordingly.

 

Situational Judgement Example Questions

Enjoy our free situational judgement test practice examples.

Working With Others situational judgement practice

EPSO SJT Example 1 – Working with OthersMost Likely / Least Likely
If a problem requiring urgent attention occurred in my department but I know that it was a colleague’s responsibility, I would…
aContinue as normal and stay focused on doing my own job.
bInform my manager about the colleague who was a fault.
cStop doing whatever I was doing and help to fix the problem.M
dConsider it my supervisor’s job to speak to the colleague at fault.L

EPSO Practice – Delivering Quality and Results competency

EPSO SJT Example 1 – Delivering Quality and ResultsMost Likely / Least Likely
Two colleagues are transferred into your team from a poorly performing team; comprised mainly of new joiners. Your manager proposes that you monitor the quality of their work. It is your responsibility to drive their performance upwards.
aProduce and review daily performance reports before sending them to your manager.
bSet additional targets of a type that each colleague has agreed will motivate them          M
cGive the two colleagues a  few days to settle in and then call a meeting.
dAsk each of the two colleagues which reward will be the most effective at motivating them.L

EPSO Practice – Prioritising and Organising competency

EPSO SJT Example 1 – Prioritising and OrganisingMost Likely / Least Likely
You have two very important deadlines to meet by the end of your working day. However it is becoming clear in your final two hours of working that you are in danger of missing both deadlines.
aSpeed up your remaining tasks so that you will still be able to meet both deadlines.
bAim to achieve one deadline and to renegotiate the delivery date for the other.
cWork out what’s left to do and then prioritise the most critical tasks for the time remaining.M
dFocus on still doing a quality job even if you must miss a deadline.L

EPSO Practice – Analysis and Problem-Solving competency

EPSO SJT Example 1 – Analysis and Problem-SolvingMost Likely / Least Likely
This year there’s already been lots of new processes causing some problems amongst your team. As a result, some of your colleagues are now very resistant handling any more change. In fact, one of your closest team colleagues has just confided in you that they really cannot cope with another increase in their daily responsibilities.
aAdvise your superior a colleague was feeling very stressed, which could be a wider team problem          M
bBe empathetic whilst suggesting that they are more confident about their capacity for coping.
cAdvise your colleague to take a deep breath for the time being, and to wait for things to improve.
dSuggest that there are specialists who are experienced at coping with stress at work.L

EPSO Practice – Resilience competency

EPSO SJT Example 2 – ResilienceMost Likely / Least Likely
Each member of your team agreed a set of priorities with your manager. Surprisingly you and the team just heard a presentation by this senior manager outlining a completely different set of priorities.
aGet your team’s perspective on the situation and pass on these views to your manager.
bEmail your senior manager a breakdown of the differences and copy in each of your team.
cFocus on the meeting and its outcomes – the previous set of priorities is clearly now out -of-date.L
dHighlight to your senior manager how what you just heard contradicts what was said before.M

Situational judgement test practice