Situational test design

We focus here on situational judgement test design.

                  I.           Match to job competencies

  • You can buy off-the-shelf SJTs specific to your industry.
  • Or which are specific to a function such as sales.
  • However, you then lose one bit of SJT magic that makes them special. This is that each question can be crafted to assess critical role behaviours.

                II.           Conduct a range of job analyses

  • You should run critical incidents role analysis to capture all the critical behaviours. These are those leading to successful or unsuccessful workplace outcomes.
  • For example, a happy or an unhappy customer interaction.
  • If there are national – or international – role differences then it’s vital to document differences. In particular for the US.
  • We recommend a blend of face-to-face strategic interviews with senior stakeholders.
  • Plus, a telephone-based approach for job incumbents.
  • Also, running separate focus groups for all line managers.

            III.           Probe interviewees (for a range of options)

A few people in your sample will provide most of the information you need. Simply must include the obligatory Pareto Law quote here and say it’s exactly 80% of what you need. Take your time and probe these individuals carefully, going beyond the critical scenarios to explore possible correct and incorrect responses to that situation. Ideally also what a less experienced colleague may do in that situation. Then voila, your interviewee has written the SJT question for you!

             IV.           Follow the “Rules of Item Writing”

More seriously there’s lots of “rules” to writing clear and concise SJT scenarios and answer options. Too many to cover here in detail. Suffice to say that for each broken item writing rule, your SJT’s validity will suffer that little bit more.

               V.           Score both answers

When designing the scoring key/format, my preference is usually to score both the Most and the Least answers chosen. This has always worked well in terms of variance and final item selection.

              VI.           Include an SME Trial

A subject-matter expert (SME) review is vital. It must be done pre-trial so that any unsuitable scenarios are discredited before any valuable trial time is wasted on them. Do schedule in your own editorial time since you need to prepare a finessed Role Incumbent Trial version next.

Aim for 15+ subject matter experts from across the business who have different selection, managing and developing responsibilities for the role being assessed. You can sell this process to senior managers as “the real role experts”, thereby enhancing their engagement in your entire SJT design project!

           VII.            Check scenario realism

Your trial phase gives you the opportunity to get respondents to rate how realistic the scenario/answer options are. This is invaluable at the trial data analysis phase.

         VIII.           Maximise trial numbers

Aim for 300 current role incumbents to trial your SJT. Do extend the trial period if necessary to achieve this. However, such a high number is needed to ensure that you have sufficient numbers within each subgroup (for your equal opportunities analysis).

             IX.            Validate

To set accurate cut-off scores, collect criterion validity evidence. You have two samples. Firstly, your trial group. secondly, your validation sample. Focus on those trial participants for whom you also have job performance data.

                X.           Updated norm group

My final, key lesson is to use an enhanced norm group as soon as you can, ideally from the start. Get sufficient numbers of “live applicants”. This is then the best norm group to use. Any cut-off/equal opportunities analysis can be adjusted accordingly.

Lazard Brothers and AECOM graduate engineers

  • Situational judgement test validations

British Army Situational Judgement Test Designs

Psychometric lead role with Kenexa IBM; managing twenty associates.

  • We developed over twenty psychometric tests;
  • Situational judgement tests for Officers and for Soldiers;
  • Realistic job previews for Officers and for soldiers;
  • Ability tests (including problem-solving test) for Officers;
  • Ability tests (including a spatial reasoning test) for soldiers;
  • Officer personality questionnaire;
  • Soldier career guidance tools.

Education and public sectors

  • Numerical reasoning, verbal reasoning, abstract reasoning and decision analysis tests.
  • Critical reasoning aptitude test (legal sector).
  • Situational judgement test (health sector).

Citibank Situational Judgement Test Design

  • Designing blended assessments for retail and call centre operations across the USA.
  • Allowing for cultural differences in Mexican operations.
  • Managing item writing team for SJTs, ability tests, biodata questions etc.

Situational Judgement Test design examples

In 2015 we produced an online situational judgement test and a realistic job preview for BUPA’s Care Assistant role.

Each Care Assistant SJT scenario addressed Bupa-specific value(s) and high quality approach to residential care. Whilst, also showing a trainee Care Assistant was highly likely to encounter in their first few days of employment.

BUPA Aims for situational judgment test design

The first priority was a reliable and face valid situational judgement test. The realistic job preview (RJP) would then be an adapted, non-test version of the SJT.

Situational judgment test design challenges

  1. Improve the current recruitment process for care assistants using a sift to remove those applicants who did not demonstrate Bupa’s caring and resilient approach to care.
  2. Use realistic job scenarios to capture the key values and behaviours for effective performance as a Care Assistant. This was in both residential and/or nursing Bupa care homes.
  3. Using an advanced situational judgement Best and Worst answer format. This was to determine which of these key values and behaviours was measureable.
  4. Produce a final version with up to 20 of the best situations. This was based upon analysis of the trial situational judgement questionnaire results.

The Key Steps to a Situational Judgement Test Solution

In consultation with Bupa, an assessment tool was produced via the following stages:

  • The Care Assistant job description and BUPA values documentation were analysed.
  • Role analysis sessions based on the collection of critical incidents.
  • Visionary interviews were held with Care Home Managers.
  • Interview notes were analysed.
  • Scenario writing was completed for each competency/value.
  • Each scenario was based on the job analysis output.

Key Steps Part II

  • Two item writers used their own item-writing style to create a broad range of scenarios.
  • Each item writer then reviewed the other’s items for ease of understanding, social desirability and suitability for assessing Care Assistant skills (using an SJT).
  • A consensus was reached as to the final set of representative trial item scenarios.
  • Produced a trial (Subject Matter Expert) version for completion and review by a small cohort of Care Home Managers. Their comments were particularly useful for editing scenarios and ensuring the realism of the trial scenarios.
  • An edited and improved trial (Care Assistant) version was then produced.
  • Arrangements were made to conduct a paper trial of 70 care assistants working around the UK.
  • For each scenario, trial participants were also asked to rate how realistic the scenario was. Plus, to add any comments.

Key Steps Part II

  • BUPA trial participants were asked to rate the effectiveness of each action associated with each situation. The rating scale was from Totally Ineffective to Highly Effective.
  • Convene a panel meeting with several Care Home Managers and recruitment personnel.
  • Then review each scenario’s trial statistics.
  • The consistency of statistical analysis was presented to these SMEs for discussion and final scenario selection.

 The Beneficial Outcomes

  • Best practice was followed throughout the design process in SJT design.
  • An SJT was produced which successfully incorporated a range of care home-specific scenarios.
  • SJT’s also make excellent company-specific tests. With a bespoke psychometric test, organisations can design the content of their test(s) to match their own industry sector. 

Additional assessment practice

Values Assessment

Our Personal Values Assessment is available online here. Read these questions. Then rate each values question from 1-5:

Personal Values Test sample question 1 –Success at work is of high value to me.

2 – My opinion is valued at work.

3 – I value the amount of recognition I receive at work.

4 – I respect the advice and opinion (on work matters) of my manager.

5 – My own goals align with the goals of most of my work colleagues.

6 – I value the time I spent with my team colleagues.

Our Passed Papers 11 plus test papers

Our first 11 plus practice test bundle contains the following 11 plus practice tests:

11 Plus English Test One || 11 Plus Maths Test One || 11 Plus NVR (Non Verbal Reasoning) Test One || 11 Plus Verbal Reasoning Test One.

Our second advanced 11 plus test bundle contains the following 11 plus advanced tests:

11 Plus English Test Two || 11 Plus Maths Test Two || 11 Plus NVR (Non Verbal Reasoning) Test Two || 11 Plus Verbal Reasoning Test Two.

10 Study Skills Tips

Find time to study.

  • In our opinion if you manage your time badly, you will be less productive. So manage it well! This can lead to increased stress and anxiety levels. In particular, around exam time.

Keep to a routine.

  • Work in the same place at the same time each day. Also, make sure you have everything you need before you start.

Work to your strengths.

  • Schedule challenging tasks for when you are most alert. And routine ones for when you are more tired.

Don’t waste time.

  • Rather than reading irrelevant material, skim and scan. Then decide if you need to read something in-depth.

Avoid distractions.

  • Switch emails and social media off. This prevents your mind wandering while trying to learn new information!

Regularly review your notes – Edit out what you don’t need. Ask yourself the question, Is this relevant to my assignment? How does it relate to what I already know?

Vary how you to take notes.

  • Use Mind Maps and diagrams to generate ideas. Or to focus your ideas for essay or report plans.

Be critical

  • Make sure that you always add your own comment to every concept or quotation that you write down. Maintain a critical and analytical approach at all times!

Plan your work.

  • If writing an assignment produce a detailed plan before you start to write it. This will make the drafting process much less stressful

Understand different styles.

  • Understand different writing styles. For exampls, academic journal and journalistic styles. You can put what you read into perspective. In particular, you can become more aware of any particular bias.

Realistic Job Preview Design

This are an area that Rob Williams Assessment Ltd also specialise in. This builds on our situational judgement test design and we are often asked to design both a realistic job preview and a situational judgement test at the same time.

Our approach to the design of realistic job previews has the following phases.

Many application processes now start online with a realistic job preview design that potential applicants take on the company website to “test” if the role still appears suitable after finding out about the realities of the role. Hence it’s a job preview but a realistic one.

Realistic Job Preview format

Typically, the test format is similar to an SJT design. However, the realistic job preview (RJP) is not a test or assessment. Thus, there’s usually a more sophisticated SJT used. This is in a subsequent phase.

In more sophisticated examples still graphics are used. Or video technology. These further enhance the Realistic job preview / SJT experience for candidates.

Realistic Role Preview Design – Step-by-step role previews 

  • Plan a representative sample.
  • Conduct a highly detailed  set of telephone interviews and focus groups.
  • Item writing for the SJT / RJP.SJT / RJP item reviewing.
  • Compile a trial version to be reviewed by 10-12 subject matter experts (SME’s) in the role.
  • Trial the SJT and rJP.
  • Conduct statistical analysis.
  • Present trial results at a standard-setting meeting with SME’s.
  • Validate the SJT and RJP.
  • Norm the SJT and RJP.

– – – Situational Test Design – – –

Our Psychometric Design Toolkit

Strengths Psychometric ~ Our psychometric test designPsychometric design criteria ~ Our personality test design  ~ Work personality test designPersonality asssessment designRealistic Job Preview Design.