We focus here on situational judgement test design.
I. Match to job competencies
- You can buy off-the-shelf SJTs specific to your industry.
- Or which are specific to a function such as sales.
- However, you then lose one bit of SJT magic that makes them special. This is that each question can be crafted to assess critical role behaviours.
II. Conduct a range of job analyses
- You should run critical incidents role analysis to capture all the critical behaviours. These are those leading to successful or unsuccessful workplace outcomes.
- For example, a happy or an unhappy customer interaction.
- If there are national – or international – role differences then it’s vital to document differences. In particular for the US.
- We recommend a blend of face-to-face strategic interviews with senior stakeholders.
- Plus, a telephone-based approach for job incumbents.
- Also, running separate focus groups for all line managers.
III. Probe interviewees (for a range of options)
A few people in your sample will provide most of the information you need. Simply must include the obligatory Pareto Law quote here and say it’s exactly 80% of what you need. Take your time and probe these individuals carefully, going beyond the critical scenarios to explore possible correct and incorrect responses to that situation. Ideally also what a less experienced colleague may do in that situation. Then voila, your interviewee has written the SJT question for you!
IV. Follow the “Rules of Item Writing”
More seriously there’s lots of “rules” to writing clear and concise SJT scenarios and answer options. Too many to cover here in detail. Suffice to say that for each broken item writing rule, your SJT’s validity will suffer that little bit more.
V. Score both answers
When designing the scoring key/format, my preference is usually to score both the Most and the Least answers chosen. This has always worked well in terms of variance and final item selection.
VI. Include an SME Trial
A subject-matter expert (SME) review is vital. It must be done pre-trial so that any unsuitable scenarios are discredited before any valuable trial time is wasted on them. Do schedule in your own editorial time since you need to prepare a finessed Role Incumbent Trial version next.
Aim for 15+ subject matter experts from across the business who have different selection, managing and developing responsibilities for the role being assessed. You can sell this process to senior managers as “the real role experts”, thereby enhancing their engagement in your entire SJT design project!
VII. Check scenario realism
Your trial phase gives you the opportunity to get respondents to rate how realistic the scenario/answer options are. This is invaluable at the trial data analysis phase.
VIII. Maximise trial numbers
Aim for 300 current role incumbents to trial your SJT. Do extend the trial period if necessary to achieve this. However, such a high number is needed to ensure that you have sufficient numbers within each subgroup (for your equal opportunities analysis).
To set accurate cut-off scores, collect criterion validity evidence. You have two samples. Firstly, your trial group. secondly, your validation sample. Focus on those trial participants for whom you also have job performance data.
X. Updated norm group
My final, key lesson is to use an enhanced norm group as soon as you can, ideally from the start. Get sufficient numbers of “live applicants”. This is then the best norm group to use. Any cut-off/equal opportunities analysis can be adjusted accordingly.
Lazard Brothers and AECOM graduate engineers
- Situational judgement test validations
British Army Situational Judgement Test Designs
Psychometric lead role with Kenexa IBM; managing twenty associates.
- We developed over twenty psychometric tests;
- Situational judgement tests for Officers and for Soldiers;
- Realistic job previews for Officers and for soldiers;
- Ability tests (including problem-solving test) for Officers;
- Ability tests (including a spatial reasoning test) for soldiers;
- Officer personality questionnaire;
- Soldier career guidance tools.
Education and public sectors
- Numerical reasoning, verbal reasoning, abstract reasoning and decision analysis tests.
- Critical reasoning aptitude test (legal sector).
- Situational judgement test (health sector).
Citibank Situational Judgement Test Design
- Designing blended assessments for retail and call centre operations across the USA.
- Allowing for cultural differences in Mexican operations.
- Managing item writing team for SJTs, ability tests, biodata questions etc.
Situational Judgement Test examples
In 2015 we produced an online situational judgement test and a realistic job preview for BUPA’s Care Assistant role.
Each Care Assistant SJT scenario addressed Bupa-specific value(s) and high quality approach to residential care. Whilst, also showing a trainee Care Assistant was highly likely to encounter in their first few days of employment.
BUPA Aims for situational judgment test design
The first priority was a reliable and face valid situational judgement test. The realistic job preview (RJP) would then be an adapted, non-test version of the SJT.
SJT design challenges
- Improve the current recruitment process for care assistants using a sift to remove those applicants who did not demonstrate Bupa’s caring and resilient approach to care.
- Use realistic job scenarios to capture the key values and behaviours for effective performance as a Care Assistant. This was in both residential and/or nursing Bupa care homes.
- Using an advanced situational judgement Best and Worst answer format. This was to determine which of these key values and behaviours was measureable.
- Produce a final version with up to 20 of the best situations. This was based upon analysis of the trial situational judgement questionnaire results.
The Key Steps to a Situational Judgement Test Solution
In consultation with Bupa, an assessment tool was produced via the following stages:
- The Care Assistant job description and BUPA values documentation were analysed.
- Role analysis sessions based on the collection of critical incidents.
- Visionary interviews were held with Care Home Managers.
- Interview notes were analysed.
- Scenario writing was completed for each competency/value.
- Each scenario was based on the job analysis output.
Key Steps Part II
- Two item writers used their own item-writing style to create a broad range of scenarios.
- Each item writer then reviewed the other’s items for ease of understanding, social desirability and suitability for assessing Care Assistant skills (using an SJT).
- A consensus was reached as to the final set of representative trial item scenarios.
- Produced a trial (Subject Matter Expert) version for completion and review by a small cohort of Care Home Managers. Their comments were particularly useful for editing scenarios and ensuring the realism of the trial scenarios.
- An edited and improved trial (Care Assistant) version was then produced.
- Arrangements were made to conduct a paper trial of 70 care assistants working around the UK.
- For each scenario, trial participants were also asked to rate how realistic the scenario was. Plus, to add any comments.
Situational Judgement Test Solution Steps Part III
- BUPA trial participants were asked to rate the effectiveness of each action associated with each situation. The rating scale was from Totally Ineffective to Highly Effective.
- Convene a panel meeting with several Care Home Managers and recruitment personnel.
- Then review each scenario’s trial statistics.
- The consistency of statistical analysis was presented to these SMEs for discussion and final scenario selection.
The Beneficial Outcomes
- Best practice was followed throughout the design process in SJT design.
- An SJT was produced which successfully incorporated a range of care home-specific scenarios.
- SJT’s also make excellent company-specific tests. With a bespoke psychometric test, organisations can design the content of their test(s) to match their own industry sector.
Seefor more info on the Army Situational judgement test design and the Officer Situational questions design.
Medical schools, for example, use situational judgment tests to assess the temperament for managing high-pressure situations in the most moral, effective way possible. These situational judgment tests are unique to the medical school context. They use bespoke scenarios representing those critical incidents a doctor needs to know how to manage. It’s important to note that these SJTs don’t require extensive medical knowledge; rather they evaluate a candidate’s decision-making skills.
10 Study Skills Tips
Find time to study.
- In our opinion if you manage your time badly, you will be less productive. So manage it well! This can lead to increased stress and anxiety levels. In particular, around exam time.
Keep to a routine.
- Work in the same place at the same time each day. Also, make sure you have everything you need before you start.
Work to your strengths.
- Schedule challenging tasks for when you are most alert. And routine ones for when you are more tired.
Don’t waste time.
- Rather than reading irrelevant material, skim and scan. Then decide if you need to read something in-depth.
- Switch emails and social media off. This prevents your mind wandering while trying to learn new information!
Regularly review your notes – Edit out what you don’t need. Ask yourself the question, Is this relevant to my assignment? How does it relate to what I already know?
Vary how you to take notes.
- Use Mind Maps and diagrams to generate ideas. Or to focus your ideas for essay or report plans.
- Make sure that you always add your own comment to every concept or quotation that you write down. Maintain a critical and analytical approach at all times!
Plan your work.
- If writing an assignment produce a detailed plan before you start to write it. This will make the drafting process much less stressful
Understand different styles.
- Understand different writing styles. For exampls, academic journal and journalistic styles. You can put what you read into perspective. In particular, you can become more aware of any particular bias.
Realistic Job Preview Design
This are an area that Rob Williams Assessment Ltd also specialise in. This builds on our situational judgement test design and we are often asked to design both a realistic job preview and a situational judgement test at the same time.
Our approach to the design of realistic job previews has the following phases.
Many application processes now start online with a realistic job preview design that potential applicants take on the company website to “test” if the role still appears suitable after finding out about the realities of the role. Hence it’s a job preview but a realistic one.
Realistic Job Preview format
Typically, the test format is similar to an SJT design. However, the realistic job preview (RJP) is not a test or assessment. Thus, there’s usually a more sophisticated SJT used. This is in a subsequent phase.
Step-by-step role preview design
- Plan a representative sample.
- Conduct a highly detailed set of telephone interviews and focus groups.
- Item writing for the SJT / RJP.SJT / RJP item reviewing.
- Trial the SJT and rJP.
- Conduct statistical analysis.
- Present trial results at a standard-setting meeting with SME’s.
- Validate the SJT and RJP.
- Norm the SJT and RJP.
Rob Williams latest book
Are you chasing a job that you really want, but need to take a verbal reasoning test to get it? With the help of this book, you’ll sharpen your skills and quickly become confident in your ability to pass.
Brilliant Passing Numerical Reasoning Tests book – also available for download from the Apple Store.
– – – Situational Test Design – – –