Psychometric Test Design. People having colour fight.

Psychometrist

Rob Williams Assessment Ltd specialises in custom psychometric test design services. We believe that a bespoke psychometric test offers significant benefits from improved face validity, predictive validity and improved ROI.

rrussellwilliams@hotmail.co.uk     I     +44 (0)77915 06395

Headed up by Rob Williams – the highly experienced author of five books on psychometric testing. Our organisation prides itself on client satisfaction. We have many positive LinkedIn reviews from our big client projects. 

Psychometric Sift Designs

Many of the UK’s and US’s leading test publishers have used Rob Williams Assessment’s to calculate and prove the reliability of their newly developed psychometric tests.

Rather than using traditional personality tests, we design much more cost-effective personality fit tools. The scoring algorithms can be enhanced over time using artificial intelligence. With training companies and consultancies, Rob Williams Assessment Ltd build coaching and employability profiling tools. With recruitment agencies we also design culture and role fit personality-based assessments to improve the efficiency of matching clients to the best-fitting jobs.

Culture Fit Designs

This reflects a growing recognition of the need to match individual applicants to organisational culture. Typically graduates favoured corporate cultures but now often find the culture of SME’s/start-ups to be more engaging and ‘meaningful’.

In fact, an increasing number of recruitment agencies use Rob Williams Assessment Ltd’s bespoke algorithms for culture fit to improve their shortlisting process for key clients. More accurate shortlisting created efficiencies for both the agency and for their clients. Shortlisted candidates are also happier; knowing that the agency is only putting their details forward to the jobs and companies that are the best fit for them, personally.

Key Steps to Bespoke Personality Test Design

Next, and in consultation with our client, Rob Williams Assessment Ltd, used similar process for designing both the Talent Window and the Hire Window personality tests. The client then arranged for online trials with in excess of 100 members of the target graduate audience.

Custom assessment outcomes

Rob Williams Assessment Ltd successfully applied the British Psychological Society’s best practice in personality questionnaire design.

Both bespoke personality questionnaire met the client’s aim of measuring the most commonly sought graduate personality traits.

Employers are increasingly using bespoke situational judgement tests (where the candidate is presented with scenarios and asked to select the best and the worst thing to do next) as a way to learn more about their character and attitudes to work.

Rob Williams Assessment also design gamified personality assessments for Hire Vue. Increasingly, these form a highly engaging component of HireVue’s video-based assessments. 

Psychometric Test Design. Man at desk working at laptop, picture taken from the top

Branching situational judgement tests 

As psychometric tests have become more commonplace, the bigger users have commissioned their own bespoke situational judgement tests. Rob Williams Assessment has worked on several such projects for High Street banks and for the European Union. Another recent innovation of test developers has been online adaptive tests. With these tests, if you are doing well, you will find that the questions get progressively harder. That can feel like a challenge since you are pushed until you reach the most challenging level you can. This is the level at which you – just like other candidates with your level of verbal reasoning – start to get questions wrong.

The innovative design of shorter and more efficient tests was driven by an increasingly aware of the immediacy of the Internet and our increasing use of emails and social media, in short, sharp bursts.  This discourages test takers from spending 30-40 minutes online doing the same questionnaire. It’s better for everyone to keep test takers engaged when being tested – not bored!

So what will adaptive tests mean for you as a prospective test taker?  The biggest difference is the shortness of the test. The second major difference is that you will find an adaptive test more challenging. Without getting into their highly technical make-up, the test adapts to your ability level. More specifically it adapts to find the most challenging question that you can answer correctly.

In the past, you may have found questions on a test fluctuating in difficulty or generally becoming more and more difficult the further on you get in the test. Consider a test of twenty questions with the first the easiest and the twentieth the most difficult.

Knowledge-based situational judgement tests 

Some or all of the scenarios presented in an SJT can test specific job knowledge. For example, a retail marketing SJT may ask questions about the 3Ps (price, position, promotion) of product marketing. Alternatively, both an SJT measuring generic decision-making skills may be used alongside a knowledge-based test.

Video-based Situational judgment tests 

Simulated situational judgement tests are increasingly common as recruitment sifts. Adding 2D or 3D workplace scenario graphics brings the situational judgment test scenarios to life. This can only promote the company brand and make employers using simulated situational judgment tests more desirable employers.

UK and US psychometric test publishers have produced both video-based and animated SJT scenarios. Animated SJTs are easier – and therefore cheaper – for global companies to develop.

Bespoke situational judgement test design

Situational judgement test design clients include BUPA, Citibank, British Airways and Boots.

Bespoke aptitude test sifts

  • Verbal reasoning test design.
  • Numerical reasoning test design.
  • Abstract reasoning test design.

Aptitude test design clients include EPSO, Kenexa IBM and HireWindow.

Verbal reasoning test practice for Managerial roles

Most managers will need to use higher levels of verbal reasoning when reading or preparing reports. They need to be able to adapt their spoken and written communication style to the situation, whether addressing their subordinates or customers/ clients. Other company reporting procedures, such as appraisals, also require clearly written documentation.

Senior managers and directors will need to use the highest levels of verbal reasoning skills when analysing company reports, dealing with compliance issues and statutory obligations. Here there is a need for concise and accurate communication.

Verbal reasoning test practice for Customer service roles

Effective oral communication is the key to handling customer queries or sales calls. Talking to customers on the phone or face to face demands a flexible communication style. For example, telesales personnel would be expected to respond differently to a customer who was complaining than to one who was a prospective sale. Persuasive presentation skills also rely upon a solid foundation of verbal reasoning skills.

Graduate jobs

Verbal reasoning test practice for PA or administrative roles

A PA’s responsibilities typically include written correspondence. For example, letters and emails, which need to use an appropriate tone and level for the intended audience. Administrative roles also need to check written documents. Also, to file these accurately. Plus, to keep on top of plans and procedures that have been agreed orally or in writing.

Verbal reasoning practice tests – Sales roles

Effective oral communication is the key for converting sales call prospects. In particular, sales roles in call centres which require an even more fluent style of communication style.

Psychometrically Sound Tests

To be psychometrically sound a test must be:

  • Objective – the results obtained are not influenced by the administrator’s personal characteristics or irrelevant factors such as the colour of a test taker’s socks.
  • Standardised – the test is administered and scored according to standard procedures and people’s scores are compared to known standards.
  • Reliable – the test measures in a consistent way. The potential error is small and is quantifiable.
  • Valid – the test measures the characteristics which it set out to measure. A test used to select a job applicant should predict job performance. A test of verbal ability should predict this area and not some other skill.
  • Discriminating – the test should show clear differences between individuals on the behaviour being tested. It should not be discriminatory I.e. unfairly discriminating against minority groups on the basis of irrelevant characteristics

Example Psychometric Test Projects 

Rob Williams has over twenty years of experience of bespoke psychometric test design. Plus ten years prior to this spent working for several of the UK’s leading test publishers. These include IBM, OPP, SHL and HireVue.

British Airways 

Blended assessment project:

  • Situational judgment test design;
  • Numerical reasoning test design;
  • Verbal reasoning test design
  • Personality questionnaire design.

For these two British Airways roles:

– Cabin crew assessment; and

– Customer service representative assessment.

BUPA SJT Design

  • Best practice was followed throughout the design process in SJT design.
  • An SJT was produced which successfully incorporated a range of care home-specific scenarios.
  • The most suitable set of scenarios could be hand-picked at the SME panel meeting, as well as gaining buy-in and discussing implementation.
  • Providing some scenarios for telephone interview sifts.
  • Setting a suitable cut-off and validating the tool.

EPSO Test Design

  • Development of project management test.
  • Design of IT skills-based aptitude tests.

Psychometric test design

Work Styles assessment design

We will work with you to design the most suitable work styles tool to suit your needs. Our Bespoke Personality Questionnaire design process aims to:

  • Firstly, include key role dimensions.
  • Secondly, reflect the personality, attitudinal and motivational aspects of the role-specific dimensions.
  • Thirdly, have face valid questions.
  • Also, to be capable of completion in 20 minutes approx.
  • And to adopt a single-stimulus question format (Likert scale)
  • Plus, adopting a normative format of scoring utilising a sten look-up table (for each personality scale)
  • And finally, using a Social Desirability scale to deal with the issue of faking or extreme scoring patterns

Psychometric test design

Situational judgement tests trends in 2019

Situational judgement tests (SJTs) have also become prevalent in graduate recruitment. These tests presents scenarios to applicants and asks them to select the best and the worst thing to do next. SJT’s are very popular in the United States due to their excellent record of fairness across different ethnic groups.

Personality test trends in 2019

Candidates may also have to take a personality test as part of the recruitment process. There is a vast array of personality tests, which pose questions about a candidate’s behaviour and personal preferences. A typical question may ask whether you prefer attending parties or staying home with a good book. These personality tests help employers to determine whether a candidate has the right profile for the role.

Personality Tests Trend 2019 – Biodata questions

Personality biodata questions measure a variety of constructs, including attitudes, personality attributes, interests, skills/abilities, past events and experiences. An individual’s learning history is of particular interest.

Bespoke graduate assessment designs

Infact it is bespoke aptitude designs which are our speciality. Otherwise known as company-specific tests. With a bespoke psychometric test, organisations can design the content of their test(s) to match their own industry sector.

Understanding graduate assessments gives those applicants a competitive edge in today’s job market.

Assessment design

At a time when the graduate market is at its most competitive it pays for all graduates to perform their best on these psychometric tests. It makes sense to be prepared. Fortunately, there is a wide variety of practice material available, allowing graduates to familiarize themselves with the sort of tests they’ll encounter as part of the recruitment process.

Graduate recruiters who want to minimise recruitment costs see online ability tests as an efficient and effective means of sifting the initial applicant pool. Verbal and numerical reasoning tests offer a reliable and standardised way to ensure that an optimal number of good candidates are seen by the business at the much more expensive ‘face-to-face’ stage. Chances are that this next stage will involve a retest in order to verify that the candidate who took the online test was who they said they were!

Aptitude test solutions

Our consultancy work focuses on aptitude test practice and bespoke psychometric test design. We believe in the benefits of practice and ensuring fairness in testing. We therefore offer some practice aptitude tests and some practice psychometric tests. The intention is to promote as ‘level a playing-field’ as possible.

Aptitude test design – Employability

All of our psychometric test practice resources are free. So, we hope you enjoy using them!   In our opinion, there is plenty of the right type of aptitude test practice, aptitude test tips and test-taking strategies.

We certainly hope you get maximum value from our free psychometric test practice resources!  Psychometric test practice works best when it’s specific to the test type and the level of difficulty of the psychometric test you will be taking.

Assessment Design / Psychometric Test Design

Aptitude test design example

Firstly, it’s fundamental to the use of any psychometric test is that users are aware of a test’s effectiveness and its known limitations.  This information is calculable through knowledge of a measure’s reliability and validity. 

Secondly, importance is attached to test results and decisions are made using test data.  Thirdly, you must be sure that the tests you are using have sound credentials and have been properly developed.

‘Bespoke’ Verbal Reasoning Test Design

Specific design criteria were applied.  Sufficient administration time was made available for a test taker to exhibit the appropriate reasoning ability. 

Other design criteria related to the target population groups. For example, this form of verbal reasoning test does not require the candidate to have any technical knowledge of grammar. Or to be able to spot minor errors in the spelling of unfamiliar words.

Practical examples are proved at the start of each test. Thus, test takers can familiarise themselves with the test format.

APTITUDE TEST DESIGN RATIONALE

Many jobs involve working with verbal information and verbal comprehension forms a core component of almost all senior managerial roles.  The ‘Bespoke’ Verbal Reasoning Test measures the verbal reasoning skills that are fundamental to effective communication in such roles. 

In many organisations, verbal reasoning skills are key to the effective dissemination of business information across the workforce.

‘Bespoke’ Verbal Reasoning Test assesses how well an individual’s verbal reasoning skills can operate at a high-level.  In our opinion, primarily understanding written communication. Although, ‘Bespoke’ Verbal Reasoning Test also encompasses the ability to understand complex discussions.

Verbal reasoning is central to many roles. Thus the ‘Bespoke’ Verbal Reasoning Test is appropriate for a very wide range of senior job roles and tasks.

APTITUDE TEST FORMAT

Many graduate and senior managerial roles require quickly extracting relevant information from written documents. And to make a judgement based on this information.  Thus ‘Bespoke’ Verbal Reasoning Test measures the ability to read. And to interpret a detailed block of text under strictly timed conditions.

The verbal information in the test appears in the form of passages of text. Each is followed by a series of 4 multiple-choice questions.  Each question requires relevant pieces of information to be extracted from the passage.And a specific judgement to be made on the basis of that information. 

TEST DESIGN

After reading a passage the test taker has to read a series of statements referring back to information contained within the passage.  The test taker has to identify whether the statement is true. Or false. Or whether it is not possible to tell. Then to decide which is the appropriate multiple-choice answer to fill in. 

Each answer must be based solely on the information presented in the passage – ignoring any background knowledge that the test taker may possess.  The questions must also be answered without any interference from the test taker’s own beliefs about the subject matter. 

This reflects work conditions where there is a need to make objective decisions based solely on the information available at that moment in time.

Number of Items:          48

Test Time:         25 minutes

Time needed for Administration (including Test Time):     35 minutes

EXAMPLE APTITUDE ITEM

For each statement, fill in either T, F or CS on the answer sheet.

These corresponds to your decision as to whether the statement is True. False. Or whether it is not possible to tell.

T: True                    

F: False                   

CS: Cannot Say      

Three key psychometric pillars

These are often taken for granted – due to the fine efforts of HireVue’s and MindX’s R&D teams’ conscientious work!

  1. Standardization
  2. Reliability – means if I take the same test next week, my results will be similar.
  3. Validity – means the test measures what it says it measures. MindX knows which personality traits are measured because we have compared our results to well-established personality tests. HireVue validates its video assessments using high performer data and job analysis results.
  4. Being Non-discriminatory

 1. Standardization  

Since the first IQ tests were developed the whole point is to compare with a group of previous test-takers.

2. Reliability

There are two key types of psychometric reliability:

  • Test-Retest Reliability
  • Internal reliability.

Test-Retest Reliability

This concerns score consistency over time. If the test is measuring some stable characteristic of the individual such as intelligence, then the test should produce similar results if a group of candidates are examined using the same test at two points in time. The time gap between the measurement is important, as some psychological characteristics change considerably over time. Yet, a short gap of two weeks should not cause major changes.

Still, an external event in the period between the two assessment might cause a change, but here it is expected that the change will affect similarly all the candidates. This type of reliability is also known as test re-test reliability. It is assessed by correlating the tests scores measured first time with the test scores measured the second time. Ability tests are expected to have a reliability of at least r = .75, yet personality tests might have somewhat lower reliability.

Internal Consistency

The internal consistency refers to the question whether all the test items measure the same concept. It can be assessed in two ways. The first method is known as the split-half reliability which require correlating the score based on half of the test items with the score based on the other half (e.g., scores on odd and even items). Alternative method is item-total reliability which requires correlating each item score with the total score of the rest of the items. There is a statistical measure called Cronbach alpha that summarises all the correlations into one figure. A test should have an alpha of at least a = .80.

Aptitude test’s psychometric terms

Norm-Referenced

Every psychometric test has at least one norm group.

The standalone score has little meaning, so a comparison with previous candidate performance is used. For example, percentile rank.

Pre-test (or Pilot) Item

A trial item as part of a larger-than-necessary set of pre-test items. The  final, refined test is created by refining those items that work best. This is based on statistical analysis of the trial item set’s results.

Aptitude Test Reliability

Firstly, is the test a reliable measure?

Most commonly the internal consistency index coefficient alpha or its dichotomous formulation, KR-20. Under most conditions, these range from 0.0 to 1.0, with 1.0 being perfectly reliable measurement. A reliable test may still not be a valid test.

Aptitude Test Validity 

Secondly, is the test valid? A measure of what it ‘says on the tin’?

In our opinion, both the initial content validation and later criterion validation analysis are vital for any bespoke psychometric test.

We recommend collecting additional recruitment data over time so that additional validation studies can be conducted. Such as assessment centre data.

There are many other types of psychometric test validation evidence, and one-off studies investigating a psychometric test’s criterion validity are common.

British Airways Aptitude Test Design Example

Blended assessment project:

  • Situational judgment test design;
  • Numerical reasoning test design;
  • Verbal reasoning test design
  • Personality questionnaire design.

For these two British Airways roles:

– Cabin crew assessment; and

– Customer service representative assessment.

British Army Aptitude Test Design Example

Psychometric lead role with Kenexa IBM; managing twenty associates.

  • We developed over twenty psychometric tests;
  • Situational judgement tests for Officers and for Soldiers;
  • Realistic job previews for Officers and for soldiers;
  • Ability tests (including problem-solving test) for Officers;
  • Ability tests (including a spatial reasoning test) for soldiers;
  • Officer personality questionnaire;
  • Soldier career guidance tools.

Aptitude Tests’ Vital Role

Many companies today are turning to testing and assessment tools to help them address these challenges and make more substantive and data-driven hiring decisions. Assessments are a great way to level the playing field and evaluate many candidates for the same skill sets in an objective fashion, using real-life technical scenarios that mirror the work they will be performing on the job. Automated assessment tools in particular can scale to make better use of your time and resources. Several key recruitment benefits are listed below.

Differentiation Benefits

Providing unbiased assessments is a great way to distinguish yourself to candidates in a crowded hiring environment. Candidates will see that you really care about hiring the most qualified tech workers in a manner that is unbiased and uncovers their true value.

Recruitment Cost and Recruitment Funnel Benefits

Automated testing helps you scale your recruiting efforts, evaluating many candidates simultaneously to save engineering hours, and ultimately the cost of those used hours, time and costs, which is particularly important for larger tech organizations.

Hiring, training and replacing a bad hire can cost almost 5x their annual salary. Making the right choice the first time around is vital to growing your business and keeping costs low.

Removing bias

Assessment tools give you a structured mechanism to remove bias from the evaluation and interview process. It’s completely objective, surfacing skill sets needed to succeed on the job without basing a decision on your intuition or gut feel. And it mitigates the disadvantage a candidate whose first language is not English may have in a traditional format.

Focusing on skills, not experience

Experience listed in a profile isn’t necessarily the most important indicator of true skills and expertise.

Assessments help you find candidates that may not have a lot of experience in a certain field. Bbut nonetheless have the skills you’re looking for.

Consistent process benefits

Assessment tools help you standardize your ongoing evaluation process. You’ll get an objective look at how candidates perform in the testing phase and can then compare it to their success on the job. A post-mortem can reveal how a good (or bad) hire performed in the assessment and help you to duplicate (or avoid) that performance in the future.

Assessment Interview Questions Example – for Data Scientist

  • Firstly, is there a trade-off between bias and variance?
  • Secondly, how can gradient descent be a problem?
  • Thirdly, when have you used Exploratory Data Analysis?
  • Fourthly, what steps do you take when choosing a Machine Learning model?
  • Next, are convolutions best for images?
  • Then, are Residual Networks important? 
  • Also, how does batch normalization work? 
  • And have you ever worked with an imbalanced dataset?
  • Next, what are the highlights of your MSc research? What didn’t work? What else could you have done? 
  • Then, how have you overcome over/ underfitting your data model?
  • What are dimensionality’s dangers?
  • Next, how would you run a Principal Component Analysis (PCA)?
  • What is data normalization?
  • Also, why is dimensionality reduction important?

Biodata tool psychometric design

Test questions asking about previous working and life history facts. Biodata questions can include personal attitudes, values, beliefs. There are therefore both autographical and biographical perspectives. For example, how effective previous working relationships were with managers and/or colleagues.

Biodata Face validity 

Whilst biodata was popular in the 1970’s/80’s in the UK, it fell out of fashion due to concerns about face validity. Face validity is how job relevant a test’s questions appear to be. This is difficult to show with biodata’s indirect approach; posing biodata fit questions about past behaviours which can seem intrusive.

Application blank psychometric design

Application blanks are a statistical oriented approach is based on the notion that each piece of information has a potential to predict later performance.

The biographic data are evaluated in the same way as test scores. The most predictive biographical items are chosen. Each one discriminates between success and failure. A good item is one in which the ratio of successes in one level of the item differs from the successes ratio in another level of the same item.

Biodata Part II

Firstly in our opinion, these suggestions cannot explain the predictive power of some biographical items.

In our opinion, construct validity studies of biodata scores are mainly based on the notion that past behaviour is the best predictor of future behaviour. Some attempts were concerned with an internal analysis of the biodata components. Also, a degree of success in previous life stages is a good predictor of success in later stages regardless of the context of the success.

Secondly, consider another extreme example. A significant correlation between attendance at a circus show and success as a door-to-door sales-person. Thus, there is a need for a distinction between items that measure past achievement, and those that indicate interest, motivation towards a particular activity, or even social status.

Thirdly, there is no theory behind the relationships between an item and the criterion; theoretically, everything goes. Any item that can discriminate between successes and failures can be used. No attention is paid to the reasons for the discriminative strength of items.

Personnel selection

The basic underlying procedure in personnel selection is the matching between job characteristics and an individual employee characteristic. The job characteristics and employee characteristics are the two main components in any model of personnel selection.

Prediction model

Jobs vary regarding the nature, number, and weight of functions they involve. For example, consider the main functions of two jobs. Perhaps a university lecturers and an accountant.

Thus, each job has its own profile of functions. The first component in a model of personnel selection is that of job functions.

Variance is also introduced because each lecturer’s job depends on the department they work in – or even their university.

A model of personnel selection

The profile of functions might vary in the weight or importance given to each function. For example, additional functions of a university lecturer. These might be unique to certain universities or departments. For example, contribution to policies within their department and or organisation, and assessment of applicants for the course.

Identifying specific job functions, involves beaking down the job into its smallest behavioural and cognitive tasks. Plus, sometimes emotional tasks.

Each job description is a theoretical construct, an abstract idea that can never be measured. Together, they constitute an ideal profile of the functions of the job.

Our Practice assessment books 

Brilliant Passing Numerical Reasoning Tests and Passing Verbal Reasoning Tests.

Passing Numerical Reasoning Tests gif

Numerical Reasoning book reviews

Buy this book to pass!

Brilliant book. Read it from cover to cover. Don’t skip the chapters. I know it can seem patronising, the way he breaks it down, but I’d recommend reading it through.

Brilliantly practical and informative

Very clear layout to follow, questions advance over the …

Very clear layout to follow, questions advance over the chapters and it sets a steady pace. This book helped me immensely to get all the practice I needed within 3 days.

A really useful book 

Numerical Reasoning book reviews

Another really useful book by Rob. I will be recommending this to my delegates on interviewing/assessment techniques training; clear, concise and, if you are applying for jobs with a strong numerical focus, the best £10 you will spend! I heartily recommend.

Five Stars

Great book and pretty hard numerical exercises.

Useful, informative and good value

A very user-friendly and easy to read guide. Really focuses your mind on brushing up on your Maths skills. The author explains clearly how you can prepare to give your best for a test. Very useful and highly recommended.

Five Stars

Great book, tons of info and test to help you understand.

Great to get some practical help on how to succeed … at this stage of selection. I will recommend to anyone who needs to get through a numerical reasoning test.

Passing Verbal Reasoning book reviews

Brilliantly practical advice

The advice and practical examples will answer all your questions about an impending assessment. The book is easy to read and covers preparation, practical examples and an overview of the usual proceedings on test day.he author also highlights questions to be asked before test day and (I was very pleased to see) encourages feedback after assessment. The recaps at the end of every chapter are great for a quick overview before test day. Definitely a worthwhile investment and sure to put you at ease before an assessment of this kind.

A MUST BUY for anyone attempting to pass any verbal reasoning test.

Verbal Book reviews

An informative yet easy to read book which helped me get through graduate-level tests to the next stages of application. Would highly recommend.

Excellent sample tests and explanations

Passed the VRT for HMRC Legal Trainee Scheme this year after reading this book. Helpful.

Very useful and good practice

The book has a good range of tests that reflect what comes up in real tests – so pretty good practice for the real thing

Highly recommended

Just what I needed to further my career. Now I’ve tried the tests I feel more confident.

Good quality and light reading!

An excellent guide to performing well in verbal reasoning tests.

Psychometric test design

Additional assessment practice

Registered Office: Rob Williams, 31 Bruton Way, Ealing, London, W13 0BY

Companies House Registration No: 6572976