Game based assessment design

We work with HireVue on their game based assessment design and new gamified assessment research. HireVue is the world leader in video assessment.

Game Based Assessment design (GBA), or gamified assessment, involves a blend of psychometric design expertise, game design and machine learning.

GAME BASED ASSESSMENT

Rob Williams Assessment Ltd designs personality fit assessments for companies, typically growing start-ups in London.

Rather than using traditional personality tests, we design much more cost-effective personality fit tools. The scoring algorithms can be enhanced over time using artificial intelligence. With training companies and consultancies, Rob Williams Assessment Ltd build coaching and employability profiling tools. With recruitment agencies we also design culture and role fit personality-based assessments to improve the efficiency of matching clients to the best-fitting jobs.

This reflects a growing recognition of the need to match individual applicants to organisational culture. Typically graduates favoured corporate cultures but now often find the culture of SME’s/start-ups to be more engaging and ‘meaningful’.

In fact, an increasing number of recruitment agencies use Rob Williams Assessment Ltd’s bespoke algorithms for culture fit to improve their shortlisting process for key clients. More accurate shortlisting created efficiencies for both the agency and for their clients. Shortlisted candidates are also happier; knowing that the agency is only putting their details forward to the jobs and companies that are the best fit for them, personally.

Rob Williams Assessment also design gamified personality assessments for Hire Vue. Increasingly, these form a highly engaging component of HireVue’s video-based assessments. HireVue use AI increasingly to build mobile-first assessments combining both video interview, cognitive test and personality assessment components.

Some of the most popular HireVue assessments we’ve helped to design have assessed the cognitive aspects of personality traits. For example, the identification of emotions in other people. OR of spotting the use of empathy can be used to influence and to lead others. Meanwhile, Hire Vue this year exceeded over ten million video interviews.

GENERAL AND SPECIFIC COGNITIVE ABILITIES

Mental abilities and aptitudes refer to the individual’s intellectual potential — What can the person do, and what are their intellectual limits? How easy will it be to develop new skills, to pass examinations, and to respond to career demands. The concept of intelligence is made of general and specific intellectual abilities. General intelligence is a broad general ability that is involved in all types of intellectual performances. Its existence is demonstrated by the positive correlations between different intellectual tests (e.g. maths, English, geography, IQ.). This general ability is made up of moderately related set of primary abilities.

There are several specific mental aptitudes. The most important ones are:

– – – HireVue Video Based Assessments – – –

Verbal reasoning assessment

This is the ability to understand verbal ideas and to reason with words.

It is often connected with literary careers but is also a very important attribute in those careers which involve the ability to find the right word in the right time.

This may be the spoken word or the written word and is the single most useful strength in any kind of academic study.

Numerical reasoning

The numerical ability is similar to, but not the same as, mathematical ability. It is the ability to think in numbers rather than the ability to manipulate them. There are comparatively few careers which require numerical ability alone; it is more often mixed with other abilities. Careers which heavily dependent upon this ability include those of an auditor, an accountant, a wage clerk, a bank teller, a bank officer, financial consultant, financial manager, and many areas of the financial businesses.

Visualization

This involves the ability to “see” abstract information and to make sense of it. It is one of the corner stones of scientific thinking. The stages of building concepts, discovery, and proving theories all rely heavily on this aptitude. Thus, it is the key to most scientific-based careers, including those as a research scientist, a laboratory technician, a veterinary surgeon, a dietician, hospital technician and similar science-based personnel at all levels.

Analytical reasoning

This is the ability to make logical, factual connections and to impose a structure on what sometimes appears to be chaotic information. This reflects the ability to think quickly, confining to the facts only, to solve problems and to deal with new ideas. It is often combined with other aptitudes to indicate the direction in which this ability to think is going to be used. This is particularly important for careers such as computer programmer, researcher, or analyst.

Spatial Ability

The spatial ability involves visual skills. It enables a person to visualise a solid three-dimensional object when given limited two-dimensional information. It is the corner-stone of understanding technical drawings, layout, and relationships between objects in space; as such it will be used heavily by draughtsmen, creative artists, photographers, architects, and designers.

Accuracy and Speed Skills

This ability allows an individual to do routine tasks quickly and with great accuracy. It is one of the few aptitudes that can increase with practice, but the results give a realistic indication as to how easy an individual finds this kind of task compared with others. It is of great importance in quality control, and of particular use in many administration and clerical areas such as filing, typing, computer operation.

Digital Assessment – Graduates

GBAs are becoming a core component of many graduate recruitment recruitment schemes. For example Accenture, Deloitte and Unilever’s graduate recruitment schemes.

Games-based assessments are the future I   GradWeb  I  April 6, 2016

Time to reboot the gamification of recruitment | Daniel Howden | June 7, 2017

Using games for graduate selection – what’s the score?  I  Managing Director, Arctic Shores

Digital Assessment Designs – High Volume

Revelian approach to game based assessment

Revelian’s Cognify

Going digital is the future of candidate assessment  I  Alexander Mann  I Mar 8 2017

Gamification in Recruiting: It’s Becoming More Fun to Apply for a Job  I  April 04, 2016

The Future of Psychometric Testing in the Workplace: Testing in a Digital Era – Gaming types

Games as Personality Profiling Tools  I  Van Lankveld, Spronck, Van den Herik, and Arnoud Arntz

Can we reliably measure the general factor of intelligence (g) through commercial video games? Yes, we can!

Psychometric Considerations in game based assessment

Digital Assessment Designs – Game Based Learning

More Than Just a Dream: How Real Teachers Use Game-Based Assessment Every Day

Designing assessments for a digitalised world | Cambridge Assessment

E-assessment – Cambridge Assessment

Expand your talent pipeline by using video interviews

Unless they introduce video interviews. These can be completed virtually. Even via any candidate’s mobile. A video interview does then allow employers to drill down on a candidate’s employment value. Just as they would do in a face-to-face interview!

So, you fill your recruiting pipeline with worthy candidates. Then, your next stage in the recruiting lifecycle is to evaluate each candidate’s skills and qualifications. To move only the right ones through the pipeline.

There are other pitfalls in the conventional interviewing process that recruiters and hiring managers rely on. First, in spite of an evaluator’s best intentions, bias can creep into their process, both in content and manner, that can put a candidate at a disadvantage.

Second, an unstructured interviewing process can also unintentionally gloss over the most important skills that lead to on-the-job success.

Finally, it’s not always easy to build a scalable, repeatable process that can evaluate a large pool of good candidates without letting some slip through the cracks.

Game based learning

The popularity of video games has drawn researchers’ attention in the exploration of the possibility of  using video games to enhance knowledge, skills, and other personal attributes.

The idea of using games for serious purposes other than entertainment is called game-based learning. Advocates of game-based learning argue that well-designed video games represent solid learning principles. A fair amount of research shows that game-based learning is at least as effective as nongame conditions, such as class- room contexts.

– – – HireVue Video Based Assessments – – –

In addition, when game-based assessment is designed following a principled design framework such as evidence-centered design or cognitive design system, the assessment is likely to have high validity and reliability.

Game-based assessment is essentially performance-based assessment. Tasks that require students to demonstrate their knowledge and skills by working through a task. Rather than a simple test of one’s ability to recall or recognize information, or supply self-reported information, performance-based assessment provides students with the opportunity to show their understanding and apply knowledge in meaningful settings.

Game based learning Part II

Scholars generally support the use of performance-based assessment to measure and support twenty-first-century skills over conventional types of assessment such as multiple-choice questions or filling in the blanks. However, there are a few challenges associated with the design and implementation of performance-based assessments. Some of the more difficult challenges include: (a) designing contexts that will fully elicit the competencies to be measured, (b) modeling the multidimensionality of constructs to be measured, (c) ensuring the validity and reliability (consistency) of the tasks, (d) providing appropriate feedback that is customized to each  individual situation, (e) automating the scoring of the various tasks, (f) accumulating the evidence across all task performances, and (g) reducing the development costs of assessments compared to traditional tests. Our premise in this chapter is that stealth assessment coupled with ECD provides a viable solution to these challenges.

Good games can provide an engaging and authentic environment designed to keep practice meaningful and personally relevant. With simulated visualization, authentic problem solving, and instant feedback, computer games can afford a realistic framework for experimentation.

– – – HireVue Video Based Assessments – – –

Another key feature of well-designed games that can enhance learning and motivation is adaptivity related to providing appropriate and adaptive levels of challenge. Gee has argued that the secret of a good game is not its D graphics and other bells and whistles, but its underlying architecture in which each level dances around the outer limits of the player’s abilities, seeking at every point to be hard enough to be just doable.

Similarly, psychologists (e.g., Vygotsky) have long argued that the best instruction hovers at the boundary of a student’s competence. Flow is another name for this phenomenon. It is a construct first proposed by Csikszentmihalyi to describe an optimal experiential state that involves complete immersion in an activity and a deep sense of enjoyment. Flow represents full engagement, which is crucial for deep learning.

Hire Vue game based assessments

Gamified Assessment Research articles

These articles explain the occupational assessment, brain training and educational fields in which the nascent field of game based assessment is rapidly gaining wider and wider usage.

Exploring the relationship between video game expertise and fluid intelligence

The new Generation of Cognitive Assessment |  Revellian |

More Than Just a Dream: How Real Teachers Use Game-Based Assessment Every Day I  Dec 7, 2015

Gamification in Recruiting: It’s Becoming More Fun to Apply for a Job  I  April 04, 2016

Competing for engagement I  KPMG

Psychometrics and Games-Based Assessment | Mark Parkinson  | November 24, 2015

Time to reboot the gamification of recruitment | Daniel Howden | June 7, 2017

5 Reasons Why Game-based Assessment is the Hottest New Trend in Education I Van Voorhis and Snyder I Nov 13, 2017

Game based assessment design research papers

Why game based assessments could be destined to fail I Irish Tech News I May 21, 2018

Clustering Game Behavior Data I IEEE Transactions on Computational Intelligence and AI in Games  I  3 Sept 2015

Designing a Learning Game for Career Readiness with DreamWakers, a Q&A  I August 10, 2017 | Waskow, Baron, & Gray

Pymetrics game based assessment provider

Gamified assessment

Here are some usefuls links that demonstrate the progress of digital assessment over the last fifteen to twenty years.

Unilever gamification; game based assessment; Unilever digital interview.

Here are a couple more niche game based assessment providers.

Digital assessments / gamified assessments

Pymetrics has a history of designing digital assessments / gamified assessments for Unilever. Pymetrics develops neuroscience games and leverages bias-free AI for.

  • Digitalised job matching.
  • Recruiting high performers (HiPOs).

Like MindX, Pymetrics adopts a more attractive and engaging game-based assessment process which is considerably shorter than traditiona psychometric testing. Twenty minutes of gameplay behaviour with Pymetrics or MindX games will assess high performer potential in specific job roles/industries. MindX focuses on the candidate’s cognitive potential, whilst Pymetrics also offers a range of emotional potential metrics.

Improving Candidate Diversity

In 2017/18 there has been an increasing focus – in both the popular press and the recruitment industry – on removing bias from how applicant women / candidate men are employed.

The use of psychometrics has always been plagued with the inherent bias of verbal reasoning – and increasingly less so – of numerical reasoning tests. Gamified assessment offers AI which can increase applicant pools and machine learning which can monitor and adjust recruitment ratios for gender.

MindX published it’s most widely read blog in early 2018 on predictive hiring bias.

Game based assessment examples

Eon Game Based Assessment

PwC Game Based Assessment preparation

Try PwC’s Gamified Assessment

PwC graduate assessment program

The latest Gamified assessment research

Al-Azawi, R., Al-Faliti, F., & Al-Blushi, M. (2016). Educational Gamification Vs. Game Based Learning: Comparative Study. International Journal of Innovation, Management and Technology, 7(4), 132–136.

Banfield, J. & Wilkerson, B. (2014). Increasing Student Intrinsic Motivation And Self-Efficacy Through Gamification Pedagogy. Contemporary Issues In Education Research, 7(4), 291.

Bittner, J. V. & Schipper, J. (2014). Motivational effects and age differences of gamification in product advertising. Journal of Consumer Marketing, 31(5), 391–400.

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From Game Design Elements to Gamefulness: Defining “Gamification”. MindTrek’11, Tampere, Finland.

DiCerbo, K. E. (2014). Game-based assessment of persistence. Journal of Educational Technology & Society, 17(1), 17-28. Hamari, J., Koivisto, J., & Sarsa, H. (2014, January).

Does Gamification Work? – A Literature Review of Empirical Studies on Gamification. 47th Hawaii International Conference on System Sciences (HICSS), Waikoloa, Hawaii. Hausknecht, J. P., Day, D. V., & Thomas, S. C. (2004).

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does Gamification Work? – A Literature Review of Empirical Studies on Gamification. 47th Hawaii International Conference on System Sciences (HICSS), Waikoloa, Hawaii.

Latest GBA research

Montefiori, L. (2016). Games-Based Assessment, insight from a Tech Start-up. Presented at a Symposium/Forum on Game-based assessment – Concepts and insight from research and practice.

Scholastic Testing Service, Inc. van Lankveld, G., Spronck, P., van den Herik, J., & Arntz, A. (2011). Games as personality profiling tools. In Computational Intelligence and Games (CIG), 2011 IEEE Conference on (pp. 197-202). IEEE.

Ventura, M., Shute, V., & Zhao, W. (2013). The relationship between video game use and a performance-based measure of persistence. Computers & Education, 60(1), 52-58.

Yee, N., Ducheneaut, N., Nelson, L., & Likarish, P. (2011, May). Introverted elves & conscientious gnomes: the expression of personality in world of warcraft. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 753-762). ACM.

Wohn, D. Y., & Wash, R. (2013). A virtual “Room” with a cue: Detecting personality through spatial customization in a city simulation game. Computers in Human Behavior, 29(1), 155- 159.

Latest Game Based Assessment Research

  • Armstrong, P., & Anthoney, S. (2009). Personality facets and RIASEC interests: An integrated model.” Journal o f Vocational Behavior.
  • Arthur, W. (2017). The impact o f emerging technologies on selection models and research: Mobile devices and gamification as exemplars.
  • Barrick, M., & Mount, M. (1991). the big five personality dimensions and job performance: a meta-analysis. Personnel Psychology.
  • Brkich, M., Jeffs, D., & Carless, S. A. (2002). A global self-report measure of person-job fit. European Journal o f Psychological Assessment.
  • Buhrmester, M., Kwang, T., & Gosling, S. (2011). Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives On Psychological Science.
  • Chamorro-Premuzic, T. (2016). New talent signals: Shiny new objects or a brave new world? Industrial and Organizational Psychology.
  • Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin.
  • Chapman, D. S., & Webster, J. (2003). The use of technologies in the recruiting, screening, and selection processes for job candidates. International Journal of Selection & Assessment.
  • Collmus, A., Armstrong, M., & Landers, R. (2016). game-thinking within social media to recruit and select job candidates. Social media in employee selection and recruitment.
  • Costa, PT., Jr., McCrae, R.R., & Holland, J.L. (1984). Personality and vocational interests in an adult sample. Journal o f Applied Psychology.
  • Cut-e (2017). White Paper: Ahead o f the game. Best practice in games, gamification and game-based assessment. Retrieved from: https://www.cut-e.com/online assessment/gamification-in-recruitment/httpwwwcut-ecomgamification-white paper.

GBA Research Papers

  • Dale, S. (2014). Gamification. Business Information Review.
  • DiCerbo, K. E. (2014). Game-Based Assessment of Persistence. Journal O f Educational Technology & Society.
  • DiCerbo, K., Shute, V., & Kim, Y. (2016). The future of assessment in technology-rich environments: Psychometric considerations. Learning, Design, and Technology.
  • Dik, Bryan J., Eldridge, Brandy M., Steger, Michael F., & Duffy, Ryan D. (2012).Development and validation of the calling and vocation questionnaire (CVQ) and brief calling scale (BCS). Journal of Career Assessment.
  • Donaldson, S., & Grant-Vallone, I. (2002). Understanding self-report bias in organizational behavior research. Journal o f Business and Psychology.
  • Holland, J.L. (1985). Making vocational choices: A theory of vocational personalities and work environments (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.
  • Hunter & Hunter (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin.
  • Hutchinson, T. (2014). Assessing the congruence of worker and workplace using the
    proximities of their RIASEC types. Australian Journal of Career Development.
  • Jaffal, Y., & Wloka, D. (2015). Employing game analytics techniques in the psychometric measurement of game-based assessments with dynamic content. Journal o f E-Learning.
  • Kim, Y., Almond, R., & Shute, V. (2015). Applying evidence-centered design for the development of game-based assessments in physics playground. International Journal o f Testing.
  • Shute, V. (2015). The interplay of game elements with psychometric qualities, learning, and enjoyment in game-based assessment. Computers & Education.
    Landers, R. (2014). Developing a Theory of Gamified Learning: Linking Serious Games and Gamification of Learning. Simulation & Gaming.

Latest Research

  • Landers, R. N. (2015). An introduction to game-based assessment: Frameworks for the measurement of knowledge, skills, abilities and other human characteristics using behaviors observed within videogames. International Journal of Gaming. Industrial and Organizational Psychology.
  • Lowman, G. H. (2016). Moving beyond identification: Using gamification to attract and retain talent. Industrial and Organizational Psychology.
  • Marcus, B., & Wagner, U. (2015). What do you want to be? criterion-related validity of attained vocational aspirations versus inventoried person-vocation fit. Journal of Business and Psychology
  • Mislevy (2015). Psychometrics and game-based assessment. In Technology and Testing: Improving Educational and Psychological Measurement.
  • Montefiori, L. (2016). Game-based assessment: Face validity, fairness perception, and impact on employer’s brand image. Assessment & Development Matters.

The latest Game Based Assessment Research

  • Moorman, R., & Podsakoff, P. (1992). A meta-analytic review and empirical test of the potential confounding effects of social desirability response sets in organizational behaviour research. Journal of Occupational and Organizational Psychology.
  • Morelli, N., Mahan, R., & Illingworth, A. (2014). Establishing the measurement equivalence of online selection assessments delivered on mobile versus nonmobile devices. International Journal o f Selection and Assessment.
  • Nauta, M. M. (2010). The development, evolution, and status of Holland’s theory of vocational personalities: Reflections and future directions for counseling.
  • Rupp (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. The Journal o f Technology Learning and Assessment.
  • Ryan (2006). The motivational pull of video games: A self-determination theory approach. Motivation and Emotion.
  • Shute (2016). Measuring problem solving skills via stealth assessment in an engaging video game. 

Aptitude test practice books

Rob Williams’s five practice aptitude tests books are all available on Amazon:

Firstly, in our opinion this is the best aptitude test practice book for Passing Verbal Reasoning Tests.

41b7m573tbl

Secondly, in our opinion this is the best aptitude test practice book forPassing Numerical Reasoning Tests.

Passing Numerical Reasoning Tests gif

– – –Game Based Assessment Design – – –

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.