Data collection for game-based learning UX research (when you're short on time)

If you're looking for user feedback on a game-based learning title (GBL) but find yourself short of time to carry out lengthy longitudinal studies with hundreds of participants, there's still hope. It's possible to acquire enough data with a 1 hr-1 hr 30 min session for each participant.

Quantitative vs qualitative

You can have a mix of both, just try to avoid claiming your quantitative data is generalisable. Instead, you want to focus on the depth of your qualitative data. Rather than simply asking participants if they feel more confident in a topic area, ask them why, and how they think the game has had any impact.

Sample size

For a long time, it was suggested that you only need to test with 5 users. However, this depends on a number of factors. If you're comparing highly distinct categories users (e.g. comparing adults with children), you'll need 3-4 users for each category as behaviour will naturally differ.

Furthermore, you'll need to ensure users are carrying out the same specific tasks throughout the study. For example, instructing one user pick up a coin level 1 while only telling another user to solve an equation in level 5, will naturally create inconsistent opinion's regarding gameplay. Instructing both participants to do the former makes for easier comparison.

Session time

If different levels or sections of the game contain distinct gameplay mechanics (for example, the entirety of one level is spent flying, while another involves walking around a building), then each of these will need to be tested so user feedback more closely represents the entire game. Conversely, if the gameplay mechanics are same throughout then (e.g. Flappybird, Space Invaders), then only a brief demo of the game needs to be tested for all user experience issues to be identified.

For my own game user research, I tested the BBC's Giving Change with 10 university students, only 10 minutes of gameplay was required before all issues had been identified. But then, the game only takes 15 mins or less to complete all the levels. Also tested were lengthier games such as Algebra Meltdown that simply increase in difficulty until the player gives up, but the gameplay mechanics remain the same throughout. Still, only 10 minutes of gameplay was required before the user had experienced all of the game's functionality.

For large AAA sandbox titles, there probably won't be enough for users to sit through the entire game, therefore you'll need to decide on the minimum viable product (MVP) to test with. Or if your users are more experienced with similar games and have basic expectations for gameplay, a more detailed minimum awesome product (MAP) may be necessary. As such, the time required to experience all the essential features of the game can vary depending on what you want to test. Game publishers tend to recommend restricting playtime to an hour, if you need more then have a 15-minute break in between to avoid fatigue.

Questionnaires and interviews

Distribute any initial demographic questionnaires (e.g. via email), before the gameplay session. This will save time having the participant fill one in during the session. Just be sure to let participants know you're happy to answer any questions they have on items in the questionnaire or on the study in general during the data collection period.

Using trait scales is a little tricky, as many require a certain test-retest period of around 2 weeks to reduce recall bias. Adopting, for example, the mathematics anxiety rating scale (MARS) to determine if your game has affected participant mathematics anxiety, after only a single 1-hour session over 2 weeks would at best, equate to unreliable results. So much could happen outside of that 1-hour session that affects mathematics anxiety (for example mathematics exams, coursework, quizzes etc.). Your best bet, in this case, would be to encourage participants to talk about how anxious they feel as they come across mathematics problems in-game and why, or use some kind bio-feedback device to measure anxiety levels as they are playing.

For interviews, this where you really get to delve deep into why's and how's of the user's behaviour. A semi-structured interview should help in this case. The number of questions isn't that important, as long they answer your research aims and objectives, and are open-ended to allow for discussion. Playback gameplay recordings to the user if necessary. This will help triangulate interview responses with gameplay responses so the user can recall what actions they took and how they felt at the time. If you're looking at traits you could ask participants if they feel the game overall has had any effect on their anxiety about mathematics.

Depending on the number of research aims and objectives you have, and the loquaciousness of participants the interview could be done in 10-30mins.

In conclusion

The way you collect UX data for GBL titles in a limited time-frame depends on your research aims and objectives as well as the game in question. Going for a more qualitatively approach to increase the depth of your data is the safest route. Be aware that qualitative data can take a lot longer to analyse, however the results can help you better understand your users, their background, motives, how they think, allowing for more informed design decisions.

#gamebasedlearning #mathematicsgames #userexperience #qualitativedata #quantitativedata #scales #traits #mathematicsanxiety #UX #userbehaviour #datacollection