©2018 by Blue Learner | Sheffield, UK

Changing my PhD methodology (feedback welcome)

August 6, 2018

Recently, I conducted a pilot study for my PhD using a methodology that looked rather robust (on paper). To measure mathematics anxiety in university students and the effect educational computer games have on the phenomenon, the 3 weeks of data collection would go like this:

 

Pilot study methodology

  • Invite students across the university via email to take part in the study (sample size of 30 would be adequate for the pilot study)

  • Use an online form to gather participant demographic info and mathematics anxiety levels, using an initial mathematics anxiety rating scale (MARS)

  • Participants are randomly split into a control group and experimental group.

  • An experimental group (n=15) play this online mathematics game for as long as they like over 3 weeks while on their course, as long as they time themselves playing.

    • Total time playing would be recorded as a factor affecting mathematics anxiety

  • A control group (n=15) continue their course without the game.

  • Participants from the experimental group are invited to take part in a 1 hour long playtesting observation and interview. 

    • Keystrokes, mouse movement/clicks, facial reactions, gameplay and think aloud recorded. 

  • At the end of the study, all participants take the same mathematics anxiety scale, allowing the researcher to compare mathematics anxiety scores before and after playing the game, as well as between the control group and experimental group.

Well, then this happened...

 

 

Unfortunately, this approach didn't quite go according to plan. Only 12 participants decided to take part in the study. On average, they only played the game for half an hour or less over the 3 week period. Only 5 participants took part in the playtesting observations/interviews. Only 2 participants took the final mathematics anxiety scale, making it to the end of the study. 

 

Possible reasons why it didn't work

 

It seems a more rigorous approach to recruiting and retaining participants is needed.

 

  • Firstly, it was near the exam period when invitations to take part in the study were sent. Students were busy with revising or submitting final coursework assignments and so had to prioritize. 

  • 6 students stated that they didn't know they had to play the game during the 3 week period.

  • 5 students forgot the study existed.

 

Ideas for improving methodology (for main study)

 

 

 

Recruitment 

 

A more engaging way to recruit students is needed. As such lecturers will be instructed to encourage students to take part in their study. Having more of a authoritative source to promote the study can enhance incentive to take part. Students will also be approached via Twitter and Facebook, as well as mathematics study groups such as MASH where students are most likely to take part in the study due to higher levels of concern about their mathematics ability. 

 

 

 

 

 

 

It seems a diary study could be a way to entice students to stay on until the end. Diary studies allow participants to record their thoughts, feelings and frustrations over long periods time. To encourage daily entry, participants will be sent an email reminder at the end of each work day to play the game for 10mins and fill in the diary entry form. This has two advantages:  

  • The researcher can acquire more real-time behavior of participants during the data collection period (not just the one playtesting session).

  • Open ended questions can help the participant highlight other potential sources of mathematics anxiety. 

  • Can monitor and analyse data as it's collected by question participants about their responses while they're still fresh in their mind.

There are a couple of disadvantages too: 

 

  • Lots and lots of data to analyse (perhaps too much)

  • Harder to implement and monitor with so many responses at once.

  • With participants having to commit more time to the study, they should be compensated, leading to increased costs.

 

Nevertheless the study looks more like this: 

 

Main study methodology

 

 

  • Invite students across the university via email, social media, and mathematics help groups to take part in the study (sample size of 360+ would be adequate for the main study for a 95% confidence interval)

    • This should take place at the start of the year where students aren't so busy with assignments/exams.

  • Use an online form to gather participant demographic info and mathematics anxiety levels, using an initial mathematics anxiety rating scale (MARS)

  • Participants are randomly split into a control group and experimental group.

  • The experimental group are provided with an online diary entry form, requiring answers to questions such as (have they played the game today, how did playing make them feel, as well as how anxious they are about mathematics and why (likert scale and open ended questions).

  • The experimental group (n=180) play this online mathematics game for 10 minutes every weekday over the 3 weeks while on their course. They then fill out the diary entry.

    • Total time playing would be recorded as a factor affecting mathematics anxiety.

  • A control group (n=180) continue their course without the game.

  • 30 participants from the experimental group are invited to take part in a 1 hour long play-testing observation and interview (sample size could be less depending on how quickly data reaches saturation point).

    • Keystrokes, mouse movement/clicks, facial reactions, gameplay and think aloud recorded. 

  • At the end of the study, all participants take the same mathematics anxiety scale, allowing the researcher to compare mathematics anxiety scores before and after playing the game, as well as between the control group and experimental group.

 

It's expected that some students may still have gaps in participation or even drop out of the study. In this case, being able to identify any trends between time spent playing the game and mathematics anxiety levels after the data collection would be beneficial.

 

What do you think? Will this new data collection method engage participants or will they silently escape like before. If the latter, what could be improved? Feel free to leave your comments below.

Please reload

Our Recent Posts

Please reload

Archive

Please reload

Tags