Games and Learning

Developing The Radix Endeavor, our multi-player online game for STEM learning, at the MIT Education Arcade meant working with lots of different groups. My colleague Susannah Gordon-Messer recently reflected on our experience collaborating with teachers, but the input of their students was equally, if not more, informative.

Over ten months, we worked with over 100 students in our target grade range, in a variety of educational settings. We all know prototype testing is important, but getting actionable feedback that addresses issues at the heart of your game is no simple task.

Beyond the basics you most likely already know, the following are six strategies that represent our top tips to ensure you get the most out of user testing:

1. Create a partnership

While some of our testing was conducted ad hoc, we formed ongoing relationships with a number of schools as well as formal and informal afterschool programs. This meant that a few of the student groups we worked with participated in playtesting several times. For some groups this meant interacting with the breadth of content in the game. Others saw one particular quest evolve over time from paper to digital prototype to final gameplay.

This allowed the students to feel like they were a valued part of the design process. More importantly, over time they got used to the types of feedback we were looking for. They became more skilled at reflecting deeply on their experiences and verbalizing their opinions.

2. Test early, design incrementally

Rather than waiting until we had a fully formed concept of the game to conduct playtesting, we worked with students very early on to help us think through how certain mechanics could potentially work for a specific purpose.

We conducted a few rounds of play testing using paper prototypes and then a few more using crude digital prototypes. This gave us a foundation upon which to build the tools and quests because we already had an idea of what students would expect to be able to do in the game, or what approach they were likely to take.

CALLOUT: Strategy 2 in action

Paper-based versions of the algebra marketplace enabled players to tweak the rules and challenges of the quests as they played, in a way digital prototypes don’t allow for. Letting vendors add their own deals and letting traders haggle over prices showed us that the amount of different goods available for trade wasn’t as important as the sense of agency in bartering. Based on players’ expectations we designed for more open-ended negotiations and added the opportunity for players to choose their own trading goals in certain quests.

Developers started testing with paper prototypes long before the Radix Marketplace feature took its final form.

3. Observe teachers observing their students

Directly observing students’ facial expressions, body language, clicks, and remarks can tell you a lot about the game’s playability. However, they don’t tell the whole story about what students are getting out of an educational game. By having teachers participate, we got to see how teachers interacted with their students to support their learning, how they helped them problem solve, and connections they made to prior knowledge.

We also got to hear teachers’ “behind the scenes” notes on which students were more engaged or persistent than in other classroom activities. This gave us a bigger picture of how Radix would fit into a classroom and what was needed to support the core game mechanics.

4. Bring the dev team

In service of efficiency, we often delineate roles in a hard and fast way. This puts us in a position of having the outreach or design team translating player feedback to the development team, and important things can get lost in translation. When developers of a particular feature witness playtests firsthand, they can more clearly envision why a particular mechanic was frustrating for users and are more likely to come up with a good solution. Developers carry this experience with them throughout the development process, greatly improving both their understanding of the team’s goals and their contributions to the project.

5. Ask thought-provoking questions

Asking for general feedback, or even questions such as “What did you like about the game?” doesn’t always elicit a response that fully depicts the experience. Students often tend to mention one or two things then stop, or simply talk about what they think you want to hear about. To go deeper, it helps to craft more targeted questions that direct a students’ thinking or frame it in a different way.

CALLOUT : Strategy 5 in action

To get at whether a genetics quest was too difficult, rather than asking, “What was hard about this quest?” we might ask something like, “What strategies did you use to breed the right bugs?” This way we are more likely to get specifics about what the player figured out, and whether they knew it right away or had to try lots of things before solving the problem. From there, as a designer we can gauge whether the level of challenge was appropriate and adjust the task as needed.

Critical to good playtesting, the Radix team found, was asking specific questions about gameplay and challenges.

6. Don’t listen to your playtesters

Of course, you should actually listen to your playtesters. Students are the experts at how other students will use and react to your game, and designers playtest in order to improve all aspects of their product. But, sometimes you make design decisions in order to move the needle in a certain area and if that is outside users’ comfort zones, they are not likely to give rave reviews right away. In these cases, if you’ve carefully analyzed their feedback and your intentions, you may decide that a negative reaction is in fact a positive sign.

CALLOUT Strategy 6 in Action

One of the goals of Radix is to provide opportunities for exploration and inquiry. This means we want students to have to figure out how to solve a problem. In playtesting evolution quests, there were many students who said they had a hard time because they didn’t know what trait they were looking for and suggested adding more instructions. Some teachers also gave similar feedback.

However, we felt that these expectations were a side effect of traditional school-based activities. Keeping the activity less directed was key to creating the space for students to make their own discoveries, so we chose not to do everything that would make it smoother for the player.