GlassLab, iCivics and Filament will connect the game “Argument Wars” to a major new assessment tool later this month.
Everybody knows a well-designed game can engage, but it’s their ability to generate data – a lot of data – about game play and learning that has emerged as a sort-of Holy Grail for educational game developers.
The result has been a quest of sorts by companies to connect their games to some sort of assessment, be it an assessment of student knowledge delivered to a teacher or a feedback loop to game producers about the effectiveness of the game.
And like the hunt for the Holy Grail, most serious learning game developers will tell you it has been going on for a while.
“The question around assessment and games have been circling for years and they still have not come in for a landing,” said Lee Wilson, CEO of Filament Games.
But Wilson, along with his partners at iCivics and GlassLab, hope a new effort to field test assessment tools may help them all get a bit closer to the grail.
iCivics Executive Director Louise Dubé said later this month the suite of online games focused on government will conduct a field test of 1,000 students playing their game “Argument Wars.” But unlike the millions of plays that have come before this time game plays will pour data into the sophisticated assessment engine developed by the GlassLab project.
In the effort, students will be tested of their knowledge of key Common Core standards around reasoning. Then data will be gathered during their play of the game. A second post-game test will be administered and then the experts will examine whether the game’s assessment of a student’s performance usually predicted their ability to meet the standard.
What they (teachers) tell us all the time is, ‘Can I use the score from the game to rate the students?’ And we didn’t think we could make that leap with the data we had before. Our hope is now we can answer that.
iCivics Executive Director Louise Dubé
The Assessment Engine of Change?
Even the most basic skill-and-drill game has some sort of score the player receives at the end of it, but the field of assessment has moved far beyond that.
GlassLab, a game development and research group based in California, has spent countless hours delving into assessments that can scientifically be proven to gauge student learning within a game.
When Jessica Lindl, general manager of GlassLab, describes the effort to turn the enormous amounts of data into understandable measures it sounds almost like an NSA project.
“[W]e are tracking every time a student is hovering a mouse over a specific object, every time they are clicking on something, every choice, decision and action they make within the game to be able to solve that problem,” she said when their first game SimCity EDU launched.
These thousands of data points per player per game-play have to be transformed into something a teacher can use, Lindl added, saying “That’s for me where the real art happens… is how you translate that into a couple of key data points for the teacher that are actionable and meaningful?”
The result was an assessment engine that could create data for the player, the teacher and the game designer.
It’s a development that appears in line with what top policy makers are saying in Washington. Just this week, Undersecretary of Education Ted Mitchell told developers at a White House Game Jam, “We need skills-based assessments that are genuine to the work that students are doing.”
And it was one that Lindl and her colleagues wanted to test more. So earlier this year GlassLab announced they would be working with iCivics and Filament to connect the engine to one of the most successful learning games out there. Some 40,000 teachers and 3 million students have used the iCivics games to tackle everything from the concepts of rights to the separation of powers to structuring a debate.
Lindl said her group jumped at the idea of working with iCivics.
“They already understood the importance of this data and were collecting their own, so we could have a real conversation about what the assessment could and should do,” she said.
Wilson adds it offered other critical things for each partner.
“The partnership allows GlassLab to go to scale very quickly, giving them access to an enormous user base. It allows iCivics to refine their games and future work with new and better data and it allows us to kick the times of the assessment engine and see how it affects our game designs,” he said.
Why does it matter?
So why has assessment become such a big deal to developers? Well, a lot of it has to do with the demands of the market. And this is a market with a lot of money.
“Games can be—and are—meaningful vehicles for learning. But for learning games to be used on a large scale in schools, games must not only help students learn, but help them learn things of value that can be assessed and reported to teachers and students in meaningful ways, “ Jane Canner of Classroom, Inc. has said.
The idea of games being not just a way to engage students but a method to ensure that certain skills or facts are learned has become one of the keys in the minds of most people for wider use of games in school.
“The potential has evolved from drill-based arithmetic games to games with sophisticated back-end data models that can help teach and assess all sorts of cognitive and even non-cognitive skills,” Robert Gehorsam, head of the Institute of Play, said. “This is, in part, a reflection of advances in the field of game design, but it’s also a reflection of a growing research base that will inform the way games are designed for learning.”
And for firms like Muzzy Lane Software, assessment could be a way for games to the move from the edge of the conversation about classroom learning to the core.
[N]obody questions whether games have a role in education any more. All the debate now is what’s the role and how do we take advantage of it. Assessment is a massive part of that discussion because everything is going online. Like it or not, auto-assessment is the Holy Grail because it relieves teachers of the burden of assessing things directly… It’s so much more efficient. Games give us the ability to bring these sorts of higher-level thinking, complex activities into that realm and I think that’s absolutely crucial to their adoption.
But over at Filament, Wilson cautions it may be some time before game developers can answer the truly core question that shapes so many school purchasing and technology decision.
“We are such a long way from having reliable conclusions and data to be able to answer the school administrator’s question, ‘will this game help the kids on the standardized test,’” he said, adding that is the lens through which schools make most decisions.
Still, Filament’s partner at iCivics sees the idea of assessment as central to the most critical struggle educational games face in the classroom… The quest for time.
“The whole industry of educational gaming still needs to capture more classroom time and one way we do that is by delivering formative assessments to teachers that help inform their teaching and help them understand where students are,” Dubé said.
Is Assessment about Teachers or Developers?
Based on the importance assessment could play in unlocking the lucrative, but slow-to-change school market it is easy to see why so much of the focus is on teachers and their access to the data generated by games, but more and more developers are talking about how these tools can improve their own work.
There are a couple reasons for this increasing interest and they come from two directions: the difficulty turning reams of data into easily consumed teacher dashboards and the power of these data streams to improve the game and future game designs.
First on the teachers. Teachers report an increasing use of assessment from games to improve and modify their classroom plans.
According to numbers released in July by the Cooney Center, fully 43 percent of teachers using games in the classroom report using the built-in assessments that come with games. That said, interestingly, almost a quarter of the teachers report not assessing student performance connected to a digital game in any way.
At the time, GlassLab’s Lindl said that did not really reflect what happened when they released their first game.
“What we have seen is dramatically different to the point that when we launched SimCityEDU we sort of walked away saying we have to totally rethink this and then invested a lot in reworking the dashboard for assessment data,” she said. “We are seeing more teachers look at it, but nothing near 50 percent. …. I worry some teachers may be looking at game-play data and think they are looking at assessment data, but the two are different.”
Wilson and others agree that there may be some definitional problem around assessment and what it is exactly.
You can have a teacher walking around behind kids in the class and checking in on whether they completed the game and that is a sort of assessment. Or teachers may look at a quiz score at the end of a game and connect that to assessment, but that is not what most of us mean.
Lee Wilson, CEO, Filament Games
Dubé added that what it really requires is to test and re-test a students ability to master a specific skill within a game and that may have very little to do with winning or losing the game.
“To create an inference that allows us to say a student has a certain skill we need to touch it multiple times and in multiple ways, to create more data points so you have enough information to make the assessment,” she said, adding, “You need … a lot of data to do that.”
And that is where the conversation about assessment gets a bit weird.
After all is said and done, most of the people we have spoken with in the last year have stressed the importance of assessment is not really for the student, teacher or parent, but for the developer.
“Generally, Filament’s thinking is the real benefits, initially, from the assessment work will be for the game design and game mechanic,” Wilson said. “There’s a rich telemetry of data produced by this assessment, but that it’s game design and instructional designers who may learn the most from what we find out.”
Others strongly agree with this finding. Derek Lomas over at PlayPower said he hopes his company can thrive in the learning game space because they incorporate assessment into their products so they can somewhat auto-improve as they are played.
“The idea is we don’t know what is going to work best so we want to try out a bunch of different things,” Lomas explained. “Instead of building in a bunch of experiments and making changes we can build in the experiments and create the changes automatically.”
But Is It a Business?
GlassLab Game Services aims to provide tools to allow for the wider use of games in school.
It is this mix of powers – to increase the marketability to schools, to demonstrate the effectiveness of their game to actually teach and to improve their game design continuously – that makes assessment the Holy Grail.
And GlassLab may be positioning itself as a unique service in that quest. While still continuing their work developing and deploying games, it is this service to other developers that may be central to their business strategy.
GlassLab is set to announce a new set of firms to serve in an active beta, expanding the work started with iCivics and Filament and Lindl said their hope is to create a fee-for-service model in the coming months.
The alpha partners both agree it is a particular set of skills the Lab offers.
As Wilson put it, “I don’t believe any of our current clients have the resources , all of them have the needs for it but not the skills.”
And Dubé added that the Lab has “skills we couldn’t hope to build with a small team like ours. They have assessment experts and psychometricians on staff full time.”
RSS Feed of News Items