GDC 2015 – Delayed Impressions – Day Two – Games User Research Everywhere!

Yesterday I talked about my first impressions of GDC15 on days zero and one. On day two it was time to get up early and get a bus out to the Games User Research Summit held in Sony’s impressive offices in San Mateo. The Games User Research Summit is an annual opportunity for those interested in Games User Research to gather, network, and share their experiences (assuming they can make it to San Francisco that is). The summit was well attended and organised and included two tracks. As such, I wasn’t able to see everything, but I did manage to catch a bunch of interesting talks. I should also say that all the photos in this blog post were sourced from the official photos on Flickr and were taken by Tiffany Young:

That is it, I am out!

Time to rage?

  • First up was a welcoming speech from the appropriately purple clothed Jordan Lynn (Games User Researcher at Volition and an organiser of the summit), which was to the point and well put. Then it was on to Rage Quitting: Predictors and Long Term Impacts by Elizabeth Schmidlin from Sony. Elizabeth shared the results of a large scale survey she had carried out among undergrads at a Texas University looking at reasons behind rage quitting. Some of the headline findings were that men were more likely, even when controlling for exposure, to report raging in FPS games, whereas women were more likely (again when controlling for exposure) to report raging in racing games. The biggest determination of rage quitting due to in-game features was reported as players feeling like they were not getting enough out of game compared to the effect they were putting into it. Followed up by getting stuck on puzzles. Whereas, in terms of more technical features a game feeling like it wasn’t responding as expected was the other most commonly cited reason for raging. Finally, social features could also lead to raging with the three most important features mentioned (tied in terms of importance) were another player being overly aggressive, another player cheating, or another player being perceived as just too good.
I couldn't not make that pun



  • UX CYA FTW – Triangulation Flexibility & Integration: Making Bioshock Series Awesome by Bill Gardner from Xperienced Points. In this talk Bill covered his time at Irrational Games and how Irrational handled user experience (UX). Going from System Shock (before he joined) all the way to Bioshock Infinite. Interesting titbits were that 8 months out from shipping the game an external playtest was run, only to find that players were scoring the game three out of ten and reporting that they had no idea who they were, who the robots were (they had no good understanding of the Big Daddies), and why they were in the ocean. Players, apparently even referred to the game as a cheap Half-Life 2 rip-off. Based on this feedback the plane crash was added and the introduction to the Big Daddies that is in the game now was also added. Similarly, while there was more focus on testing earlier in Bioshock Infinite the introduction was initially not hitting with the players who found there was no sense of urgency. Based on this the note and corpse were added to the Lighthouse, therefore increasing the feeling of tension at the start of the game. Interesting stuff. He also discussed internal arguments over keeping the controls from Bioshock or moving to an Aim Down Sights model. These control methods were apparently playtested and it was found that while the aim down sights model didn’t impact on vigour use (a predicted outcome) it did lead to people standing still and aiming more often, an outcome the developers were not happy with. Finally, Bill discussed the Skylines in Bioshock Infinite as being an exciting feature, but one that players struggled with due to the high cognitive demands of maintaining control, shooting, and navigating the skylines at the same time. As such, a Skyline Playground zone was built to test with and allow fine tuning of the mechanic. In the end he wrapped up by stressing the importance of user research at developers, recommending starting with expert reviews and internal playtests, and then moving to external playtests (but not waiting too long for this).
Much panels...


  • Recruiting Tactics for a Modern Audience and Lab Space Design these panel sessions were a joy for a methodology nerd such as myself. In them the different recruiting methods for getting participants for user research were discussed and the hassles and victories that can come with setting state of the art user research facilities were examined. Hint, phone screen players to make sure they have the knowledge and experience that you need, and consider the noise that can come over cable trunking. Oh and OBS was rightfully unanimously praised as being awesome.
Insert Zombie Joke Here

Brains! And flowers

  • More Brains: GUR in PvZ Garden Warfare by Veronica Zammitto from EA. The message from Games User Researchers is, rightly, test early and test often. Do it, all the time. Playtest. Please. As soon as possible. Like when you have paper and/or digital prototypes. But that doesn’t mean that there is no value in testing later in production. In this talk Veronica focused on the user research carried out in the last 6 months before the launch of PvZ (with playtests being run as late as two weeks before launch). This testing was mainly focused on map and class balance and Veronica stressed the importance of using large sample sizes, with longer play sessions and a mixture of repeated quantitative and qualitative measures across a long play session. Asking players as a day of play went on how motivated they were to continue playing, how engaging the game was, and how much enjoyment they were getting as well as looking at their behaviour. She also stressed that user research at this time should be about fine turning and needs to be focused to address issues that can be fixed relatively quickly.
Spell casting gesture number 1

Subjective baddies

  • Know Your Enemy – Combatants in Destiny by Jennifer Ash from Bungie. In this talk Jennifer discussed the work she had carried out on Destiny focused on the combatants in the game. She outlined the early work where there was a focus on players evaluating concept art and looking at images that were presented to them rapidly (for about 5 seconds) to see if enemies could be spotted amongst various backgrounds and if their design gave appropriate indications of their threat and faction/race affiliations. There was also a focus on how enemy weapons looked, the threat they communicated, and on what abilities and weakness players expected different enemy types to have. Jennifer then discussed some of the subjective measures used to gain deeper understanding of player perceptions. Many of which she said were tracked over year long periods, allowing comparisons to be made across tests as the game developed. These including general questions about how combatants behaved and interacted with the player, indications of how dangerous enemies were, and how players approached bosses and how satisfactory they were to fight. She also discussed the objective data that was collected, such as event based data on when players were killed, how, by what, etc. One interesting case study from this was the “Shank Flank” where it was found that the low powered Shank enemies in the game were getting too many kills. This was, they found, because Shanks would spawn behind the player and stay there. Then due to their low damage players weren’t noticing and Shanks were finishing the players off when they were engaged with more powerful enemies that they could see. This was fixed by changing the Shank spawn points (World Design) and changing their behaviour to attack from the front (AI).
  • Destiny User Research Q&A featuring Jennifer Ash, John Hopson, and Nick Hillyer from Bungie. In this panel three user researchers from Bungie faced a barrage of questions (I may have snuck in two…) about their processes and various factors in Destiny including, but not limited to, the Cryptarch, loot drop balancing, class balancing, colour blindless, simulation sickness, Phogoth’s tummy, the move to aim down sights, the surprise of how the community reacted to the Iron Banner, and how they work in terms of carrying out user research within Bungie.
  • User Profiling by Anders Drachen from Aalborg University. Anders gave a comprehensive presentation on how to start to deal with the masses of player data that is often being collected. Specifically, he outlined four steps: 1. Knowledge discovery, 2. Selection, 3. Interpretation, and 4. Application. He stressed that you must go through these steps and that it was cyclic process as player behaviour changes over time, as do games, which echoes into the data. He then discussed player profiling, mentioning bottom up profiling (or “explorative profiling”) where you work from data coming in, which can lead you open to biases. The alternative is top down profiling, where you go out with specific hypothesised profiles and see if they work with the data. Anders advocated for the idea of building “protean” profiles from day 1 (e.g. persona like profiles) and then following players through the whole life cycle of play (from start, until they abandon the game). Anders also made sure to point out that data profiling is not an objective process. This is because as a profiler you have so many different choices to make about processing data and what you chose will change the outcome. He also stressed that you can get pretty far even with just descriptive statistics and segmentation of player groups. Anders then went on to cover some other methods such as clustering. All and all his talk was jam packed with details.

At this point, my iPad, on which I was taking all my notes crashed. As such, from this point on (and for the rest of GDC) I am pretty much relying on my memory. So, if there are less details it is not that the talk wasn’t interesting, it is just that memory is not as good as note taking (not even close).

Spell casting gesture number 2

Dragon Aging

  • Dissecting the Dragon: UXR for Dragon Age: Inquisition by James Berg from EA. For this talk James took us through some of the challenges and situations that came up when carrying out user experience research on Dragon Age: Inquisition. These included working on a good plan to make sure a diverse and LGBT recruit would be carried out, and then finding that due to Bioware’s relatively inclusive history in this area that a normal recruit could just be run and would result in a diverse crowd coming in to test the game anyway. James also stressed the difficulty of carrying out user research in such a big, open game – which was also still under development so constantly changing. He cited one specific example where a player spent three hours just wandering around in featureless, content-less mountains and generally having a great time. Great for that type of player, but not so useful if the playtest session was to test out class balance or some other mechanical aspect of the game. Much like Bungie with Destiny, concept art was also tested in Dragon Age. In this case they looked at how recongisable the characters where, romantic interest, enemy threat, and so on. James also covered how DA:I originally had a Skyrim like compass navigation system, but users just couldn’t use it, leading to the (much, much better in my opinion) system that is in the game at the moment with the bottom mounted radar. Other interesting nuggets of information were that about 25% of players use tactical mode all the time, with most others only using it where necessary, that bears were the bane (and ultimate opponent) of many playtesters, and for a long time Solas had text to speech for his lines, which some players thought suited him.
Voted best t-shirt of the day


  • After a little bit of housekeeping from David Tisserand (current head of the IGDA Games User Research SIG and researcher at Ubisoft), it was time for the keynote UX Invaders: We come in peace! How we can collaborate with developers to frame “fun” by Celia Hodent from Epic Games. She had the best t-shirt of the day and discussed different ways to approach user research and getting developers on-side. This started off by addressing common misconceptions, like user research is just “common sense” (no such thing, this is hindsight bias!) and that the claim that Dark Souls shows that user research isn’t needed misunderstands the process. User research isn’t about making games easy, it is about making the best games possible for a certain target audience. As such carrying out user research on Dark Souls would be entirely possible, you just have to consider the audience, and in this case make sure the game was punishing enough for them. Ultimately, Celia was stressing that user experience is not just the concern of user researchers and user interface designers, but that it is about the whole studio achieving their goal. A sentiment that you just can’t disagree with. She then went into her usability plus game flow = good user experience model. To do so she went through the heuristics she uses for usability (Signs & Feedback, Clarity, Form Follows Function, Consistency, Minimum Workload, Error Prevention & Recovery, and Flexibility) and for game flow (Pacing, Motivation, and Emotion). Celia provided clear examples for all of these, calling mostly on the work that she has been carrying out at Epic on Fortnite. The talk in general reminded me of listening to a lecture in Human Factors class, and I mean that in a good way (I know some people had bad experiences at University, but I loved it and had great lecturers). Celia finished up her talk by outlining four steps for collaborating with developers in user research: 1. Debunk UX misconceptions, 2. Insist on testing early, even with paper prototypes to demonstrate early, easy, wins, 3. Come up with a “model” (her use of quotes, not mine) that can be communicated and understood by developers in order to get buy in, and 4. Put user experience at the centre of the studio, as mentioned earlier it is up to everyone, not just the user research team.

After that it was time to get back on the buses, head back to SF, and to party until late with the wonderful user research community. Good times were had.

Tomorrow, Hearthstone, Clash of Clans, (some more) Destiny, and how to survive harassment.

You may also like...

  • Did anyone else feel the compulsion to stand and give the vulcan salute when Celia showed a picture of Nimoy?