Back to resources

Telemetry in the QA Process

Innovation / Experiment
QA Team
Team Reaction
Team Processes

13 October, 2021

Phillip Derosa
Phillip Derosa

Global Director of QA at OneSpan

Phillip Derosa, Global Director of QA at OneSpan, lives to know what his players are getting into as they explore every corner of the worlds created for them.

Problem

When people play games nowadays, we measure everything: the characters that they select, how many times that they’ve died, all of that stuff. This enables us to make better decisions as we conduct playtests -- we can hone in on a level where more people are dying more frequently, to name one example. The boss is too hard. There are a million possible problems.

I was at an industry show and saw some stuff that got me thinking a lot about telemetry in gaming. This was before telemetry was really a thing in our industry. Some people were unsure about it at first.

Actions taken

I ended up asking two of my QA folks to go through one of our games and to keep track of various stats, such as how many times that they died throughout the level. Instead of capturing events and dumping them out to a log, I asked them to count each tally manually on an Excel sheet by hand.

They played the whole game through and counted how many conversations and combat encounters that they had, among other things. We generated a nice report that outlined all of this information and sent it back to the Director of Design.

They were ecstatic; they described it as being like a window into the game that they had never had before. It was objective data that they would be able to rely on. They asked for more.

It was not practical to have human beings with stopwatches document all of these things, which was when introducing telemetry into the fold started to interest me greatly. Building the technology into our game gave us so much more to work with. Our developers were super excited about what we were showing them.

The guy who got me into telemetry at the conference was actually a programmer themselves; they ended up working with us as we implemented it into our work and process. Two or three of our own Engineers took these insights back to their desks; after a couple of days, they had come up with a prototype for a new system.

The technology was cool, and it was giving everybody the thirst. Within a few weeks, we had all of this new stuff, all of this extra data at our fingertips. This new source of information taught us so much about our own gameplay that we would be able to refer to later on.

Lessons learned

  • Once I was able to show our Director of Design the value of data like this, there was no turning back. Some people rely on what their gut tells them when deciding whether or not they’re going in the right direction. It’s good to have this type of intuition, but nothing beats real data.
  • When workshopping toys based on kids’ shows, R&D teams will do something similar. They will have some kids watch a show in a room full of toys and when the kids lose interest in the show to play with the toys, they take note of the time to review the show's content and adjust it. It’s low-tech, but it’s still first-hand feedback from a real user of the product.
  • Sometimes, you get a wall put in front of you. The goal for me is to make the best game possible, which is where I find the energy to continuously push forward. Nothing should be “just a QA thing”; everybody should feel some responsibility.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

The Art of Asking Why: Narrowing the Gap Between Customers and Users

24 May

Jord Sips, Senior Product Manager at Mews, shares his expertise on a common challenge for product managers – finding root causes and solutions.

Customers
Innovation / Experiment
Product
Personal Growth
Leadership
Stakeholders
Users
Jord Sips

Jord Sips

Senior Product Manager at Mews

Streamlining Product Processes After a Reorganization

16 May

Snehal Shaha, Lead Technical Program Manager at Momentive (fka SurveyMonkey), details her short-term technical strategy to unify processes among teams following an acquisition.

Acquisition / Integration
Product Team
Product
Building A Team
Leadership
Internal Communication
Collaboration
Reorganization
Strategy
Team Processes
Cross-Functional Collaboration
Snehal Shaha

Snehal Shaha

Senior EPM/TPM at Apple Inc.

Navigating Disagreements When It Comes to Priorities

9 May

Pavel Safarik, Head of Product at ROI Hunter, shares his insights on how to deal with disagreements about prioritization when building a product.

Innovation / Experiment
Product Team
Product
Dev Processes
Conflict Solving
Internal Communication
Collaboration
Convincing
Strategy
Prioritization
Pavel Safarik

Pavel Safarik

Head of Product at ROI Hunter

The Optimization and Organization of Large Scale Demand

4 May

Kamal Qadri, Senior Manager at FICO, drives the importance of setting expectations when optimizing large-scale requirements.

Managing Expectations
Delegate
Team Processes
Prioritization
Kamal Qadri

Kamal Qadri

Head of Software Quality Assurance at FICO

Why You Should Take Technology Risks in Product Development

25 April

Matias Pizarro, CTO and VP of Residents at ComunidadFeliz, recalls a time in his early career when he took a technology risk that had wide-ranging benefits to his product's user experience.

Innovation / Experiment
Product
Scaling Team
Dev Processes
Matias Pizarro

Matias Pizarro

CTO and VP of Residents at ComunidadFeliz

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.