Gamedays are really important for us when it comes to building and maintaining digital solutions. It’s a collaborative learning exercise that helps simulate and test what might happen in the case of a security or major incident situation.
The term ‘gameday’ was coined by a former Amazon employee, Jesse Robbins, who wanted to improve the resilience of systems by putting them under stress through regular simulations. Many companies use similar gamification exercises to increase their system’s reliability, including Google, Netflix and Facebook.
So what is a gameday exactly?
A gameday involves the role-playing of scenarios, designed to train the team in dealing with real-world unplanned events.
To ensure it’s as close to the real thing as possible, our gamedays run for two to three hours throughout the day and alongside normal work activity. It involves three/four scenarios of differing severity based on previous real-world incidents, which makes them both relevant and realistic.
Gamedays are ideal for simulating what could go wrong
Gamedays are a bit like running a fire drill. During an emergency, it can be easy to panic and not follow the correct processes. By creating a safe environment to practice, we check that all systems, processes and services respond as expected. Just as important is the human aspect; these simulations ensure the teams are capable and confident in their roles and tasks if a real-life emergency occurs.
What are the benefits of running a gameday?
Gamedays are ideal for simulating what could go wrong — whether that is a single point of failure or a major incident. It ensures everyone understands the whole process, rather than just the parts they deal with.
The benefits of running these gamedays include resilience testing, delving deeper into where the knowledge gaps are, and highlighting dependencies and ownership. For the client, it gives them confidence that the teams and systems involved can respond quickly and efficiently in real time.
Failure is normal on gamedays. By testing how and where it occurs, we prepare and have a plan in place if the worst was to happen.
An example of a recent gameday
At a recent gameday for a client, we simulated all the different roles and played the day out exactly how we thought it would be on ‘go live’ day. This made the day more exciting and got people engaged.
We encountered a few areas that required more work. For example, some users couldn’t log in — an issue that we should’ve addressed earlier. But this is precisely what gamedays are about — to find gaps that naturally occur in projects.
The developers and engineers who build the product don’t always see it in action. This was a brilliant way to get them involved in the product as a user. It allowed them to empathise and understand the needs of the user and the client.
Ultimately, the gameday gave the team and our clients more confidence in the platform.
What has client feedback been like?
Most of our clients have never heard of gamedays and so they’ve found it very interesting and useful. The gameday itself is clearly valuable in addressing any technical issues.
But, as ever with these things, it’s the ‘doing of the thing’ that provides the real value. By getting different teams, both from the client and supplier side, to work together on the activity, you create a shared understanding. And this makes communication between the teams and stakeholders, and the product, much better.
We run retros at the end of the day to get everyone’s feedback. Here are just some of the comments from another recent gameday.
The retro acts as a debrief for the day and helps us plan improvements and future exercises. These gamedays serve as evidence of continuous improvement and ensure all aspects of the client’s systems are up to date, working and prepared for any unexpected changes.