Using Incidents to Learn About Development Mistakes

Problem

As your company begins to grow so shall your development issues. You will have more complex problems and a larger amount of failed scenarios that you will need to look into. The fact is things are going to break. So the best place to begin tackling this concern is through the incident process. Here are the first steps to take to set up your own process.

Actions taken

The best way to know what is going wrong is to let things happen and catch them in action. Set up a process that defines the following:

• how to report an incident once it is found
• whether it needs rushed priority or not
• how to communicate to the rest of the company and customers what is going on
• how to report the results of the incident investigation such as lessons learned and the root cause
• who and what needs to be done to fix the problem for the long-term This process can be as simple or as complex as you need. I have seen an incident analysis that contains 155 questions while another has only 5. In the end, both answer the question of why it happened and identify how to solve the root cause. Two positions can assist with fixing your incidents. First, devops. They have a direct relationship with the incident process. The core of devops is observability, being able to catch things. Thus, it's not about avoiding something happening but making sure that you comprehend what happened and doing something about it. The second role is QA. They can really help you understand what went wrong and how you can avoid it happening again in the future. This is a large piece of the development picture, being able to sidestep the problem the next time it occurs, and QA usually has strong insight into the correction of error

Lessons learned

Never truly accept the fact that the system is broke. Usually it is human error that has caused the mistake. Don't go around announcing who made that error, or even the role that they are in. Instead, simply focus your attention on the problem and the solution.