Setting Guardrails and Containing an Increase in Cloud Costs

Problem

In recent years, a great number of companies are worried about their increasing cloud costs. Cloud costs impact their revenue and are usually correlated with the development headcount. Furthermore, an increase in cloud cost is not linear but exponential and can only be matched with the exponential significance of the cloud itself.

Without a structure in place that will monitor the usage and efficiency of cloud costs, developers will be left on their own to spend dizzying amounts of money on it.

Actions taken

First off, it is crucial to identify all the risks. For every contract signed, you should identify what are the levers given to your developers.

Then, create metrics that would monitor usage and efficiency and that would be easy to understand and track. To make it easier for your developers to see if they are overspending, you should create benchmarks that are comparable. While there are many metrics monitoring usage they rarely tell you if you are being efficient. Adding efficiency metrics (for example, how much I am spending serving a million users per day) will help you optimize your costs.

Allocating your costs and improving your attribution is a necessary next step. You need to granularly attribute costs to the systems you are running and more importantly, attribute them to specific teams.

Once you are cognizant of all costs and have solid metrics that can help you track the costs, you should put in place guardrails or governance rules to ensure that teams can’t create excessive or untraceable costs by mistake or accident. These guardrails should include blocking developers’ work and setting the alarm in most dramatic cases.

To secure buy-in from the leadership frame metrics in a way that the business can understand or Finance can link to headcount. For example, how much do we need to spend per data scientist in cloud costs just for analysis or how much the business spends to process each terabyte knowing that the data you generate is proportional to your user base.

In the end, you will need someone to execute on this. I have created a Cloud Economic team, and if you are spending a couple of million dollars on cloud costs it is worth having someone dealing with it.

Lessons learned

• As the company scales your ability to control all the costs will be diluted and they will keep growing exponentially. If you don’t put guardrails before you start to scale, you will suffer financial loss.
• If you don’t come up with access policies in terms of the use of resources, engineers will come up with their own. While you may have some idea what is going on with 20 or 30 engineers around, with a couple of hundreds you will have no clue about the costs.
• The nature of cloud costs is that the development costs should be proportionate to headcounts. However, the organizational complexity entails more and more infrastructure to have an efficient development and you will have less control over what is being created. This is when the costs start to increase exponentially.