Who Should Define Error Budgets in SRE-Led DevOps Teams?

In the world of Site Reliability Engineering (SRE), a core principle is the strategic management of a service’s reliability. The concept of an error budget is central to this philosophy. An error budget is a simple, yet powerful, tool that quantifies the amount of unreliability a service is allowed to have over a period of time without violating its Service Level Objective (SLO). In simpler terms, it's the agreed-upon tolerance for failure. This idea represents a major shift from the traditional mindset where 100% uptime was the often-unrealistic goal. An error budget provides a data-driven way to balance two competing forces: the business's desire for speed and innovation, and the SRE team's mandate for stability and reliability. When a service is performing well, the team can "spend" the error budget on new features, which often introduce some level of risk. When the budget is running low, the team must stop all feature development and focus solely on reliability work to prevent a larger incident. The question, then, is not whether to use error budgets but who should be responsible for defining and managing them? The answer is a collaborative effort that involves multiple stakeholders, which is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers and is a major part of a successful business that is looking to scale its operations.

Table of Contents

• What Are SLOs, SLAs, and Error Budgets?
• How Do Product Managers Fit In?
• What's the Role of the SRE Team?
• Who Are the Other Key Stakeholders?
• How Do They Balance Velocity and Reliability?
• A Comparison of Roles and Responsibilities
• What Is the Process for Defining an Error Budget?
• Conclusion
• Frequently Asked Questions

What Are SLOs, SLAs, and Error Budgets?

To understand an error budget, a team must first understand the concepts of Service Level Agreements (SLAs) and Service Level Objectives (SLOs). An SLA is a formal agreement between a service provider and a customer that defines a certain level of service. If the provider fails to meet the SLA, there are often financial penalties. An SLO is the internal, aspirational target for a service's reliability. It is a key input for the error budget. For example, a team might have an SLO of 99.9% uptime, which means the service can have up to 0.1% downtime over a given period of time. The error budget is the amount of downtime the service is allowed to have before it violates the SLO. It is a quantifiable measure of the "acceptable" unreliability. This is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers and is a major part of a successful business that is looking to scale its operations.

The Role of the Service Level Indicator (SLI)

Before defining an SLO, a team must first define a Service Level Indicator (SLI). An SLI is a quantitative measure of a service's reliability, such as latency or request success rate. It is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers.

How Do Product Managers Fit In?

Product managers are the voice of the customer and are responsible for defining the user experience. They are the ones who should initiate the conversation about error budgets. They must understand the user's tolerance for downtime and how much a user is willing to pay for a higher level of reliability. This is where a product manager's business acumen and user empathy are crucial. They must define what "acceptable" is from a user and a business perspective. For example, a product manager might determine that users are willing to tolerate 10 minutes of downtime per month for a free service, but they will not tolerate any downtime for a paid service. This is the key input for the SLO, which is the foundation for the error budget. A product manager's role is to ensure that the error budget is aligned with the business's goals and the customer's expectations, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

The Voice of the Customer

A product manager is the voice of the customer and is responsible for defining the user experience. They must understand the user's tolerance for downtime and how much a user is willing to pay for a higher level of reliability, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What's the Role of the SRE Team?

While product managers define the SLO, the Site Reliability Engineering (SRE) team is the one who provides the technical context and the data to show what is a realistic and achievable reliability target. They are the ones who are responsible for measuring the service's reliability and for monitoring the error budget. SREs provide the quantitative, data-driven side of the equation. They can show a team how the service is currently performing and what is a realistic SLO. They are also the ones who monitor the error budget and who sound the alarm when the budget is running low. When the error budget is exhausted, the SRE team is responsible for ensuring that the team stops all feature development and focuses solely on reliability work. Their role is to provide the technical expertise and the data that is necessary to make a data-driven decision about a service's reliability, which is a major part of a successful business that is looking to scale its operations and is a major part of the modern workflow.

The Technical Experts

SREs are the technical experts who translate the SLO into an actionable error budget. They provide the technical context and the data to show what is a realistic and achievable reliability target, which is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers.

Who Are the Other Key Stakeholders?

Defining an error budget is a collaborative effort that involves a wide variety of stakeholders. In addition to product managers and SREs, engineering leadership, such as the CTO and engineering managers, play a crucial role. They are the ones who must facilitate the negotiation and who must ensure that the error budget is taken seriously. They are responsible for enforcing the rule that when the error budget is exhausted, the team must stop feature development and focus on reliability work. Executive leadership, such as the CEO, is also a major part of the conversation. They must provide the vision and the strategic alignment for the error budget. They must understand the trade-offs between velocity and reliability and must be willing to make a data-driven decision about a service's reliability. The development team is also a key stakeholder. They are the ones who are responsible for building and for maintaining the service. They must understand the SLO and the error budget and must be willing to make a data-driven decision about their work, which is a major part of a successful business that is looking to scale its operations and is a major part of the modern workflow.

The Role of Engineering Leadership

Engineering leaders are the ones who must facilitate the negotiation and who must ensure that the error budget is taken seriously. They are responsible for enforcing the rule that when the error budget is exhausted, the team must stop feature development, which is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers.

How Do They Balance Velocity and Reliability?

The concept of an error budget is a tool to manage the tension between velocity and reliability. It gives a team the freedom to experiment and to innovate, knowing that they have a set amount of "unreliability" they can spend. This is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers. The budget is a simple, yet powerful, tool that helps a team to make a data-driven decision about their work. It provides a clear and consistent way to communicate the trade-offs between velocity and reliability. When the budget is running low, the team must focus on reliability work. When the budget is healthy, the team can focus on new features. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations and is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers.

The Role of Shared Responsibility

The error budget promotes a shared responsibility model, where the entire team is responsible for a service's reliability. It is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers and is a major part of a successful business that is looking to scale its operations.

A Comparison of Roles and Responsibilities

The following table provides a high-level comparison of the roles and responsibilities of the various stakeholders in defining and managing an error budget. It is designed to quickly illustrate the inherent limitations of the old approach and the corresponding strengths of the new one, making the value proposition of a modern approach readily apparent. By evaluating these factors, an organization can easily determine if they have reached the point where a traditional approach is no longer a viable or safe option for their business and is a major part of the strategic conversation that is needed for any organization that is looking to scale its operations.

What Is the Process for Defining an Error Budget?

The process for defining and negotiating an error budget is a collaborative effort. It begins with the product manager, who defines the user-facing SLO based on the business's goals and the customer's expectations. The SRE team then provides data on the service's current performance and proposes a realistic and achievable error budget. The stakeholders then negotiate and agree on the final SLO and error budget. The entire team, including the development team, must commit to a plan of action for when the budget is exhausted. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations and is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers.

The Role of Negotiation

The process of defining an error budget is a negotiation between the business's desire for new features and the SRE team's need for system reliability. It is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

Conclusion

The definition and management of an error budget is not a task for a single person or team. It is a collaborative process that requires a strong partnership between product managers, SREs, and engineering leadership. Product managers provide the business context and the user-facing SLO, SREs provide the technical expertise and the data, and engineering leadership provides the accountability and the strategic alignment. By working together, a team can use an error budget as a powerful tool to manage the tension between velocity and reliability, which is a major part of a successful business that is looking to scale its operations and is a major part of the modern workflow that is focused on providing a high level of service to the business and its customers.

Frequently Asked Questions

What is an error budget?

An error budget is a quantifiable amount of unreliability a service is allowed to have over a period of time without violating its Service Level Objective (SLO). It is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What is the difference between an SLO and an SLA?

An SLO is an internal, aspirational target for a service's reliability, while an SLA is a formal, external agreement with a customer that often includes financial penalties. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

Who defines the Service Level Objective (SLO)?

The Service Level Objective (SLO) should be defined by the product manager, as they are responsible for understanding the user's tolerance for downtime and for defining the business's goals for reliability. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

Who monitors the error budget?

The Site Reliability Engineering (SRE) team is responsible for monitoring the error budget. They are the ones who provide the technical expertise and the data to show what is a realistic and achievable reliability target, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What happens when the error budget is exhausted?

When the error budget is exhausted, the team must stop all feature development and focus solely on reliability work to prevent a larger incident. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

How do error budgets help balance velocity and reliability?

Error budgets help balance velocity and reliability by providing a data-driven way to manage the tension between the two. They give a team the freedom to experiment and to innovate, knowing that they have a set amount of "unreliability" they can spend, which is a major part of the modern workflow.

What is a good error budget?

A good error budget is one that is based on data and that is realistic and achievable. It should be a collaborative effort between the business's desire for new features and the SRE team's need for system reliability, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What is a Service Level Indicator (SLI)?

A Service Level Indicator (SLI) is a quantitative measure of a service's reliability, such as latency or request success rate. It is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

How do SREs use an error budget?

SREs use an error budget as a tool to manage a service's reliability. They are the ones who monitor the budget and who sound the alarm when the budget is running low, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What are some common pitfalls in defining an error budget?

Some common pitfalls in defining an error budget include: setting an unrealistic budget, a lack of buy-in from all stakeholders, and failing to take action when the budget is exhausted. These are a major part of the modern workflow that is focused on providing a high level of service to the business and its customers.

What is the role of the development team in an error budget?

The development team's role is to work within the error budget to deliver features. They must understand the SLO and the error budget and must be willing to make a data-driven decision about their work, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What is a "stop on red" rule?

A "stop on red" rule is a rule that says that when the error budget is exhausted, the team must stop all feature development and focus solely on reliability work. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

How does an error budget help with accountability?

An error budget helps with accountability by providing a clear and consistent way to communicate the trade-offs between velocity and reliability. It is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What is the role of engineering leadership in an error budget?

Engineering leadership's role in an error budget is to facilitate the negotiation and to provide accountability. They are the ones who must enforce the "stop on red" rule, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

How do you calculate an error budget?

An error budget is calculated by taking the total time a service is available and subtracting the agreed-upon SLO. For example, for a service with a 99.9% SLO, the error budget is 0.1% of the total time, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What is the role of a DevOps team in an error budget?

A DevOps team's role in an error budget is to work with the SRE team to implement the necessary tools and processes to measure and to monitor a service's reliability. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What is a good way to start with an error budget?

A good way to start with an error budget is to choose a simple service and to define a clear SLO and error budget. This will allow a team to get a feel for the process and to build momentum, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

What is the difference between an SRE and a DevOps engineer?

An SRE is focused on the reliability of a service, while a DevOps engineer is focused on the automation of the software delivery lifecycle. This is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.

How does an error budget help with feature development?

An error budget helps with feature development by providing a clear and consistent way to manage the tension between velocity and reliability. It gives a team the freedom to experiment and to innovate, knowing that they have a set amount of "unreliability" they can spend, which is a major part of the modern workflow.

What is the role of executive leadership in an error budget?

Executive leadership's role in an error budget is to provide the vision and the strategic alignment for the error budget. They must understand the trade-offs between velocity and reliability and must be willing to make a data-driven decision about a service's reliability, which is a major part of the modern workflow and is a major part of a successful business that is looking to scale its operations.