14 Microservices Design Best Practices for DevOps
14 microservices design practices for DevOps teams and their trade-offs: boundaries, data ownership, sagas, outbox, retries, observability, pipelines.
Microservices only pay off for DevOps teams when each service can be built, deployed, scaled and debugged on its own. The 14 design practices below aim at that: draw boundaries around business capabilities, give each service its own data, design for failure with timeouts and idempotency, make every service observable, and give it its own pipeline and owning team.
This guide looks at microservices from the operating side: which design decisions make a system easy or painful to run in production, and what each one costs. If you are still deciding whether microservices suit you, start with 12 benefits of microservices architecture. If you are breaking up an existing application, the step-by-step plan is in 18 DevOps steps to migrate a monolith to microservices. Pattern names and sources were checked in November 2026.
Key takeaways
- Service boundaries are the most expensive decision to change later. Base them on business capabilities and bounded contexts, not on technical layers.
- Each service owns its data. Cross-service consistency then comes from events, sagas and the transactional outbox rather than shared tables.
- Every network call can fail or hang. Timeouts, limited retries with backoff and jitter, and idempotent handlers are mandatory.
- Observability (logs, metrics and traces tied together by a trace ID) is a design requirement, not something to add after launch.
- Independent deployability needs independent pipelines, backward-compatible APIs and progressive delivery.
- Too many small services can be worse than a monolith. Martin Fowler's "Monolith First" argues for finding boundaries in a monolith before splitting.
The 14 practices and their trade-offs
| # | Practice | What it buys you | What it costs |
|---|---|---|---|
| 1 | Boundaries by bounded context | Teams change services without coordinating | Up-front domain analysis |
| 2 | Database per service | Independent schema changes and scaling | No cross-service joins or transactions |
| 3 | Contract-first, versioned APIs | Safe independent releases | Discipline around compatibility |
| 4 | Asynchronous events between services | Loose coupling, resilience to slow consumers | Eventual consistency, harder debugging |
| 5 | Sagas for multi-service workflows | Business transactions without distributed locks | Compensating logic to write and test |
| 6 | Transactional outbox and idempotent consumers | No lost or duplicated side effects | Extra table and relay process |
| 7 | Timeouts, retries with jitter, circuit breakers | Failures stay contained | Tuning and testing effort |
| 8 | Stateless services with external config | Easy scaling and replacement | State moves to managed stores |
| 9 | Health endpoints and graceful shutdown | Safe rollouts and self-healing | Careful probe design |
| 10 | Built-in observability with trace context | Fast diagnosis across services | Instrumentation and storage cost |
| 11 | One pipeline per service | Independent, frequent releases | More pipelines to maintain |
| 12 | Progressive delivery and feature flags | Small blast radius for bad releases | Tooling and flag clean-up |
| 13 | Gateway, discovery and service identity | Controlled entry and secure service calls | Extra infrastructure to run |
| 14 | Team ownership and right-sized services | Clear accountability, fewer hand-offs | Organizational change |
Designing boundaries and data
1. Draw boundaries around bounded contexts
A bounded context, a term from domain-driven design, is a part of the business where words have one consistent meaning. "Order" in fulfilment is not the same thing as "order" in billing. Services built around such contexts (catalogue, ordering, payments, shipping) change for business reasons that rarely overlap, so teams can release them independently. Services split by technical layer (a "database service", a "validation service") end up changing together and need coordinated releases, which defeats the purpose.
A practical test: if most features require changes to three services at once, the boundaries are wrong. Event storming workshops with domain experts are a common way to find better ones.
2. Give each service its own database
The database-per-service pattern means a service's data is private and other services reach it only through its API or events. That lets each team change its schema, pick a storage engine that fits (relational, document, key-value), and scale independently. A shared database couples every service to every schema change and turns the database into a single point of failure.
The cost is real: no joins across services and no single ACID transaction spanning them. Practices 4 to 6 are how you live with that. "Own database" can mean a separate schema on a shared server at first; the rule is about ownership and access, not hardware.
3. Design contract-first, versioned APIs
Write the API contract (OpenAPI for HTTP, protobuf for gRPC, a schema for events) before the implementation and keep it in version control. To release services independently, changes must be backward compatible: add optional fields, never remove or rename fields in place, and accept unknown fields. When a breaking change is unavoidable, run the old and new versions side by side and retire the old one once consumers have moved. Consumer-driven contract tests in each provider's pipeline catch accidental breaks before release.
Designing communication and consistency
4. Prefer asynchronous events between services
Synchronous chains (A calls B calls C) multiply latency and failure: if any service is down, the whole request fails. Publishing events such as OrderPlaced to a broker lets other services react in their own time, and a slow consumer does not slow the producer. Keep synchronous calls for queries where the user is waiting for an answer.
The trade-off is eventual consistency. The order service knows about the order before the shipping service does, and your UI and support processes must accept that short delay.
5. Use sagas for workflows that span services
A saga is a sequence of local transactions, one per service, where each step publishes an event or message that triggers the next. If a step fails, the saga runs compensating transactions to undo earlier steps: refund the payment, release the reserved stock. Sagas can be choreographed (services react to each other's events) or orchestrated (a coordinator tells each service what to do). Orchestration is easier to follow and monitor once a workflow has more than three or four steps.
6. Use a transactional outbox and idempotent consumers
A classic bug: a service updates its database and then publishes an event, but crashes between the two, so the event is lost. The transactional outbox pattern writes the event to an outbox table in the same database transaction as the business change, and a separate relay publishes it to the broker. Most brokers deliver at least once, so consumers will sometimes see the same message twice. Make handlers idempotent: record processed message IDs, or design updates so applying them twice has the same effect as once.
BEGIN;
INSERT INTO orders (id, status, ...) VALUES (...);
INSERT INTO outbox (id, aggregate_id, type, payload)
VALUES (uuid, order_id, 'OrderPlaced', '{...}');
COMMIT;
-- A relay process reads unsent outbox rows, publishes them, then marks them sent.
Designing for failure
7. Set timeouts, retry with backoff and jitter, and break circuits
Every remote call needs a timeout; without one, a slow dependency ties up threads until the caller falls over too. Retries help with brief failures, but uncontrolled retries multiply load on a service that is already struggling. The AWS Builders' Library article on timeouts, retries and backoff recommends capped exponential backoff with jitter, so clients do not retry in lockstep, and retrying only operations that are safe to repeat. A circuit breaker stops calling a dependency that keeps failing and returns a fallback, giving it time to recover.
| Mechanism | Protects against | Common mistake |
|---|---|---|
| Timeout | Hung dependencies | Using library defaults that are minutes long |
| Retry with backoff and jitter | Brief network or node failures | Retrying at every layer, multiplying attempts |
| Circuit breaker | Repeated failures overloading a sick service | No fallback, so the breaker just returns errors faster |
| Bulkhead (separate pools) | One slow dependency exhausting all resources | A single shared connection pool |
8. Keep services stateless and configuration external
The Twelve-Factor App guidelines recommend keeping config in the environment and running processes as stateless, share-nothing units. A stateless service can be scaled out, restarted or moved by Kubernetes at any time. Session data, uploads and caches belong in managed stores (a database, object storage, a cache cluster). Configuration comes from environment variables or ConfigMaps, and secrets from a secret manager, so the same image runs in every environment.
9. Expose health endpoints and shut down gracefully
Orchestrators need to know whether a service is alive and whether it is ready for traffic. Provide separate endpoints: liveness should check only that the process works, and readiness should check that it can serve requests. On shutdown, stop accepting new requests, finish in-flight work, close connections and exit, all within the termination grace period. Add PodDisruptionBudgets so node maintenance never takes down every replica at once. Probe design is covered in 10 Kubernetes health checks.
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 3
Designing for operations
10. Build observability in from day one
One user request may touch ten services. Without correlation, debugging means searching ten log streams by timestamp. Instrument every service with OpenTelemetry, which covers traces, metrics and logs, and propagate trace context on every call and message. Log in structured JSON with the trace ID in each line. Give each service the same base metrics: request rate, error rate and latency. Useful metric sets are listed in 18 observability metrics for cloud-native systems.
11. Give each service its own pipeline
Independent deployability means a change to the payments service builds, tests and deploys only the payments service. Whether you use one repository per service or a monorepo with path-based triggers, each service needs its own pipeline, versioned artifact and deployment. Shared libraries should be versioned and pulled in deliberately, not rebuilt into every service on each commit. Pipeline designs used by large engineering organizations are collected in CI/CD pipeline examples from big tech companies.
12. Release progressively behind feature flags
With many services releasing daily, some releases will be bad. Canary releases send a small share of traffic to the new version and compare its error rate and latency with the stable version before going further. Feature flags separate deploying code from turning a feature on, so a problem can be switched off without a redeploy. Remove flags once a feature is fully released, or they turn into hidden configuration debt. Rollout options are shown with manifests in Kubernetes rollout strategies with examples.
13. Put a gateway at the edge and identity between services
An API gateway gives clients one entry point and handles routing, authentication, rate limiting and request shaping, so individual services do not each reimplement them. Inside the cluster, services find each other through service discovery (Kubernetes Services and DNS), and a service mesh can add mutual TLS so every call is encrypted and authenticated by service identity. Grant each service only the permissions it needs, in the network and in the cloud. The difference between discovery and a mesh is explained in service discovery vs service mesh.
14. Align services with teams and keep them right-sized
Each service should have one owning team that builds it, deploys it and answers its pages. Services owned by nobody, or by three teams at once, decay fastest. Size follows ownership: a team of six can usually run a handful of services well, not forty. If most services are trivial wrappers, merge them. A modular monolith with clear internal boundaries is a reasonable choice for a small team, and Fowler's "Monolith First" essay argues that it is often the better starting point because the right boundaries are easier to see once a system exists.
Design review checklist for a new service
- Which bounded context does it belong to, and which team owns it?
- Which data does it own, and who else needs that data (API or events)?
- What is its API contract, and how will it evolve compatibly?
- Which calls are synchronous, and what are their timeouts and retry limits?
- Which workflows span services, and how are failures compensated?
- Are message handlers idempotent?
- Does it expose liveness and readiness endpoints and shut down gracefully?
- Does it emit traces, metrics and structured logs with trace IDs?
- Does it have its own pipeline, canary step and rollback plan?
- What are its SLOs, and who is on call?
Learn to build and run microservices
Containerizing services, writing their pipelines and running them on Kubernetes with probes, rollouts and monitoring is the core of our DevOps training, with image builds covered in depth in our Docker training. Both run in our Kharadi, Pune classroom and live online.
FAQ
How do I decide microservice boundaries?
Base them on business capabilities and bounded contexts from domain-driven design, not technical layers. If most features need changes to several services at once, the boundaries need rethinking.
Should every microservice have its own database?
Each service should own its data and expose it only through its API or events. It can start as a separate schema on a shared server; the rule is about ownership and access, not separate hardware.
How do microservices handle transactions across services?
With sagas: a series of local transactions where each step triggers the next, and failures trigger compensating steps. The transactional outbox pattern makes sure events are published reliably alongside database changes.
Why are idempotent consumers important?
Most message brokers deliver at least once, so a consumer can receive the same message twice. An idempotent handler produces the same result whether it processes a message once or several times.
How should microservices retry failed calls?
Always set a timeout, retry only safe operations, use capped exponential backoff with jitter, and avoid retrying at every layer. Add a circuit breaker with a fallback for dependencies that keep failing.
What observability does a microservice need?
Traces, metrics and structured logs, all carrying a shared trace ID propagated across calls and messages. OpenTelemetry provides a vendor-neutral way to instrument all three.
Can microservices share a CI/CD pipeline?
They can share pipeline templates, but each service should build, test and deploy independently, with its own versioned artifact. Otherwise you lose independent deployability.
When should a team avoid microservices?
When the team is small, the domain is still unclear, or the operations platform is immature. A modular monolith is often the better start, as Martin Fowler argues in "Monolith First".
References
- Martin Fowler and James Lewis: Microservices
- Martin Fowler: Bounded Context
- Martin Fowler: Monolith First
- microservices.io: Database per service
- microservices.io: Saga
- microservices.io: Transactional outbox
- AWS Builders' Library: Timeouts, retries and backoff with jitter
- The Twelve-Factor App
- OpenTelemetry: Signals
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0