Skip to main content

Reduce Deployment Coordination

Context​

Microservices are being adopted. Multiple microservices' instances are being deployed. Responsibility is given to the teams but there is still a release coordination process.

Problem​

  • A central release coordination is slowing down the innovation speed and the agility of the project.

Solution​

Remove the organizational necessity for release coordination and empower the microservice teams.

The necessity for release coordination can be manifold:

  • There is no strategy to evolve microservices to keep their API contracts stable.
  • There are deployment dependencies. If two microservices are always deployed together, they might be better refactored into one microservice. Revisit the service cut.
  • There is a shared quality assurance stage, e.g. for testing. On the one hand, this is an indicator that there might be no automated on-demand test environment. Providing this capability to microservice teams allows them to test for compatibility between microservices themselves. On the other hand, testing strategies like consumer-driven contract tests might remove the need to run all services for testing purposes minimizing the inter-team coordination.
  • There is no trust into the microservice teams to handle deployment and operation or the microservice teams are not confident to fulfill this role. Providing good monitoring capabilities and rollback mechanisms can mitigate those concerns and establish trust into automated deployments.

Resolve the applying points before empowering the microservice teams to automatically deploy their new microservice versions without central coordination.

Maturity​

Proposed, to be evaluated.

Sources of Evidence​

L3:

  • Integration tests required running whole set of tests if just one service changed
  • replaced with consumer-driven contract tests => independent testing of each services
    • minimizes inter-team coordination
    • a bit more complex testing strategy
    • enables forming smaller teams as DevOps practice

L8:

  • rapid development of microservices in multiple execution environments
    • on arbitrary schedules
    • bare minimum of centralized management

L19:

  • Deployment of microservice should happen without need for coordination with other service's owner
  • if 2 services always releasing together => might be sign they should be one thing => revisit service decomposition
  • loose coupling => more frequent and rapid deployments
    • eventually improve responsiveness of application regarding users' needs

L32:

  • After migration to microservices, no need to queue for merges
    • no need to resolve merge conflicts with other teams
    • no need to coordinate with and wait for other teams' changes
    • each team can deploy changes independently

L34:

  • If deployment dependencies => then on much lower scale since no general context or central governance exists
  • Primary goal of microservices: independent service deployments and evolution

LM45:

  • Context: interviews and insights from multiple cases on technologies and sw quality in MSA
  • P6 and P15 report about independent release cycles

Interview A:

  • Need for versioning
  • Need for traceability
  • Need to validate compatibility
  • Changing release management as central part of the migration

Interview B:

  • Organizational: who accepts the service?
    • everyone: just send to production
    • release coordinator: has to accept the changes
      • has a global view on the releases
      • works in projects with 5-300 microservices, not with 2000
      • => not really microservices? not clear..
    • Question of acceptance/trust
      • Teams are theoretically responsible for service
      • Need for mechanisms
        • to recover from faulty deployment
          • Old version to switch back to
        • to monitor deployed service
      • But also resistance from teams: have to take blame for that as well
        • Only works if I feel really responsible for the service

Interview D:

  • Context: why use microservices?
  • if I really need independent deployments of business capabilitites (90%)
    • organizational topic: if I need to move with independent teams in their own pace
    • => that was the challenge of the hyperscalers: to build systems fast
    • large system often forces you into a lock-step => you can do everything only at the same pace
    • Etsy as counter example: cleanly structured monolith that is deployed over 50 times per dynamically
      • also have independent teams within monolith
      • => goal of independent teams and pace can also be achieved without microservices
      • but almost noone does it because they don't have they right surroundings: teams, how to deal with requirements, how infrastructure is built, do experiments
  • Need for lock-step releases
    • maybe 12 per year
    • motivated by test teams
      • since no reasonable on-demand test environment
    • even introduced SAFE to centralize top-down => release train
  • Is the price for microservices really worth it if not benefit from independent deployment?
    • if everything comes together in a release train
    • because you are not reaping the benefits of microservices
    • different non-functional requirements can be solved less expensive

Interview F:

  • Context: problems with independent deployment
  • regular point of conflict because team had different cycles and different development models
    • but need to communicate over API in the end
    • breaking API changes breaks other parts of the system
    • solution: staging environment where everything was deployed to
    • if everything works, deploy everything at once to prod environment
    • lockstep release on breaking changes
  • ideal way: communicate beforehand so that everything is ready in time
    • worked quite often, but still, cycles were different
    • joint meeting every 2 weeks to present state and coordinate
    • If know that breaking changes => spawn own environment where involved teams could already make sure that the services are compatible
  • Frequent deployment led to quick discovery time of breaking changes, kept waiting time short
  • => coordinated releases
    • not at the beginning because everything was Changing
    • would have been a hurdle with strict versioning and so on
    • it has to be possible to make mistakes and then deal with the consequences, esp. at beginning of project
  • Context: standardize the cycles of different teams?
    • No, accepted that pragmatically
    • Takes too much effort in such a large company
    • easier to work around it by good planing ahead and coordinating