CI/CD for Automated Deployment

Context

Microservices are being adopted. Intergation and deployment is done manually.

Problem

• Integrating and deploying the increasing amount of microservices becomes less manageable
  • Slows down the innovation speed or agility of the project
• Manual steps slow down the deployment frequency

Solution

Introduce decentralized CI/CD pipelines to automate the deployment of all microservices. By removing manual steps in the deployment process, the large amount of microservices can be deployed frequently and independently.

The term "decentralized" CI/CD was coined in L7 an expresses that every microservice has their own CI/CD pipeline. The CI part of the pipeline usually automates the build and test of a single microservice and creates a deployment artifact, e.g., a container image. The CD part automates the deployment of the deployment artifact into the deployment environments.

In larger projects, the scalability of CI/CD pipelines and their infrastructure might become a challenge L14.

Maturity

More data required (only 2 sources).

Sources of Evidence

L3:

• Need for automated deployment
  • due to growing number of services
  • to decouple the build lifecycles of services
• Preparing CI pipeline
  • CI first step towards CD
  • integrate early and regularly to prevent conflicts
  • requires CI server (as a service of self-hosted), and an artifact repository
  • they chose Jenkins for CI, Artifactory as artifact repo, GitLab as repo
• Figure 3: example of e CI pipeline with multiple steps vs. monolith
  • each microservice has own pipeline with steps fetch, build, test, deploy artifacts, deploy to Kubernetes
  • indendent delivery for every service => independent deployment
  • previously used integration tests, run all when a single service changed => replaced by CDCT
• Table 1: DevOps impact of CI => first step towards CD, a devOps practice

L5:

• Context: quality metrics of microservices
• Fewer occurances on CO topic among others => need for further research
• CD under category "Agility"

L7:

• Decentralized continuous delivery is practiced during service development
  • requires/promotes high degree of automation and autonomy
• Table 1: Lewis: "Infrastructure automation" as subset of Newman's "Adopt culture of automation"
• Table 2: Characteristics of microservices
  • infrastructure automation; SOA has no direct pendant
• Fig 1:
  • infrastructure automation as mix of development and physical VP
  • Automated CD as cross-cutting concern
• Table 4: Build tool chain out of scope of SOA, but part of Microservices by decentralized continuous delivery

L8:

• To achieve agility, microservices are packaged and deployed to cloud using containers
• and follow DevOps practices and are supported by fully automated software integration and delivery
• 6th wave: CD technologies like Ansible and Drone
  • provide general integration solutions to automate many of the DevOps practices
• Table 1: Microservice tools
  • Cagetory Continuous delivery: Ansible, Drone, Spinnaker, AWS CodePipeLine, Otter

L12:

• Continuous Delivery enables on demand deployment to any environment
  • automates delivery life cycle as muc has possible
  • technique likes Continuous Integration and Continuous Deployment, embraces DevOps
• Microservices => number of services will increas => need for mechanism to automate delivery process
• Need for automated deployment: number of services were growing
  • problem: automate deployment process and decouple build life cycles of services from others
  • solution: Configuration server and Continuous Delivery components
• Continuous integration is first step towards continuous delivery
  • allows integrate early and often, prevent future conflicts
  • they chose Jenkins as CI server

L13:

• proliferation of service processes requires automated deployment mechanism
  • sometimes called CI or CD

L14:

• Automation is key to DevOps success: building out of version management repositories
  • automated execution of tests
  • automated deployment in test and productio nenvironments
  • performance benchmarks
• Context: Microservices at otto.de
• Planned t practice DevOps incl. CD to deliver frequently to the customer
• Scaling delivery pipelines
  • usage of CD pipelines for every single application
  • every commit => checked out, compiled, package, deployed, tested in CI stage
    • on pass: deploy container to next stage: testing
      • load and integration tests, approve stories by PO
    • contains latest dev versions of all verticals
  • pre-live stage: test version against other currently deployed services
    • further intergation tests, ensure compatability of new and old versions
  • finally: production stage
• High number of pipelines => Jenkins reached its limit
  • need to engineer pipelines as critical software components
  • internal DSL LambdaCD
  • pipelines run on same infrastructure as microservices
  • also allows running locally without extra CI server; can be debugged
• quality assurance measures for CI/CD takes effect for reliability
  • Fig 2: low number of incidents while increasing number of deployments

L16:

• Context: technologies that led to microservices
• continuous delivery: treat every commit as if it could the next release
  • includes automatic builds, tests, calculating software metrics
  • CD as advance to CI: cover automated deployment
• Context: containerization of microservices
  • technologies as Docker support automated deployment of containers => automated deployment
• CI/CD is partly appled in IoT/CPS

L17:

• DevOps toolchain is supported by microservices
• each microservice developed and deployed independently
• requires effort for DevOps infrastructure
  • needs to be taken into account next to development effort

L19:

• Each ersvice should be treated as independent applicaiton with own delivery pipeline
• Context: challenge failure isolation in microservices
  • need to divide to concur model where break into smaller chunks, fast tooling to support continuous delivery
  • enables devs to change one thing at a time => know that's the only thing that broke
• Context: case study
  • Travis integrated with GitHub to run tests
  • If all tests passed => use webhook to trigger building docker container, pushing it to a container registry
  • store deployment scripts for different environmenty: docker compose
    • stil some manual activities: update compose file with container versions and push ti repo
    • from there, webhooks will trigger deployment

L20:

• security risks: CI/CD 5th largest pain with 1 occurance
• significant gain is CI/CD, a DevOps practice to continuously release updated versions of software
  • 15 of industrial studies (3rd place): microservices natively support CI/CD
• CI/CD can be directly operated on each microservice => indepdentent updates

L21:

• Containersization avoids inteferences between two versions of a service
• naturaly leads to more frequent updates into the direction of continuous deployment

L22:

• Mike: additional aspect next to learning from SOA: adopting modern principles of operations like DevOps and automated deployments
• James: CI and TDD are complementary
  • more tests make CI more attractive
  • CI makes different types automated testing more attractive
  • => reach global maxima of full automated build pipelines
    • prerequisite for microservices
• James: built DSL for deployment to take away pain of multiple small repos
• Mike: multple production releases per day to get features and fixes out
  • automated things, established test and sign-off proceduresnt updates into the direction of continuous deployment

L24:

• continuous delivery s devops practive for on-demand deployment to any environment thru automation
• essential companion of micrservice archtectures
  • number of deployable units tends to increase (comp. to monoliths)
• DevOps requirements
  • microservices should follow practice of continuous delivery pipeline
  • CI, automated testing, dependency management, automated deployment, container orchestration
  • R4: support for CI for benchmark system
    • new code automatically integrated with existing software code base after every commit
    • Tools: Jenkins, TeamCity
    • may trigger additional tasks as code quality check and testing
• Spring Cloud Demo Apps and Socks Shop: use TravisCI

L25:

• emergence of deployment patterns => deeper coverage of continuous development, integration and deployment

L30:

• Microservices considered as enabler for CD and DevOps
• Technical autonomy allows creating independent deployment pipelines per service
  • => changes can be repidly delivered
  • only affected service needs to be built, tested
  • pipelines can run in parallel
• challenge: align performance testing with CD practices and speeding up test stages
  • pace hard to achieve with extensive integration and performance testing
  • => choose scope of tests carefull
    • which test to choose on a commit? which for consolidated set of commits?
    • => dynamic mechanisms to decide which tests to run
    • adapt (semi-)automatically to guarantee reasonable testing strategy

L31:

• Migration practice MP1: Enable continuous integration
  • Intent: build CI pipeline
  • Context: migration from monolith
  • Problem: number of services increases => how always have available production-ready artifacts? how to prepare for CD?
  • Solution: set up CI
    • build and test to ensure available of production-ready artifacts
    • contains code repo, artifact repo, CI server
    • each service in separate repo => clearer history, separate build life cycles
    • create CI job for each service
    • fetch, test, build artifacts, publish artifact to artifact repo
    • failure => terminate job, inform team of errors
    • team should fix error before doing anything else
    • rule: changes should not break system's stability and should pass tests
  • Technology Stack: Gitlab, Artifactory, Nexus, Jenkins, GoCD, Travis, Bamboo, Teamcity
  • No occurance in the three case study projects
• Microservices have close relation to CD => radical change in software development
  • need to understand deployment architecture
• MP13: Containerize services
  • Context: CI is in place per service
  • Part of solution: build container image in CI and store in image repository
    • image run in production and development to produce same behavior
• MP14: Deploy into cluster
  • Context: CI is set up with building container images

L32:

• Context: benefits of microservices
• Shorter deployment time
  • reduced deployment time from 30 to 3 minutes after migration from monolith
  • shorter pipeline time important for practicing CD
    • min time to release code change to production => e.g., to release a fix
    • influences code commit behavior => more frequent commits
• decomposition makes deployment procedures simple => easier to develop deployment automation that completely eliminates manual activities
  • common tool chain to handle any microservice becomes feasible
  • adding new microservices does not require new tool => more scalable approach
• traditionally: hand over software to ops
  • not able to handle significantly increased number of services with frequent releases => exceeds what ops team can handle
  • solution: CD platform
    • CD pipeline per service
    • builds, runs automated tests, provisions testing env, release on click of a button
      • If coverage below 90% => fail
    • removes ops as bottleneck
• always online system integration test environment
  • part of CD pipeline
  • deploy service with all dependent services
  • execute tests to ensure they work

L34:

• Self-contained microservices more efficient for continuous deployment with fast demand response among others
• independent dev and deployment brings individual scalability and continuous delivery
• continuous delivery => enables on-demand automated deployments to support system elasticity to request load
• Context: Deployment operations as category for research challenge
  • focuses on deployment efforts, automation, ...
  • among keywords: devops, automated deployment

L35:

• Context: background section to understand microservices
• ability to immediately integate updates led to continuous software delivery model
  • teams incrementally deliver features

L41:

• new images deployed to registry after CI system successfully built and tested
• Context: FX Core Microservice Architecture
• Automation: alservices have automated CI and CD pipeline
  • utilize rolling update feature etc. provided by Docker Swarm
  • modern CI and CD can be applied => bring full automation to the process
    • => reduce barriers between teams

L42:

• Challenge of configuration and setup of testing automation, i.e., in CI and CD pipelines

L43:

• Context: Case Study
• CD = ability release software whenever we want
• Continuous delivery incorporates deployment pipeline with validatins taht software must pass on its way to production
  • compile code, packaged by build servr, tested, tag as "releasable"
• Need to support multiple applications for multple clients => CD pipeline for eac happlication
  • pipeline might differ, but structure broadly similiarly
  • used tool might be even different
• Key benefits
  • minimization of difference between development and production environment
  • frequency reduces risks => rapid diagnosis on small increments
  • deployment common occurance => removes fear from process
  • everything is automated, no human intervention required unless there is a probalm
• CD goes well with containerization
  • they used Jenkins
  • tools increased production and deployment speed

L47:

• Context: costs of quality assurance and deployment
• deployment cost = time spent by operational teams to deploy release
  • can be heavily decreased by automation
  • automation of regression tests
  • cost increases with number of services
    • if not automated: linearly
    • if automated: asymptotical

L49:

• Context: shifting complexity from application into infrastructure
• includes automation of service deployment, remination, continuous integration and delivery among others

L59:

• Companies like amazon make thousands of changes per day
• continuous delivery becomes powerful tool for devs updating SaaS and onlin systems
  • => powerful tool for business agility and competitive advantage
• Microservices utilize automated scripts in cI systm
  • localized changes shouldn't impact system (automated testing)
  • smaller test layer than the tests running on the output of the larger application => drives down time and costs of testing
• Leads to continuous delivery
  • each microservice can move independently
  • fed into CI to make sure the whole system works correctly
  • benefit from microservices if you dun continuous delivery of online service

L61:

• Inrastructure automation allos reducing manual effort to build, deploy, and operate microservices => continuous delivery
• Examined studies that looked at DevOps: two perspectives
  • 1. set of practices to reduce time between comitting change and deploying to production while maintaining high quality
  • 2. hvae dev and ops team work closely together for achieving rapid and continuous release cycles

L62:

• Neutral development technology: microservices can be deployed independently via their own delivery pipelines

LN41:

• Containers can provide standardizes building, CI, and CD
• Microservices are independently deployed by fully automated deployment tools, e.g., Jenkins

LN42:

• Context: migration to microservices
• use DevOps tools on each phase, e.g., fuge, git, github, jenkins, phantom, kubernetes
• microservices push automation from app to infrastructure
• Config management tools facilitate fast production provisioning

LN43:

• DevOps: teams set up CI and CD pipelines
• Microservices can be released frequently => only if possible to validate software behavior
  • automated tests
  • CI/CD go hand in hand with microservices
  • without: hard to handle multiple microservices, their deployments, and behavior validation
• new challenges with using microservices in production
  • hundreds of services with multiple instances
  • necessity for tools to automate deployment, scaling, and service management
  • manual deployment is not an option if done multiple times per day
• Fast change cycle one of the main points of microservices => deployments need to be fast and smooth
  • called continuous delivery
  • shorten release cycle of application by making devs and ops work together

LN48:

• Deployments become a challenge with increasing number of microservices
• CI/CD major challenges in any enterprise application due to short release cycles
• any update to any microservice needs to be integrated into the application with minimum effort and downtime
• Context: Container-based design for microservices and their deployment
  • deployment server fetches code from repo, compiles, tests it to create binaries
  • create container image and store in container repository
  • deploy to the deployment server
  • store configuration to config storage
  • switch configuration to new version
  • if anything failes: rollback to old version
  • => minimum downtime
• Exeprimental setup
  • use Jenkins to pull code, compiles, tests, packages microservice to docker image
  • image then deployed to deployment server

LM43:

• Context: SLR findings about microservices in DevOps
• S20 recommends using CD platform
  • to deal with operational and configuration complexity -provides CD pipeline for each service
    • gives control over the integration of microservices
  • frequent deployment through CD pipeline
• S19 introduces SONATA NFV plaform
  • tools like GitHub, Jenkins, Docker
  • can create CI/CD pipeline to autonate software delivery
• Continuous integration tools
  • code of MSA based system needs to be integrated frequently
    • into a shared repo
    • 7 tools for CI
    • eg Jenkins
    • build, test, deployment => integrates various operations
    • automate DevOps practices, eg CI CD
• Version control
  • GitHub provides convenient mechanism for CI/CD
  • BitBucket as well
• Migration pattern recommends enable CI for migration from monolith
  • need for patterns tu support microservice CI/CD pipeline
• Jenkins as popular CI/CD technology
  • interacts with version control, build automation, testing, configuration management

LM45:

• Context: interviews and insights from multiple cases on technologies and sw quality in MSA
• C3-S5
  • some alignment with 12 factor app principles
  • degree of automation rather low
  • CI/CD pipeline is currently implemented
    • to decrease cycle time
    • to increase software quality
• C4-S6
  • High degree of automation using GitLab CI
  • CD will be implemented at a lager stage
• C5-S8
  • use Jenkins for CI/CD pipelines
  • production deployment still involves manual work
  • separate dev, ops, and devops teams in close collaboration
• C6-S10
  • No DevOps practices besides basic CI/CD
• C8-S12
  • high degree of automation with CD pipelines
  • every accepted commit goes live without manual effort
• automation very different
  • most: decent degree of automation, usage of CI/CD
  • only fully automated CD in 3 cases (S9, S12, S14)
  • some described alignment with 12 factor app principles
  • CI/CD as challenge in 6 cases (P2, P3, P5, P6, P9, P14)
  • finding qualified DevOps engineers as challenge (P1, P9, P12)

LM48:

• Context: microservice migration describes an examples project (FX Core) and compares back to monolith
• microservices lead to increasing HTMLProgressElement, hard to maintain
  • manually managing is overhead: deployment, configuration, maintenance for every instance and host
  • automation as solution, introduction of new microservices = no management overhead
    • only maintenance of scripts
• CI can deploy container images in container registry
  • every service has CICD pipeline (they use GoCD): build, test, deploy
  • interacts with Docker Swarm API for rolling updates
  • checks if deployment works correctly

Interview A:

• You can deploy microservice manually via a shell, questions that it makes sense
  • always comes in combination with modern CI/CD philosophy and pipeline
  • Which tool to use? e.g. Jenkins, GitLab
• Need to rework contracts of customer relations: one release per year, or maybe two
  • releases are painful, with much pizza and cold coffee
  • most often it doesn't work
  • => CI/CD
• Journey
  • 9 years ago started with devs having root priviledges on prod fixing bugs
  • neigher versioned, nor documented
  • at best: Sunday night because the one who had the knowledge was on their way to vacation
  • crashing on Monday and noone could comprehend what actually happened
• Big pain, still there: topic of rethinking stages and pipelining with obligatory quality gates
  • commitment: if software doesn't pass uality gate => does not go into production
  • pragmatism to say 80% worked, ah lets deploy => conservative ops says no
  • ops people still tend to quickly go on prod machine and fix since it works
  • started with Ansible and now Helm Charts including configuration has to be in version control => traceable
  • Need to keep track of the compatibility of software
• have build runner, ansible rnner, test runner, I-don't-know-what runner and modelling pipelines
  • big question in past: gitlab vs. Jenkins
    • hand know-how with Jenkins and can do almost everything with the army of plugins
      • different levels of quality and maintenance of plugins
      • but they need to build a critical infrastructure
      • if it crashes, you can send 40 devs and 10 testers to vacation
    • => so decided for GitLab
      • more modern
      • not all capabilities of Jeninks, but they don't need it
      • still think it was the right decision
    • on lower level still Ansible

Interview B:

• Where happens testing?
  • testing all services for themselves is relatively easy
  • making sure services work together => test API
    • Consumer-driven contracts
    • => automated testing
    • isolated from each other => works well with CI/CD pipelines
• Context: standardization
  • when I can deploy my service by miself => also need to bring it return it quickly and cheaply
    • if I notice that something is off
    • best case: discovered by CDCs => doesn't make it into production with the fault
    • but it happens in practice => need to roll back incl. database
  • way into production is easier than out of it, often neglected
    • usually project start of with only a few services, every call is watched closely
    • with increasing services, at some point, this close supervision is not possible any more
    • => important aspect for migration: start early to automate!
      • not everyhing from the start, like resilience patterns
      • but automation to deploy and rollback, incl. tests => CI/CD pipeline
• with technology I introduce complexity that I need to deal with
  • trick: move complexity from an non-atomizable area to an automatable one
    • => if I dont automate I get huge problems
  • CI/CD pipeline, testing, versioning, etc.
  • only works with initial amount of automation, and in bigger systems with a large degree of automation
  • big readjustment to some who usually have their application server

Interview D:

• Intrastructure automatization also makes sense without microservices
• Context: who is repsonsible for deployment and operation platform?
  • deployment was done by dev teams, initial automatization as well
  • at some point, keep all teams somewhat synchronized => enough work was there for an own team
  • but already quite automated to spawn test system, usage was easy
  • CI scripts still done in dev team
    • building
  • infrastructure / release management team making sure standards are adhered, there are automated tests, and the available of build infrastructure like CI server
    • everything about putting the microservices together and actually deploy
  • had many experienced people, no big issues with knowledge barriers

Interview F:

• Context: compatibility of interfaces in staging environment
• integration tests that spawn staging environments, deploys / starts services, run tests
  • all automated!
• Context: releases
  • before deployment production: dedicated environments to test compatability after breaking changes
  • if still something went wrong, just roll back and wait until other services catch up
  • most services deployed very frequently; wait time for changes not long
• Context: how deployment solved
• self-made deployment, all the automation done at beginning of project
• became more and more over time, had to keep teams in sync somehow
  • enough work for a dedicated team which emerged
    • ensure standards applied
      • automated tests
    • manage CI servers and interface to it
    • final deployment
  • CI scripts to build services still done by teams
• need for automation and CI knowledge within teams
  • many experienced people
  • tools were known
  • inexperienced people grew into it
  • worked pretty well