Skip to main content

Decentralize Conceptual Models

Context​

Microservices are in use or are planned to be adopted. In combination with the service cut, a global data model was introduced or is currently considered.

Problem​

  • All microservices depend on the global data model. This dependency hampers the independent evolution of the individual microservices.
    • Changes are expensive since everyone has to agree.
    • Changes are expensive since other microservices have to be changed as well.
    • Changes in the global data model are scarce due to their expensiveness.

Solution​

Decentralize the data model instead of introducing a global one.

Data entities (e.g. User, Order, ...) should not be standardized across microservices. Instead, each microservice should have its own view on the data entity, adding attributes valuable in the microservice's context, changing existing attributes to their needs, or ignoring attributes they don't need. These different views on the data entities emerges by applying Domain-Driven Design for microservice decomposition. On the architectural and technical level, the decentralization of the data model can be enforced by isolating the data of each microservice.

This approach might lead to redundancy of data but contributes significantly to the scalability and agility of individual microservices by ensuring their independence. Some projects might still hold on to a global data model if the costs of decentralizing the data model outweigh the scalability and agility characteristics of the system. Those "degenerated" microservices, as one of our interviewees called them due to their loss of independence, can still contribute to the maintainability of a software system, e.g. by partially partitioning the data model.

Maturity​

Proposed, requires evaluation.

Sources of Evidence​

L3:

  • Need for decentralized data governance in context of a system
    • every time anyone wanted to add some metadata, needed to be added to DeveloperData
    • not a good habit because they were independent units only sharing contracts with other parts of system

L6:

  • Decentralized data management
    • decentralization of conceptual models and storage backends
    • different services have different conceptual models of the world
      • e.g. different attributes of the same entities
    • enables polyglot persistence => coexistence of different storage technologies
  • Context: DIMMER platform for IoT devices
    • Services use own views or concepts of the IoT devices
    • Example both microservices maintain their bounded contexts
      • Resource Catalog Service: provides basic information about the devices config, deployment, and supported comm. tech.
      • Semantic Datastore Service: describes same devices with additional attributes and relations to other entities and platform services using semantic weg
  • => lets each partner concentrate on their work while maintaining overall system compatibility

L7: Context: microservice definitions

  • Decentralized data management and polyglot persistence by Fowler
    • part of principle "decentralize all things" by Newman

L12:

  • Need for decentralized data governance in context of a system
    • every time anyone wanted to add some metadata, needed to be added to DeveloperData
    • was an integration point among services
    • not a good habit because they were independent units only sharing contracts with other parts of system
    • => rearchitect system so that every service governs its own metadata and data by themselves

L14:

  • decentralizing responsibility for data across microservices
    • implications on managing updates: no ACID transactions

L17:

  • main issues of migration
    • 2nd rank: migration and splitting of data in legacy databases
  • one hand: all non-consultants just connected to legacy db / existing db cluster
    • partially reduce benefits of microservices
    • not always able to split the existing data
  • other hand: consultants recommended splitting data
    • each microservice has own private database

L19:

  • microservices: critical to have well defined bounded contexts
    • encapsulates data model, domain model; defines integration points
    • some models need to be shared, some others only exist within service

L25:

  • need to split data in order to implement db-per-service pattern

L34:

  • Each service only maintains its context => own perspective over particular data
    • possibly duplicates
  • price of flexibility of microservices: restate and redefine data definitions (and business rules)
    • introduces replications in DB
    • lacking a centralized view on overall system processing, rules, constraints, ...
  • orchestration via mediation layer => introduces canonical data model
    • agree and standardize on data models they exchange
    • ends up with one kind of business object: single Person, Order ,... with matching attributes and associations
    • (+) easy to introduce such model with orchestration
    • (-) later changes very difficult since all parties have to agree
    • (-) individual systems have limits on evolution
  • Alternative: bounded contexts from DDD
    • each service with business objects in specific context
    • consider certain attributes, ignore others
    • large model divided into small contexts
    • allows to model business objects based differently on particular needs
    • not all service have same needs => should have independence to design their needs
  • Each microservice in charge of its own data model
    • possibly replication
    • (+) avoids bottleneck
    • (+) less coupling / more autonomy by reduced sta structure dependencies
  • When fragmenting data model to introduce bounded contexts, multiple strategies cited:
    • (1) shared kernel: each domain model shares common elements, in specialized areas they differ
    • (2) customer/supplier: subsystem provides domain model that is determined by caller
    • (3) conformist: caller uses same model as provided by subsystem and reuses its knowledge
    • (4) anti-corruption layer: translation mechanism to keep systems decoupled
    • (5) separate ways: no integration
    • (6) open host service: system provides special services for everybody's use to simplify integration
    • (7) published language: unchangeable linguistic elements (contracts, events, ..) visible to outside with a meaning in multiple subdomains
    • 1-3, 7: tie domain models together
    • 4-6: independence
    • coordination effort perspective (ascending effort): 5, 3, 4, 6, 7, 2, 1

L40:

  • Segmentation of application data by partitioning monolith
    • avoid having single monolithic storage
    • required by decentralized data governance (sharding pattern)

L42:

  • half of participants: existing data during migration is not migrated
    • doesn't align well with principles "hide impl detail" or "decentralize data management"
    • might hinder evolving services independently and scaling services and their data
    • => if no need for scalability, they put no effort in migration of data

L46:

  • CRM example
    • at DB level: will share same database including coupling at that level
    • nature of current systems to have domain model represented in a common database schema
      • => provision business functionality quickly and efficiently
      • "Almost all business system packages are architected this way"
  • In these environments: independent microservices down to DB level is wishful thinking
    • since very costly
    • without providing immediate business value
    • introducing complexity overwhelming any advantage of agility or scalability

L54:

  • DB broken into smaller DBs
    • each containing what is beneficial to certain type of customers
    • shared databases => can be a service themselves
  • Challenging to split DB into smaller independent units
    • especially when relying on unstructured data repositories and possibility of error occurrences
    • data recovery mechanisms required to avoid data corruption

L58:

  • Context: Synapse integration platform
    • different services can operate on same data but demand different data structures can be deployed independently with their own DB
      • apply updates before applying data to their DB
      • e.g. add fields, denormalize, send notification
      • solved by callbacks / hooks
    • pub/sub for data sharing => choose own data update semantics to match need

L59:

  • centralized data store => propagates problems when you change the data structure
    • need to test all code that uses that data
    • if data volume increases: difficult to put the changes in one place
    • central shared DB has become outdated

LN42:

  • Context: threats to migration towards microservices
  • among others: data splitting

LN48:

  • teams can make own decisions on technology
  • each microservice can have its own data model

Interview C:

  • Context: asked for practical experience with consistency and transactions
    • Solution 1: no real microservices, centralized DB
      • seen most often in practice for important parts
      • services lose independence, can't manage themselves anymore
        • too many people want different things of DB (scalability)
        • single point of failure even if scaled (is still the bottleneck)
      • might be an option if microservices used for main reason of maintainability, and not independence
      • global schema can be partially partitioned
      • this kind of degenerated microservices seems to work well sometimes

Interview E:

  • Context: UI integration - sharing of widgets between micro-frontends
  • Tradeoff: sharing widgets => less redudancies but higher coupling
    • alternative: aseparate ways, no sharing (like DDD suggests) => strong isolation, but redundant code
    • If micro-frontend resp. for one domain: maybe different views on the same thing anayway
  • Same trade-off at communication
    • Much communication => breaking encapsulation
    • Not much communication => good encapsulation, but user maybe has to select the same things 7 times (in each micro-frontend once)
  • Context: consts of independency
    • decoupling of the system => redundancies in the backend
    • comparison between redundant data since system self-contained and not depedent of others

Interview F:

  • Context: definition of a microservice
    • self-contained unit with own data model that is responsibile for one certain task
  • Context: data replication vs. on-demand data fetch of other microservice
  • consideration of staleness of data
  • possibility to have a own data model - looks different than on other services