How to maintain data consistency across Microservices?
While Microservice applications are built as a set of modular components, they are easier to understand, simpler to test and effortless to maintain over the life of the application. It enables organizations to achieve agility and be able to improve the time it takes to get working enhancements to production.
On the downside, there are quite a bit of challenges as well, each service has its own database and wherever business transactions span multiple services you need a mechanism to ensure data consistency across services. For example, If you are building a web store application where customers have a credit limit. The application must ensure that a new order will not exceed the customer’s credit limit. Since Orders and Customers are in different databases the application cannot simply use a local ACID transaction.
In this post, let’s look at the available options to implement data consistency across Microservices.
Cross-Posted : ContainerJournal
#1.Compensating Transactions Pattern
Implement each business transaction that spans multiple services as a sequence of local transactions. Each local transaction updates the database and publishes a message or event to trigger the next local transaction. If a local transaction fails because it violates a business rule then executes a series of compensating transactions that undo the changes that were made by the preceding local transactions.
There are two ways to do it:
- For each of the local transaction, publish domain events that trigger local transactions in each other services.
For example,If you’re building store application that uses this approach,it would create an order that consists of the following steps:
Order Servicecreates an Order in a pending state and publishes an ORDER_CREATED event
Customer Servicereceives the event attempts to reserve credit for that Order. It publishes either a
CREDIT_RESERVEDevent or a
Order Servicereceives the event and changes the state of the order to either
- Use an orchestrator kind of service to tell the participants what local transactions to execute
Order Servicecreates an Order in a pending state and creates a
ReserveCreditcommand to the
Customer Serviceattempts to reserve credit for that Order and sends back a reply
CreateOrderOrchestrationreceives the reply and sends either an
RejectOrdercommand to the
Order Servicechanges the state of the order to either
Issues and considerations
- Determination of when a step that implements eventual consistency has failed. A step might not fail immediately, but instead, it could block. It may be necessary to implement some form of time-out mechanism.
- Compensation logic is not easily generalized. A compensating transaction is application-specific; it relies on the application having sufficient information to be able to undo the effects of each step in a failed operation.
- Steps in a compensating transaction should be defined as idempotent commands. This enables the steps to be repeated if the compensating transaction itself fails.
- A compensating transaction does not necessarily return the data in the system to the state it was in at the start of the original operation. Instead, it compensates for the work performed by the steps that completed successfully before the operation failed.
- The order of the steps in the compensating transaction does not necessarily have to be the mirror opposite of the steps in the original operation. For example, one data store may be more sensitive to inconsistencies than another, and so the steps in the compensating transaction that undo the changes to this store should occur first.
- Consider using retry logic that is more forgiving than usual to minimize failures that trigger a compensating transaction. If a step in an operation that implements eventual consistency fails, try handling the failure as a transient exception and repeat the step.
To address these issues, eBay has developed GRIT protocol for globally consistent distributed transactions that combines ideas from optimistic concurrency control (OCC) , 2PC, and deterministic databases to achieve, for the first time, high-performance, globally consistent transactions across microservices with multiple underlying databases.
Devops Engineer Masters Program will make you proficient in DevOps principles like CI/CD, Continuous Monitoring and Continuous Delivery, using tools like Puppet, Nagios, Chef, Docker, Git & Jenkins. It includes training on Linux, Python, Docker, AWS DevOps Certification Training and Splunk. The curriculum has been determined by extensive research on 5000+ job descriptions across the globe.
In the next section, we look at eBay’s new protocol to address the above issues.
#2.GRIT: a protocol for distributed transactions
The following diagram illustrates the GRIT protocol in a microservices application with two databases.
To understand better, let’s look at key components that make up globally consistent distributed transactions.
- Global Transaction Manager (GTM): It coordinates global transactions across multiple databases. There can be one or more GTMs.
- Global Transaction Log (GTL): It represents the transaction request queue for a GTM. The order of transaction requests in a GTL determines the relative serializability order among global transactions. Persistence of GTLs is optional.
- Database Transaction Manager (DBTM): The transaction manager at each database realm. It performs the conflict checking and resolution, i.e. local commit decision is located here.
- Database Transaction Log (DBTL): The transaction log at each database realm that logs logically committed transactions that relate to this database (including single database transactions and multi-database transactions). The order of transactions in a DBTL determines the serializability order of the whole database system, including the global order dictated by the GTM. A log sequence number (LSN) is assigned to each log entry.
- LogPlayer: This component sends log entries, in sequence, to the backend storage servers for them to apply the updates. Each DB server applies log entries of logically committed transactions in order.
Now that we have understood the components, we look at the steps for a distributed transaction.
Steps for a distributed transaction
Following diagram shows the main steps for a distributed transaction
In GRIT, a distributed transaction goes through three phases:
- Optimistic execution (steps 1-4): As the application is executing the business logic via microservices, the database services capture the read-set and write-set of the transaction. No actual data modification occurs at this phase.
- Logical commit (steps 5-11): Once the application requests the transaction commit, the read-set and write-set at each database service point are submitted to its DBTM. The DBTM uses the read-set and write-set for conflict checking to achieve local commit decision. The GTM will make the global commit decision after collecting all the local decisions of DBTMs for the transaction. A transaction is logically committed once its write-sets are persisted in log stores (DBTLs) for databases involved. This involves minimum coordination between the GTM and the DBTMs.
- Physical apply (steps 12-13): The log players asynchronously sends DBTL entries to backend storage servers. The data modification occurs at this phase.
Issues and considerations
- GRIT is able to achieve consistent high throughput and serializable distributed transactions for applications invoking microservices with minimum coordination.
- GRIT fits well for transactions with few conflicts and provides a critical capability for applications that otherwise need complex mechanisms to achieve consistent transactions across microservices with multiple underlying databases.
Though there is no single approach that fits all, Cloud applications typically use data that is dispersed across data stores. Managing and maintaining data consistency in this environment can become a critical aspect of the system, particularly in terms of the concurrency and availability issues that can arise. You would need to trade strong consistency for availability. This means that you may need to design some aspects of your solutions around the notion of eventual consistency and accept that the data that your applications use might not be completely consistent all of the time.
Useful Resources :
Like this post? Don’t forget to share it!