Microservices: It’s not (only) the size that matters, it’s (also) how you use them – part 3

April 23, 2014

Categories: Domain Driven Design, Enterprise Architecture, Event Driven Architecture, Microservices, SOA, Software Development, System Integration

Tags: Cohesion, communication patterns, Coupling, Data model, DDD, Distributed computing, Domain model, EDA, integration patterns, JSON, Layered SOA, Microservice, MSA, REST, Service model, SOA, WebService

by Jeppe Cramon

Danish version: http://qed.dk/jeppe-cramon/2014/04/22/micro-services-det-er-ikke-kun-stoerrelsen-der-er-vigtigt-det-er-ogsaa-hvordan-du-bruger-dem-del-3/

Part 1 – Microservices: It’s not (only) the size that matters, it’s (also) how you use them
Part 2 – Microservices: It’s not (only) the size that matters, it’s (also) how you use them
Part 4 – Microservices: It’s not (only) the size that matters, it’s (also) how you use them
Part 5 – Microservices: It’s not (only) the size that matters, it’s (also) how you use them
Part 6 – Service vs Components vs Microservices

In Microservices: It’s not (only) the size that matters, it’s (also) how you use them – part 2, we again discussed the problems with using (synchronous) 2 way communication between distributed (micro) services. We discussed how the coupling problems caused by 2 way communication combined with micro services actually result in the reinvention of distributed objects. We also discussed how the combination of 2 way communication and the lack of reliable messaging and transactions cause complex compensation logic in the event of a failure.
After a refresher of the 8 fallacies of distributed computing, we examined an alternative to the 2 way communications between services. We applied Pat Hellands “Life Beyond Distributed Transactions ? – An Apostate ‘s Opinion” (PDF format) which takes the position that Distributed transactions are not the solution for coordinating updates between services. We discussed why distributed transactions are problematic.

According to Pat Helland, we must find the solution to our problem by looking at:

How do we split our data / services
How do we identify our data / services
How do we communicate between our data / services

Section 1 and 2 were covered in Microservices: It’s not (only) the size that matters, it’s (also) how you use them – part 2 and can be summarized:

Our data must be collected in pieces called entities or aggregates (in DDD terminology).
Each aggreate is uniquely identifiable from an ID (for example a UUID / GUID).
These aggregates need to be limited in size, so that they after a transaction are consistent.
The rule of thumb is: 1 use case = 1 transaction = 1 aggregate.

In this blog post we will look at section 3 “How do we communicate between our data / services”

How should we communicate between our data / services?

As we mentioned several times before, using 2 way (synchronous) communication between our services causes hard coupling and other annoyances:

It results in communication related coupling (because data and logic are not always in the same service )
- It also results in contractual-, data- and functional coupling as well as high latency due to network communication
Layered coupling (persistence is not always in the same service )
Temporal coupling (our service can not operate if it is unable to communicate with the services it depends upon)
The fact that our service depends on other services, undermines its autonomy and makes it more unreliable
All of this results in the need for complex logic compensation due to the lack of reliable messaging and transactions.

Reusable service, 2 way (synchronous) communication and coupling

If the solution is not synchronous communication, the answer has got to be asynchronous communication?

Yes, but it depends on …… 🙂
Before we dive into the details of what it depends on, let’s first look at the characteristics of synchronous and asynchronous communication:

Based on these characteristics, we briefly categorize the forms of communication as follows:

Synchronous communication is two way communication

The communication pattern visualized in the drawing above is called the Request/Response and is the typical implementation pattern for Remote Procedure Calls (RPC).
With the Request/Response pattern a Consumer sends a Request message to a Provider. While the Provider processes the request message, the Consumer can basically only wait* until it receives a Response or an error (* some might point out that consumer can take advantage of asynchronous platform features so that it e.g. can perform several calls in parallel while waiting, etc. This unfortunately does not solve the temporal coupling between the Consumer and Provider – the Consumer simply cannot continue its work BEFORE it has received its Response from the Provider). The typical execution path flow of Request / Response or RPC is visualized in the diagram below.

Synchronous communication flow

As shown in the drawing, there is a strong coupling between the Consumer and the Provider. The Consumer can not perform its job if the Provider is unavailable. This type of coupling is called temporal coupling or runtime coupling and is something we should minimize between our services.

Asynchronous communication is one way communication

With asynchronous communication, the Sender transmits a message to a Receiver over a (transport) channel. The Sender waits briefly for the channel to confirm the receipt of the message (so seen through the eyes of the Sender sending the message to the channel often occurs synchronously), after which the Sender can continue its work. This is the essence of one way communication and the typical execution path flow is visualized below:

Asynchronous communication flow

Asynchronous communication is also often called messaging. The transport channels in asynchronous communication are responsible for receiving the messages from the sender and responsible for delivering the messages to the recipient (or recipients). The transport channel, so to speak, assumes responsibility for the message exchange. Transport channels can be both simple (eg. supported with Sockets ala 0MQ) or advanced distributed solutions with durable Queues / Topics (as e.g. supported by ActiveMQ, HornetQ, queuing, Kafka, etc.). Messaging and asynchronous communication channels offer different guarantees that govern the message exchange: Guaranteed delivery and Guaranteed message ordering.

Why isn’t asynchronous communication the full solution?

In reality it is, but sadly it is not as easy as asynchronous versus synchronous. The integration pattern used between our services determines the actual coupling level.

If we’re talking true asynchronous one way communication then we are on target with regards to most cases.

The challenge is that two way communication comes in several forms /variations:

Remote Procedure Call (RPC) – synchronous communication
Request / Response – synchronous communication
Request / Reply – also known as synchronous over asynchronous communication.

I have repeatedly seen projects that used Request/Reply (synchronous over asynchronous) to ensure that their services were not temporally coupled. The devil is, as always in the details.
Seen from the Consumer, and for most applications of the Request/Reply pattern, there is not a big difference in the degree of temporal coupling between RPC, Request/Response or Request/Reply as they are all variants of two way communication that forces our sender to wait for a response before it can proceed:

So what is the conclusion?

The conclusion is that 2 way communication between services is the root of many problems and these problems are not getting any smaller by making our services smaller (microservices).
We have seen that asynchronous communication can break the temporal coupling between our services, but ONLY if it takes place as a true one-way communication.

Our solution 🙂

The question is how do you design services (or microservices) that basically only need asynchronous one way communication between each other (communication between UI and services is another matter, which we will soon enter)?

In the next blog post we will look at how we can divide up our services and how they can communicate with each other via asynchronous one way communication.

Appendix over Message guarantees

Guaranteed delivery

Guaranteed delivery covers the degree of certainty that a messages will be delivered from the sender to the receiver. These delivery guarantees are typically implemented by a reliable messaging infrastructure.

At Most Once

With this delivery guarantee the Receiver will receive a message 0 or 1 time. The Sender guarantees that message is sent only once. If the Recipient is not available or able to store data related to the message (e.g. due to an error), the message will NOT be resent.

At Least Once

With this delivery guarantee, a message is received 1 or more times (i.e. at least once). The message will be resent until the channel has received an receive-acknowledgment from the Receiver. This means that the message can be received more than once. Lack of acknowledgment from the Receiver can be due to unavailability or failure. Repeated delivery of the same message can be handled by making the recipient’s handling of the message idempotent (idempotence describes the quality of an operation in which result and state does not change, if the operation is performed more than 1 time).

Exactly Once

This delivery guarantee ensures that messages are received exactly once . If the recipient is not available or is not able to store data related to the message (e.g. due to an error ), the message will be resent until an acknowledgment of receipt has been received. The difference from “At least once” is that the delivery mechanism is controlled through a coordinating protocol that ensures that duplicate messages are ignored .
Some of the protocols used to implement Exactly Once delivery guarantee is WS- ReliableMessaging and various implementations of 2 Phase Commit .

Another way to obtain the same as qualities, is to use operations that are idempotent and combine this with the At Least Once delivery guarantee.

Supporting idempotence almost always requires that there is a unique identifier / message ID in every message, such that the receiver can use this ID to verify if the message has already been received and processed. The unique identifier could for instance be a GUID or timestamp. Some operations may satisfy idempotence without requiring a unique identifier (such as Deleting an Aggregate).

Guaranteed message ordering

Guaranteed message ordering, also known as “In order delivery ” ensures that messages are received in the order they were shipped. Guaranteed message ordering can be combined with above mentioned delivery guarantees.

Where guaranteed delivery focuses on delivery of each message, guaranteed message ordering is concerned with the coupling or the relationship between several messages.

Challenges around the message ordering will occur if one or more of following circumstances occur:

There are multiple paths through the channel ( multipath ) e.g. introduced by clustering, a load balancer and other fault tolerance built into the network and infrastructure.
Dead letter queues. If a message is placed on a Dead Letter queue, due to problems with delivery or handling, it gives us a challenge as to how this error should be handled and what should happen with subsequent messages until the failed message is delivered.
Clustering of sender or receiver poses the challenge that messages might not be delivered to the channel in the right order or not processed by the receiver in the same order they were delivered.

Pascal

May 16, 2014 at 21:20

I like your piece on one way communication. I have difficulty with the channel concept mentioned here. In what regard is a channel not similar to a micro service? Would it not have been simpler to combine the channel behavior into the receiver?
The sender uses a synchronous call to deliver its event. See the third diagram. It could this just as easily to the receiver. So the change address event goes straight to the receiver which does not act on the event by changing its entity but first confirms the receipt of the event. Internally it can subsequently proces the event and send its own event, I.e. I-changed-the-address event.

Jeppe Cramon

May 17, 2014 at 23:02

The channel is an abstract concept, which can be backed by several different implementations.
Classically the channel would be supported by a Queue (1-1 messaging) or a Topic (1-m messaging) which is support by e.g. JMS. This concept is know as a smart pipe with dumb endpoints. The opposite of this is smart endpoints and dumb pipes, which is what I believe you’re talking about.
For 1-1 asynchronous messaging, with reliable message delivery (i.e. at least once delivery), I would not call directly from the senders active thread to the receiver. The receiver could be unavailable and I don’t want to burden my senders business logic concerns such as retries and durability. I also don’t want to render my own service unavailable just because the receiver is down.
Instead I would hand off the message to a component inside my service that is responsible for sending messages reliably. This component could delegate the responsibility to e.g. a JMS queue (or MSMQ queue). Or it could store the message in some durable way (e.g. on the file system, a database/document store, etc.) and then it self read the message from there, attempt to deliver it to the receiver (on another thread) again and again until the receiver acknowledges the message as being received and handled.

I hope this make it more clear?

/Jeppe

Bill T

June 4, 2014 at 14:50

So, what is the lesson? How they can communicate with each other via asynchronous one way communication?

June 5, 2014 at 10:28

For many integration cases one way communication will result in a much looser form of coupling (which of course is also associated with a certain cost).
In Part 4, which has just been posted, I go deeper into how we can use one way communication to integrate service and drive our business processes.
I hope it answers why one way communication would be something worth considering.

June 12, 2014 at 03:23

Just followed up on part 4. It does indeed. We are actually looking at a couple of architectural tweaks as a result.

Very good series. Kudos!

Paulo Merson

May 5, 2015 at 22:23

Another great post, raising the key question at the end. I have a few comments:

1) The text says “The challenge is that two way communication comes in several forms”. The problem is that the three bullets that follow are not three forms of two-way communication. From an architectural perspective, the generic connector in discussion here is called request/response, aka request/reply, aka call-return. Then there are variations, which are orthogonal: (i) the interaction may be local (e.g., in-vm method call) or remote (e.g., RPC, SOAP request-response MEP); (ii) the caller always wait for the conclusion of the callee, but it may or may not be blocked; (iii) the interaction may be implemented via a mechanism that is natively synchronous (e.g., in-vm method call, SOAP request-response MEP) or it may use sync over async.

2) The Appendix about message guarantees fails to explain that these guarantees are typically implemented by some reliable messaging infrastructure. I wouldn’t expect the developer of the business logic to implement these guarantees. But I’d expect the architect of the solution to know what guarantees are appropriate for a particular service consumer-provider interaction, and how to set it up in the infrastructure and in the business logic (e.g., idempotence for at-least-once).

3) Your description of idempotence will be more precise if you remove the word “even”, because state may change the first time the operation is performed.

September 25, 2015 at 19:18

Hi Paulo

Thanks for your comments 🙂

1) I was hoping it was clear that I was talking purely about remote communication (in the light of service to service communication). You’re right, they are all synchronous (i.e. the caller waits) – and forms is probably not the right name to describe the names of the different communication patterns.

2) I’ve added a comment about this. Thanks.

3) Thanks, I’ve removed “even”.

Chris Pitman

September 28, 2015 at 03:46

Doesn’t Request/Reply remove temporal coupling as long as the consumer either:

1) Persists any information about the ongoing conversation, then continues on to the next task it must perform. It may die and restart however many times, but at some point it processes the reply.

2) Relies on the guarantees of the channel/provider to handle the request, and keeps no state regarding the request. In this case, the reply must have enough information for the consumer to continue processing, but again it does not matter when the reply is received.

September 28, 2015 at 08:56

Hi Chris,

1+2) It sure does 🙂

The reason why I included request/reply (sync over async) here was in the context of service to service integration where you have a user waiting on the other end for a response.
In case we have a user waiting for a response (e.g. in a browser) doing request/response (sync) or request/reply (sync over async) has the same level of temporal coupling.
If you don’t have a user waiting directly (e.g. for a page to render the response) and you can’t avoid having to split the business task between two or more services, then request/reply is a very useful pattern for e.g. delegating a (long running) sub-task to another service. Sometimes request-reply is the right pattern, other times a Command->Event or an Event->Event pattern will make more sense because it aligns better with the nature of the business issue – see blog post 5.