SOA goals and realities
SOA has been long been the holy grail of Increased Business Agility and Cost Reduction which should be achieved through Increased Usability, Improved Maintainability and Redundancy Reduction.
Common SOA Realities
You have spent a considerable amount of money on technical infrastructure (e.g. Enterprise Service Bus) trying to become more aligned to your business, but you have most likely arrived at the following realities:
- Systems are more fragile
- Development and Maintenance costs are higher
- Your services are not being reused
- You thought SOA would solve your integration problems, but they’ve gotten worse
- No one want to build or maintain services
- System performance is worse
Why did it come to this? The short answer is that most organizations are still doing point-to-point integration even if they are using ESB’s in the belief that the tool will provide them with the needed de-coupling. In other words, we have not changed the way we approach integration, but just added more, expensive and complex technology to our solution stack.
To understand why SOA was introduced and why your Service landscape looks the way it does today we need to look at a bit of history first.
Before System Integration became the norm
Most applications built or bought during the 1980’s and 1990’s were designed as silo applications that lived in their own world. They often didn’t consider integration needs apart from what could be fulfilled through their User Interface.
To complete a whole business workflow (e.g. a customer purchases goods which gets invoiced, the inventory count for the goods sold are reduced and finally the shipping of the goods are handled) employees typically had to manually interact with each other using paper handovers. Systems typically didn’t have the capability to talk to each other through other means than human to human interaction.
Simple System Integration – Database integration
Automation of manual work processes became the next goal and this required that applications could talk to each other. Most systems developed before year 2000 didn’t provide system integration interfaces, so people naturally turned to what was possible, which was usually database based integration.
Database integration means that applications talk to each other through their respective databases. This means the applications need to be granted access to each other databases so they could share data and perhaps react to database triggers.
This was the simplest thing possible at that time, but as time went on people started to realize how brittle this integration contract was. Database based integration might seem simple and safe, but it will turn on you fast and leave you with a big maintenenace nightmare if you’re not careful.
The temporal coupling between databases can in worst case render all applications unavailable just because one key database is unavailable!
Enterprise Application Integration (EAI)
As the problems and limitations of Database integration became obvious Application Vendors and Developers started opening up their application allowing (typically) Synchronous Integration through Open Host Interfaces (Corba, RMI, DCOM, SOAP, etc.). This allowed one application to request information from another application or to instruct it to perform a task on behalf of the calling application.
EAI has many advantages over simple Database integrations, but it also comes with its own share of problems. When ever an integration point (interfaces) changes all other applications are required to change as well. Problems with interface versioning soon becomes a problem and an explosion of point-to-point integrations doesn’t help either.
Since most integrations are synchronous, ie. the calling application has wait for the other application to respond, it leaves you open to problem of latency (the period from when you’ve called another application until it responds), lack of cross application transactions (what should the calling application do if it never receives a response from the other application it calls) and a high degree of coupling (aka. temporal coupling where the problem of one application affects all other connected system, possibly rendering all applications unavailable).
WebServices to the rescue
It was rare two applications spoke the same language and shared the same technical platform that would allow for easy integration. Integrating DCOM with Corba was far from simple. You would need to handle different communication protocols and different interpretations of data types. The response to this was the “invention” of XML and later SOAP (over HTTP), which allowed all applications to exchange information and interact using the same conventions and protocols.
WebServices made application integration much easier and soon corporations experienced an explosion of point to point integrations that quickly became unmanageable.
Enter the Enterprise Service Bus (ESB)
Ad hoc point-to-point service integration is hard to manage. A large number of synchronous service integrations results in an unflexible system landscape with low reuse and poor stability (partly due to uncontrolled service evolution and partly due to synchronous communication). Instead of enabling business agility and cutting costs then corporations typically were experiencing much higher costs and lower performance. Something clearly had to be done so all the big vendors decided that an Enterprise Service Bus (ESB) would be solution to all our problems, but what most didn’t realize was that early ESB’s were just old EAI solutions re-labeled!
At the surface ESB’s seemed like a good idea. They could handle different protocols and formats and they allowed for a nice decoupling between consumers and producers. Through this decoupling layer we could enforce rules, apply security and log all communications. The logging would of course only cover the ESB and not be end-to-end logging (with correlation) and enforcing security at the ESB only makes sense if your network effectively hinders consumers to communicate directly with producer.
The problems related to synchronous communication (latency, temporal coupling) are not solved by the ESB, in fact the ESB only makes latency problems worse.
One good thing about the ESB, at the surface level we no longer have point-to-point integrations. Everything goes nicely into the ESB where we can monitor service usage, etc.
One of the problems with SOA is that it’s so many to so many people. So to frame what we mean by SOA, we will define in terms of the 4 tenets of SOA:
- Boundaries are explicit
- Services are autonomous
- Services share schema and contract, not class
- Service compatibility is based on policy
Point 1 – Boundaries are explicit
Boundaries are explicit is another way of saying that a Service is defined by its Boundary. A boundary tells us what parts make up the service and how the services interacts with the outside world (as defined by the service contract – see point 3 and 4). The internal implementation and platform of a Service (what’s inside the boundary) is irrelevant. The only way to interact with a service is by way of message exchange through its public endpoints.
Point 2 – Services are autonomous
The common interpretation of Autonomy is that services are deployed and versioned independently from each other.
To us, autonomy means a lot more than just this:
Without this kind of service autonomy, one failing service can cause a domino effect where other services, which depend on this service, will fail!
This means we need to eliminate temporal coupling between services by limiting communication to use asynchronous messaging.
Point 3 – Services share schema and contract, not class
Point 3 covers interoperability and how this can be achieved without coupling consumers of services to the producers technical platform. It also defines that the contracts describe the structure of messages exchanged between consumer and producer and allow for machine verification. Contracts and Schemas must remain stable over time, as changes to contracts are costly.
Point 4 – Service compatibility is based on policy
Policies define the rules which govern service interaction. Policies may cover both requirements and capabilities, such as Encryption, Bi-directional security, delivery guarantees, etc.
How to approach Real SOA?
With these more precise and well defined SOA tenets we have a really good guideline for how to define and design our services.
The need for loose coupling and autonomy, to enable Business Agility, calls for a different approach to SOA than what’s commonly practiced.
The keywords are asynchronous messaging, event driven architecture (EDA), event driven business processes, sagas, service mashups.
What’s important is to be pragmatic and decide which solution (EAI, Real SOA, etc.) works best for the different integration scenarios.
So how does this relate to microservices? If you look at the tenets of SOA and look at the qualities around Microservices, you will see that Microservices basically form a SOA Delivery model.