SOA – Hierarchy or Organic growth?

SOA – Hierarchy or Organic growth?

Note: This is the english version of Jeppe’s qed.dk blog post

There is an interesting change in the approach to SOA and it comes from an unexpected area.

I have worked with SOA (a great acronym which really says nothing as it has been watered out) for over 10 years and through my time I have seen companies build more and more hierarchy (or rather layers) and structure into their SOA landscape as a way to achieve “business agility through business and IT alignment and increased recycling”. If you tend to get a gag reflex by hearing the phrase, I feel you.

The idea and the intention is good, but the execution many places is to the grade F-. Instead of having achieved agility through loosely coupled services, these companies instead end up with hard-coupled services, an expensive and cumbersome organizational Enterprise Service Bus (ESB), which is placed centrally to coordinate the hundreds to thousands of services we have built. It is a bit like herding sheep with a chest freezer and a rope tied to each sheep.
Let me make it clear, I don’t believe SOA, nor the ESB, is to be blamed for these failures. In my experience it has been the way they were implemented and deployed that cause the majority of problems.

Herding sheep using a chest freezer and rope

Herding sheep using a chest freezer and rope

It’s messy and unstable. Not unstable as in the ESB is unstable, but unstable in that we’re desperately trying to create order out of chaos without understanding the mechanisms of action.
Let me make it clear, I don’t believe SOA, nor the ESB, is to be blamed for these failures. In my experience it has been the way SOA were implemented and deployed that has caused the majority of problems.

Everyday life in any major company offers a torrent of new service contracts , amendments to existing service contracts. New systems are purchased and systems are being built to replace the old systems ( with the result that there is now one extra new system in addition to the old system that ends up still being in operation). The growth is organic , but the way we handle it is by introducing more hierarchy , bureaucracy and coordination. It is the forces of nature that rule and govern and as you know, if you choose to pee against the wind you get wet.

So what do you typically do to control this disorder? You hire Enterprise Architects who can draw stylized diagrams of how the organization and the systems should be. You hire data architects who can draw canonical models across the business in the hope that we have time and money to ensure that everyone has the same understanding of concepts such as a customer. Most of it is has proven to be a waste of money and time.

What else do we do? We make a central organizational unit will be responsible for managing the ESB and who heroically (they really fight to make this work) ensure that services can talk to each other and new integrations gets built. We have inserted an intermediary who needs to understand the business , service the customer and service providers needs and requirements. That is a lot to expect of an intermediary.

All the control and influence can very easily end up with the ESB group and thus form an organizational- and project related bottleneck. This is where responsibility / or lack there of lives and it is here that we quietly end up moving more and more of our business logic in the form of orchestrations. Slowly our services gets transformed to CRUD (Create / Read / Update / Delete) Data Services. Orchestrations  definitely has its place, but was I’ve seen is that the nature of these orchestrations become unmanageable and entangled.

But surely it cannot be bad? We achieve high levels of (data) service reuse and isn’t that ‘s why we wanted SOA in the first place?
How many can recognize the following SOA (architecture) picture ?

Layered SOA ("best practice")

Layered SOA (“best practice”)

What if we drew the same responsibilities in this way?

Layering with a Database

Layering with a Database

In my eyes both solutions achieve the same result with approximately the same high degree of data / logic coupling. The SOA solution has a slightly weaker technical coupling, at the expense of lower performance and more complex transaction handling. This increases the cost and makes SOA solution much more technically complicated. We have virtually replaced the database transaction handling and table joining functionality with SOAP , XML and HTTP. And no, the solution is not to make the database a key player again . The whole problem has been turned upside down and there are much better solutions.

Who was it that suggested the layered SOA should be best practice when it contributes to high coupling ( both in data-wise and temporally), high latency and heavy technical baggage in order to handle updates of multiple autonomous services?

In the next blog post I will delve more into the problems of layered SOA, the challenges of the centralized ESB and how a different approach to the design and split of responsibilities perhaps can help.
But what about the interesting development, what it is and where it comes from? This will be covered at the end of the next blog post 🙂