Agile Integration and the End of Heavyweight Platforms

BLOG POST: Souciance Eqdam Rashti is a senior integrator at Pulsen Integration. In this blog post he talks about how the developer world moves from heavyweight platforms to fast, agile and lightweight integrations.

In 2007, when I started my career in the world of system integration, there weren't that many players in the system integration landscape. You had IBM, Microsoft, Webmethods, Tibco as well as smaller companies, and then open source frameworks like Apache Camel. Back then the core language support in Java, and .Net was not as strong as today. Javascript hadn't exploded and a cloud was just a thing you heard about in the weather report.

A lot of the big platforms, and some of the platforms that entered the scene later on, had one core idea in mind — to make it easier for the developer of the product to build integrations. Rather than writing boiler-plate code to connect to a database, it would handle that for you. You just worried about the business logic part. This was extremely beneficial, up to a certain point.

A move towards lightweight frameworks

However, as time progressed, and the rest of the tech world moved towards lightweight frameworks like spring boot, CI/CD processes, .Net becoming the powerful language it is and with the rise of devops, gitops and cloud, it left these heavyweight products in the rear seat. Why? Their focus was on simplicity of usage, and more often than not they were reactive rather than proactive.

Take IBM for example: their core integration product – IBM Integration Bus in version 10 and in the new App Connect Enterprise – is still based heavily on Eclipse and still lack some fundamental tooling you need from a powerful IDE. The debugger is extremely basic; the content-assist is non-existent, very few errors are caught at compile time and, more importantly, in 2019 there is still no support to build integrations in a test driven manner. Their main focus has been to build a stable and robust product that is easy to use. To credit IBM, it is a robust and easy to use product, but at the same time it is not a particularly flexible product and it suffers from the same weaknesses as some of the other major vendors.

Speed of development

Today, if I want to deploy a simple web app to the cloud, say Google App Engine or Azure App Service, it is extremely easy. There is built in support for most platforms so you don’t need to worry about infrastructure. It also allows me to work in an agile and test driven manner, and in most cases you can work in a gitops fashon — simply push your code to your repo and the cloud app service will build your app for you. If you go serverless, this becomes even easier. In a sense this is the future of building fast, agile and light-weight integrations. It gives the developers all the tools at their disposal whilst making it easy for them to focus on writing code and making deployment, test and logging flexible and easy to setup.

If I want to deploy an API app as a docker container to Google I can easily do this in a few lines of code and maven commands. There is no need for another layer of vendor package to add another layer of complexity for you. I highly doubt these packages will take off. For one, the speed of development in the cloud native world is so fast that the frameworks they are building the new hybrid platforms on may not even exist in 2-3 years' time.

At the end of the day, as a customer you need to look at your needs and your future. What do you want to focus your time and money on? Is maintenance of infrastructure, OS upgrades and application setup a core business for you? Do you want to release to production in one click several times a day? These are all relevant factors to consider when setting up your integrations, whether they be in the cloud or on-premise.

As a developer, I don't want to focus on single vendor or product. I would like to provide the best possible result for the customer and that means broadening my toolbox and embracing new technology, but using those that are mature and fit my problem at hand.

The age of agile integration does not mean we absolve responsibility in terms of architecture and maintenance. It's the opposite, we have now more freedom and more tools to decide more openly the kind of landscape we finally want to base our business on.

If you look at the future, my impression is that the major vendors are not going to increase any market share, especially if a large portion of the customers move to the cloud. If customer X moves most of their applications and infrastructure to Google or Azure, there is not incitement to keep a integration platform in your system landscape. All the system integration features are readily available in core languages and in the cloud services, and they can easily fit in your CI/CD process.

The right tools for the job

Of course, there will be instances where you will need a hybrid platform, or you need integration with a lot of legacy systems. But is there a future for larger integration platforms in 5 years? In 2 years? I doubt it.

The future lies in choosing the right tool for the job. One problem may require nodejs, another problem may require spring boot, a third may require .Net. Some customers need OAUTH security, others need Active Directory lookup, another wants interaction with nosql databases. As new technologies are used and added, so should your toolbox adapt to this dynamic landscape.

Now, some vendors have taken steps to build a flavor of agile or hybrid product which support kubernetes style deployments for container engines. Usually the product is based on open source frameworks like helm, ingress, kubernetes and others. There are problems with these, just like there are problems with the open source frameworks which they are based on. For one, the frameworks themselves are not completely mature yet and sometimes lack documentation and enough examples to get you by. To build a product on these is a risky move.

As a developer I need to worry about configuring lots of YAML files and read cryptic log messages. Besides, if I want to enter the world of containers, why not use the cloud for this? All the major vendors provide first class support to make it easy to deploy and scale your apps in a containerized environment.

How can agile integration improve your organization?

Choosing the right tools for the job is imperative to success. Developers at Pulsen are well versed in modern development methods and processes, and can help you select the frameworks that benefit your organization and your needs the most. Please contact my colleague Jacob Welsh, sales and marketing manager at Pulsen Integration, for more information.