Common pitfalls with integration

Integration is vital to practically all organizations. There is an increasing demand to make the data available and a constant need to access the data. Digitalization and automation require applications to interact.

Integration enables connection between the applications and, eventually, the capabilities these applications deliver.

However, integration errors may lead to an excessively complex and expensive landscape, hindering an organization’s delivery agility. This article points out some pitfalls that overshadow the bigger picture, along with relevant takeaways.

• Pitfall 1: Integration happens outside an application
• Pitfall 2: Integration as a “Commercial off the shelf” product
• Pitfall 3: Integration as purely a technology activity
• Pitfall 4: Missing out the user perspective

Pitfall 1: Integration happens outside an application

Often a business application requires data from another application or data source. Likewise, it may need to trigger a process or activity in another application. Consequently, there is a need for integration. Therefore, integration is not an individual entity outside an application. Its need is the application’s need. Treating it as something separate is a common mistake that has been around for many years and in many organizations.

Centralized integration that performs transformations and even worse domain-specific operations leads to all sorts of issues such as reduced scalability (both technical and team), unclear ownership of code and data, and reduced flexibility in terms of responding to change. A concrete example of such implementation is famous as Enterprise Service Bus (ESB). But the industry has learned. See here, here and here. There are limited occasions where such a centralized classic integration may be an okay solution if implemented carefully, e.g., when an organization has only third-party COTS software and no development teams. However, a single synchronization point leading to a centralized dependence both runtime and organizationally can reduce long term flexibility.

Keeping the smartness, for example, converting the data to some domain-specific format, filtering and enriching within the applications, leads to clarity on the ownership and accountability of such smartness. These may happen at the data source where an application provides some APIs that make the data available as per some standard. These may also occur at the consumer side, where an application receives data and applies relevant logic. An example of it would be ELT, where the transformation “T” happens within the ownership of the consuming application. There may still be a need for the roads (e.g., a streaming platform) so that the data could travel. But it is essential to keep such roads as simple as possible and free from domain-specific smartness. Let the applications handle anything domain-specific.

Keeping the domain specific integration logic inside the application provides a clearer separation of concerns and leads to a more scalable and flexible integration architecture.

Takeaway: Integration is not a standalone activity, let alone a separate team. The integration team is part of the application team, and so is the domain-specific integration logic a part of the application logic. However, a domain-agnostic integration platform can be an enabler. This “integration platform” can have, and perhaps must-have, a separate platform team that ensures the platform-as-a-service delivery model on behalf of the application teams.

Pitfall 2: Integration as a “Commercial off the shelf” product

Vendors keep marketing their integration tools to buy and magically solve the problems. A company not critically looking at these marketing tactics will be making dangerous and costly mistakes that could take years to recover.

Buying an integration tool is just a first step. A company needs to establish ownership, work out an operating model around the tool, and access the long-term consequences in the ecosystem. For example, how will it affect the time to market if there are issues with the integration tool and the vendor is slow to respond with a patch? Or, how sound is the exit strategy or puts it the other way: how flexible is the overall architecture and technology choices if there should be a need to replace it with something else?

An integration tool is not like other software products. Just buying it and deploying it in the production environment will not be sufficient. Therefore, the second step will be working on it and living with it. Application teams will need to work with it and possibly adjust some of its internal implementations as per the tool. Doing this will require person-hours and costs. So, a COTS integration tool’s capital and operating expenditures will escalate. In the worst case, such adjustments may lead to reduced flexibility in the application. Even more dangerous will be the resurrection of Pitfall 1 if the architects, engineers and other decision-makers are not careful.

Takeaway: View COTS products for integration as enablers and not the final product. The running cost and efforts with such products can be considerable. Suppose it is a part of a transformation, and some similar legacy technology needs to be phased out. It will then be essential to consider the cost and effect of potential changes in integration patterns for the applications and application teams. Therefore assess the overall benefit and ROI given its financial and effort costs.

Pitfall 3: Integration as purely a technology activity

Implementation of integration happens on a technical level. But its benefits and ROI are actually on the business value level and capability level.

Capturing the value chain of the business helps map out a set of activities to provide valuable products and services to the target market. These activities require business capabilities. A business capability refers to a combination of resources necessary to reach the target state. These resources are not just technologies or software. These are also people, organizational structures, assets, collaborations, contracts and other things that provide value. A successful journey to the target state requires a combination of these multiple parts. As such, capabilities and their components are often interrelated. This inter-relation and combination at various value stream stages help the business deliver.

Integration is at the center of this combination. It provides a technical baseline for the teams to collaborate, exchange data, and create a stronger value proposition through combined capabilities.

Consider following business capabilities, as an example: “Prospecting new customer”, “Pricing” and “Credit Evaluation”. While looking out for new customers, it may be required to perform credit evaluation of the prospects. Likewise, a lookup of pricing and the possibility for discounts can be convenient. The credit score of each customer can also help provide more tailored pricing. So, while these capabilities can exist independently, they will generate more tangible outcomes when used together.

Decomposing capabilities into lower levels and uncovering the interrelation and building integrations enhances the overall value proposition.

Integration can also lead to more cost-efficient operations. There may be duplication of essential functions throughout the business unit and units, which is neither cost nor resource-efficient. The integration enables business units and teams to provide such functions as a service, creating reuse and cost reduction opportunities. Internal consumers in an organization, therefore, becomes significant stakeholder.

Takeaway: Look away from implementation-specific tunnel vision and consider the whys from capabilities and business value perspectives. By generating synergies between the capabilities, the business enhances its ability to reliably and consistently deliver specific business outcomes. An organization can benefit long-term and wholly by viewing the integration as a business enabler and growing out from a technology-only enclosure.

Pitfall 4: Missing out the user perspective

The user perspective significantly impacts the technology and integration patterns choice.

First, let us consider the end-users. Thanks to the proliferation of faster internet, advancements in mobile computing, and the digitalization of services, instant gratification is one of the increasing demands of most end-users. This demand alone highlights the need for digital transformation and end-to-end digitalization with APIs and near-real-time streaming solutions. Implementing such changes in a large organization with a long history and large legacy base will take time, cost and careful prioritizations in the transformation roadmap.

Second, the user here is not necessarily only the end-users. The internal users form a large part of it. This Gartner article (requires paid membership) coins four personas that primarily make up internal users: Integration specialists, Ad-hoc integrators, Citizen integrators, and Digital integrators.

Integration specialists and business users (internal and external end-users) are involved in the entire value stream. But ad-hoc integrators are more focused on application integration around specific parts of the value stream.

Integration specialists are data integration architects, engineers and other technical people involved in providing the platform and data transfer services. It is essential to understand that this group of users is tiny, and putting too much responsibility on them (consider cognitive load) will lead to scalability issues and reduced quality. Integration strategy that avoids pitfall 1 will help them focus on the platform as a service while offloading domain concerns to ad-hoc integrators.

Ad-hoc integrators are usually application teams, and citizen integrators are business users. Application teams sit with the development of customer and end-user facing applications. In the Gartner article, business users represent internal IT users, but this category can also include external end-users. The requirements of these groups reflect the business requirements more accurately. Therefore, the ability in integration architecture to scale and quickly adapt to these users’ changing needs is critical. Addressing the needs of citizen integrators require the architects and decision-makers to balance the prioritizations and carefully create a holistic integration target state roadmap.

Takeaway: Considering the internal and external users helps identify what truly matters. A careful evaluation of value from the user’s perspective and prioritization of user needs sets the scope and baseline for milestones in the roadmap. Decisions such as how real-time the data exchange needs to be, what services and data can be internally available, and what needs to be public boil down to user requirements and expectations relative to the market.

Conclusion

Technology is important. So are APIs, event streams and other integration patterns. The COTS integration tools can also be helpful. But they do not yield any benefits on their own. They are costly and miserable to live with when misused, especially without adequately understanding the users and their requirements. Therefore, focus on improving business outcomes and always start with a user-first principle. Consider both short-term and long-term benefits. While long-term benefits help justify the target state, short-term benefits deliver quick wins. Integration is also development; hence, continuous assessment and the flexibility to pivot to a different approach and pattern are equally applicable.

Note that these pitfalls are guidance on what to avoid. The takeaways provide recommendations. Both business value and user requirement contexts play a crucial role in determining the extent to which each of these pitfalls is relevant.