A field guide to integration · 1960s to today
Joined up: how integration architecture evolved
No business runs on one system. Orders, stock, invoices, customers and shipments live in different software, and they all have to agree. For sixty years we’ve kept inventing new ways to connect them. Each new style fixed the worst problem of the one before, and each brought problems of its own.
Sixty years at a glance
its heyday still in use
- Why integration at all?
- Batch jobs and file transfer
- Point-to-point and shared databases
- Message-oriented middleware
- EAI: hub and spoke
- SOA and the enterprise service bus
- Web APIs and REST
- API-led connectivity
- Microservices
- Event-driven and streaming
- Cloud, SaaS and hybrid
- Serverless
- Today: everything at once, plus AI agents
- All twelve side by side
- There is no final architecture
Before the timeline
Why integration at all?
Software is bought and built one job at a time. Finance buys an accounting package, the warehouse gets a stock system, sales get a CRM, and marketing signs up for three SaaS tools before lunch. Each one is good at its job, and each keeps its own copy of the facts it needs: who the customer is, what they ordered, what’s on the shelf.
Integration is the work of making those systems agree. Without it, people fill the gap: they re-key orders from one screen into another, export spreadsheets, and reconcile at month end. When it’s done badly, a change to one system breaks three others that nobody knew depended on it.
It has got harder every decade, under three pressures:
- More systems. Packaged software, mergers, then SaaS. A large company now runs hundreds of applications.
- Less waiting. From next-day reports to parcels tracked by the minute.
- More outsiders. Suppliers, partners, customers on phones, and now AI agents acting for people.
Six questions every integration answers
Whatever the decade, connecting two systems means answering the same questions. The eras below differ mostly in where they put the answers: in each application, in a central product, or in shared standards.
- Transport
- How do the bytes get there? A tape, a socket, a queue, HTTP.
- Format
- Do both sides agree what an “order” looks like?
- Routing
- Who needs to know? Who decides that?
- Timing
- Does it need an answer now, or only to happen eventually?
- Failure
- What happens when the other side is down?
- Trust
- Who is allowed to ask for what, and who checks?
The arithmetic behind all of it
Connect every system directly to every other, and the number of links grows with the square of the number of systems: n(n−1)/2. Put something in the middle, and each system needs only one connection. Drag the slider.
links
Every system to every other
links
Everything through one hub
01 1960s–1980s
Batch jobs and file transfer
Programs share data the way offices shared paper: one writes a file, and another picks it up later.
Computers were expensive mainframes, run by operators in a machine room. Applications were written for one department at a time, and the natural way to pass data between them was a file: a tape, later a dataset on disk, written by one job and read by the next in a nightly schedule. Between companies, the same idea became electronic data interchange (EDI): orders and invoices sent as files in a strict, agreed layout.
What it solved
- Systems written separately could share data without knowing anything about each other’s code.
- Huge volumes processed cheaply, overnight, while the machine was otherwise idle.
- A simple audit trail: the file is the record of what was sent.
What it broke
- Everything is a day old. Stock, balances and addresses are only as fresh as the last run.
- The file layout is the contract. Widen one field and the next job quietly reads rubbish.
- Failures surface late: a missing file at 2 a.m. means a manual rerun before the morning.
Milestones
- 1964 IBM’s System/360 brings one compatible family of mainframes to business.
- 1971 FTP is first specified (RFC 114), for moving files between networked computers.
- 1979 ANSI charters the X12 committee to standardise EDI in the United States.
- Late 1980s The UN’s EDIFACT standard does the same internationally.
Go deeper: why batch never went away
Banks still settle in end-of-day files; retailers still send suppliers EDI; data warehouses were loaded by nightly ETL (extract, transform, load) jobs for decades. Gregor Hohpe and Bobby Woolf call this style file transfer, the first of their four integration styles: file transfer, shared database, remote procedure invocation and messaging.
Its virtues are real. It’s simple, it decouples systems in time, and it handles very large volumes efficiently. Its costs are latency and a contract (the record layout) that nothing checks until something reads it. Modern descendants include SFTP drop folders, files landing in cloud storage buckets, and the bulk APIs of SaaS products.
What came next: once terminals and networks put users online, a day-old answer wasn’t good enough. Systems needed to ask each other things now. Point-to-point →
02 1980s–1990s
Point-to-point and shared databases
Online systems want answers straight away, so each one calls the others directly, or they simply share a database.
Minicomputers, then client–server networks, spread computing out of the machine room, and departments bought their own systems. When the order screen needed to check stock, the quickest fix was a direct link: a remote procedure call, a socket, a file for just this one pair, or reading the other application’s tables. Each link was built by whoever needed it, in whatever way was easiest that month.
What it solved
- Real-time answers: check stock while the customer is still on the phone.
- Quick to build: one link, one project, one team.
- Remote procedure calls made a call across the network look like an ordinary function call.
What it broke
- The spaghetti: up to n(n−1)/2 links, each one different and few of them documented.
- Tight coupling: if Stock is down or slow, so is the order screen.
- Shared databases tie applications to each other’s internal tables, so neither can change them.
- Nobody sees the whole picture. Every change starts with “who else uses this?”
Milestones
- 1984 Andrew Birrell and Bruce Nelson describe how to implement remote procedure calls.
- 1988 Sun’s RPC protocol is published as RFC 1057.
- 1991 CORBA 1.0, from the Object Management Group: distributed objects across languages.
- 1996 Microsoft’s DCOM extends COM objects across the network.
Go deeper: the trouble with making the network invisible
RPC’s promise, that a remote call could look just like a local one, was also its trap. The “fallacies of distributed computing” (the network is reliable, latency is zero, bandwidth is infinite, and so on) were collected at Sun Microsystems in the 1990s precisely because systems built this way kept assuming them.
CORBA and DCOM added interface definition languages and object brokers, but they were complex, struggled to cross firewalls, and didn’t work with each other. Shared databases deserve a special mention: they feel like integration for free, but the schema quietly becomes a public interface that no one owns and everyone is afraid to change.
What came next: something had to sit between the systems, first to carry messages reliably and then to translate them. Messaging →
03 1990s
Message-oriented middleware
Instead of calling each other, systems leave messages on queues. The sender doesn’t wait, and the receiver doesn’t have to be running.
Banks and trading floors hit the limits of direct calls first: prices had to reach hundreds of screens at once, and a payment couldn’t be lost because a server restarted. Message-oriented middleware (MOM) put a reliable post office between applications. A queue manager accepts a message, stores it safely and delivers it when the receiver is ready. Queues carry instructions to one receiver; publish and subscribe topics broadcast to many.
What it solved
- Decoupled in time: the sender carries on, and the receiver catches up when it’s back.
- Guaranteed delivery over unreliable networks, with transactional queues.
- Load levelling: a burst of orders waits in the queue instead of overwhelming a system.
- One message can reach many subscribers.
What it broke
- Messaging moves bytes; it doesn’t translate them. Every pair still has to agree on a format.
- Routing (“who needs this?”) is still written into each sender.
- Hard to debug: messages disappear into queues, and one bad “poison” message can block the rest.
- Proprietary products that didn’t talk to each other.
Milestones
- Mid-1980s Teknekron’s Information Bus, later TIBCO, publishes market data across trading floors.
- 1993 IBM releases MQSeries, today’s IBM MQ.
- 1998 Java Message Service (JMS) gives Java one API for many messaging products.
- 2006 AMQP begins at JPMorgan Chase as an open wire protocol for messaging.
Go deeper: from “call and wait” to “send and trust”
Messaging introduced vocabulary that’s still in daily use: queues and topics, at-least-once and exactly-once delivery, dead-letter queues, durable subscriptions and idempotent receivers (ones that can safely process the same message twice).
The key shift is from “call and wait” to “send and trust”: the sender gives up knowing exactly when the work happens, in exchange for not depending on the receiver being up. Messaging is the foundation the EAI hubs and service buses were later built on, and the ancestor of today’s event streaming.
What came next: with reliable pipes in place, the rest of the problem remained: translating formats and deciding where each message should go. Vendors put that intelligence in one place. EAI hubs →
04 Late 1990s–early 2000s
EAI: hub and spoke
Put one clever broker in the middle. Every system connects to it once, through an adapter, and the hub translates and routes.
By the late 1990s, large companies ran packaged ERP, CRM and supply-chain suites (SAP, Siebel, PeopleSoft and others) beside their old mainframes. Mergers doubled the count overnight, and Y2K projects exposed how tangled everything was. Enterprise application integration (EAI) vendors sold a fix: a central hub with ready-made adapters for the big packages, a canonical data model so each system translates to one shared format, and graphical tools for mapping and routing.
What it solved
- From n² links to n: each system needs one adapter, not one link per partner.
- Translation and routing live in one place, not inside every application.
- Packaged adapters for SAP and friends saved months of custom work.
- One place to watch every flow.
What it broke
- The hub is a single point of failure and a bottleneck.
- A central team owns it, so every change joins their queue.
- Proprietary, expensive and hard to hire for.
- The canonical data model is hard to agree on and harder to change.
Milestones
- 1996 webMethods is founded; TIBCO Software follows in 1997.
- 1998 XML 1.0 becomes a W3C Recommendation: a common, readable format for messages.
- Late 1990s The EAI boom: Vitria, SeeBeyond, CrossWorlds, Mercator, NEON and others.
- 2000 Microsoft ships BizTalk Server 2000.
Go deeper: the canonical data model
In Enterprise Integration Patterns terms, the hub is a message broker. Most EAI hubs were built on messaging, adding transformation (mapping one format to another), content-based routing (deciding where a message goes by what’s in it) and simple process orchestration.
The canonical data model outlived the products. It comes back in ESBs, in the “system APIs” of API-led connectivity, and in today’s event schemas. The lesson each time is the same: a shared model is valuable only if it’s small, versioned and owned by someone.
What came next: the idea was right, but the products were closed. The web’s open standards offered a way for every system to speak the same language. SOA and the ESB →
05 2000s
SOA and the enterprise service bus
Wrap each business capability as a service with a published contract, and connect them all with a standards-based bus.
XML and the web made a vendor-neutral way to call software across a network possible. Service-oriented architecture (SOA) organised the business into reusable services, such as “get customer”, “check credit” and “create invoice”, each described by a machine-readable contract (WSDL) and called with SOAP messages. The enterprise service bus (ESB) became the usual way to connect them: a successor to the EAI hub that routed, transformed, orchestrated and secured calls, built on open standards instead of proprietary adapters.
The leading buses came from the messaging and EAI vendors of the decade before. TIBCO built on its messaging roots with BusinessWorks and then ActiveMatrix; IBM had WebSphere Message Broker and WebSphere ESB; Oracle took over BEA’s AquaLogic Service Bus; Software AG bought webMethods; Microsoft extended BizTalk. Open-source buses such as Mule offered a lighter alternative.
What it solved
- Interoperability: Java, .NET and mainframe services could finally call each other.
- Contracts first: every service published what it accepts and returns.
- Reuse: build “check credit” once and use it from every channel.
- Governance: security, policies and versions managed in one place.
What it broke
- Heavyweight: the WS-* family grew dozens of specifications for security, reliability, transactions and more.
- “Smart pipes”: business logic crept into the bus, where few people could see or test it.
- The central integration team became a bottleneck again.
- Reuse was promised more often than achieved, and many SOA programmes turned into expensive governance exercises.
Milestones
- 1996 Gartner analysts introduce the term “service-oriented architecture”.
- 2000 SOAP 1.1 is published as a W3C Note; WSDL 1.1 follows in 2001.
- 2002 Sonic ships the first product called an ESB, and Gartner popularises the term. TIBCO releases BusinessWorks.
- 2003 Hohpe and Woolf publish Enterprise Integration Patterns.
- 2006 TIBCO launches ActiveMatrix, its platform for building and running SOA services.
- 2009 Anne Thomas Manes writes “SOA is Dead; Long Live Services”.
Go deeper: what the bus did, and why it got the blame
The ESB’s features were all sensible: protocol mediation, message transformation, content-based routing, orchestration (often in BPEL, a language for business processes) and central security. The trouble was where they lived. With orchestration in the bus, a business process was split between the applications and a central middleware team, and every change needed both.
Many ESBs were technically distributed, but organisationally they were usually one big shared thing. SOA’s principles survived its products: loose coupling, explicit contracts, autonomous services and discoverability. Microservices are often described as “SOA done right”.
What came next: meanwhile, on the public web, companies were opening up with something far simpler. Web APIs →
06 2000s–2010s
Web APIs and REST
Plain HTTP and JSON, documented for anyone to use. Integration becomes a product.
Salesforce and eBay opened web APIs in 2000, and others followed. Then the iPhone arrived in 2007 and every company needed a mobile app, which needed a back end it could call over the internet. REST (resources with URLs, acted on with standard HTTP methods) and JSON proved far easier for web and mobile developers than SOAP and WSDL. APIs stopped being internal plumbing and became products, with documentation, developer portals and sometimes revenue of their own.
What it solved
- Simplicity: anything with an HTTP client could integrate, often in an afternoon.
- Reach: one API served the web, mobile, partners and outside developers.
- API gateways added keys, OAuth, rate limits and analytics at the edge.
- Self-service: developers integrated from documentation instead of meetings.
What it broke
- The client orchestrates: the web shop now calls four APIs itself, and has to cope when one fails.
- Versioning: every published API is a promise, and breaking it breaks strangers.
- Chatty, inconsistent APIs, designed project by project.
- Security moved to the internet’s edge, where mistakes are public.
Milestones
- 2000 Roy Fielding’s doctoral dissertation defines REST.
- 2000 Salesforce and eBay launch web APIs.
- 2007 The iPhone. Mobile apps need back ends they can call.
- 2012 OAuth 2.0 (RFC 6749) standardises delegated access.
- 2015 The OpenAPI Initiative forms around the Swagger specification.
Go deeper: REST, and the API mandate
REST is an architectural style, not a protocol: stateless requests, resources with identifiers, a uniform set of operations, cacheable responses and, in Fielding’s definition, hypermedia links between resources (which few APIs called “RESTful” actually use).
The bigger shift was organisational. Around 2002, Jeff Bezos reportedly required every Amazon team to expose its data and functions only through service interfaces, designed as if for outside developers, as Steve Yegge later recounted. That idea, that interfaces are products and teams own them, went on to shape both microservices and API-led connectivity.
What came next: hundreds of APIs built project by project started to look like a new kind of spaghetti. The next idea was to organise them. API-led connectivity →
07 Mid-2010s onwards
API-led connectivity
Arrange APIs in three layers, so each new project reuses what the last one built.
With mobile, SaaS and digital projects arriving faster than central IT could deliver them, organisations had APIs everywhere but little reuse. MuleSoft popularised API-led connectivity in 2015; its white paper called it “the next step in the evolution of SOA”. APIs are sorted into three layers and published as building blocks of an “application network”. The name is MuleSoft’s, but the layering works on any API platform.
What it solved
- Reuse: the mobile app reuses the process and system APIs built for the web shop.
- Separation: systems of record hide behind system APIs, so replacing the ERP doesn’t break the web shop.
- Speed for front-end teams: each channel gets exactly the data it needs.
- Clear ownership per layer, discoverable in a catalogue.
What it broke
- More hops: every request crosses three layers, adding latency, infrastructure and cost.
- Layers for their own sake: thin pass-through APIs that add nothing.
- Still mostly synchronous, so each layer’s outages add up.
- Platform licensing can tie the architecture to one vendor.
Experience APIs
Shape data for one channel: the web shop, the mobile app, a partner. Owned by that channel’s team, and they change often.
GET /web/orders/123 returns exactly what the order page
shows, in one call.
Process APIs
Business logic that spans systems, such as placing an order or onboarding a customer. Owned by a product or line-of-business team; they don’t care which channel calls them.
POST /orders reserves stock, takes payment and books
shipping.
System APIs
Wrap one system of record and hide its quirks. Owned by the team that knows the system, and they change rarely.
The ERP’s MATERIAL_STOCK table becomes
GET /stock/123.
Milestones
- 2015 MuleSoft publishes its API-led connectivity approach.
- 2018 Salesforce acquires MuleSoft for about US$6.5 billion.
Go deeper: old ideas, new delivery model
API-led connectivity is close kin to layered SOA (presentation, business process and data services) and to the backend for frontend pattern for experience APIs. What was new was the delivery model: a catalogue of reusable APIs, and a “centre for enablement” that helps other teams build, rather than a central team that builds everything for them.
Used well, it stops point-to-point creeping back in. Used dogmatically, three layers for every integration triples the work for no gain.
What came next: the biggest web companies had gone further: not just APIs between applications, but applications made out of APIs. Microservices →
08 2010s onwards
Microservices
Break the application itself into small services, each owned by one team, with its own data, deployed on its own schedule.
Netflix, Amazon and others needed to change software many times a day without coordinating hundreds of engineers. They split large applications into services built around business capabilities, each with its own database, talking over lightweight APIs. Docker (2013) and Kubernetes (2014) made running hundreds of small services practical. James Lewis and Martin Fowler’s 2014 article set out the principles, including “smart endpoints and dumb pipes”, a direct rejection of the clever ESB.
What it solved
- Independent releases: a team ships without waiting for anyone else.
- Scale only what’s busy: ten copies of checkout, one of returns.
- Ownership: you build it, you run it, and you own its integrations.
- The right language or database for each job.
What it broke
- A distributed system is hard: networks fail, latency adds up, and one bug can span a dozen services.
- Chains of synchronous calls couple uptime: if Payments is down, so is checkout.
- No shared transactions: keeping data consistent across services is now your problem.
- Operational overhead: tracing, service discovery, a service mesh, a platform team.
Milestones
- 2011 “Microservices” is discussed at a software architects’ workshop near Venice; the group adopts the term in 2012.
- 2013 Docker makes containers easy to build and run.
- 2014 Lewis and Fowler publish “Microservices”; Google announces Kubernetes.
- 2016–17 Service meshes (Linkerd, then Istio) take over retries, encryption and tracing between services.
Go deeper: the distributed monolith
Microservices move integration inside the application. The patterns that matter: a gateway or backend-for-frontend at the edge; timeouts, retries and circuit breakers between services (a circuit breaker stops calling a service that keeps failing, and uses a fallback instead); correlation IDs and distributed tracing, now usually OpenTelemetry, to follow one request; and “database per service”, which forbids the shared-database integration of the 1980s.
Conway’s law shapes the whole thing: service boundaries end up mirroring team boundaries. The common failure is the distributed monolith, services that must be released together because they call each other synchronously and share assumptions. The usual cure is events.
What came next: to stop services waiting on each other, teams stopped asking and started announcing. Events →
09 2010s onwards
Event-driven and streaming
Services publish facts (“an order was placed”) to a durable log. Anyone who cares subscribes.
LinkedIn built Kafka to move activity data between its systems in real time, and open-sourced it in 2011. Unlike a classic queue, it keeps events in an ordered, durable log that many consumers can read, each at its own pace, and replay later. Together with patterns like event sourcing and sagas, it let microservices cooperate without calling each other, and turned data integration from nightly batches into continuous streams.
What it solved
- Producers don’t know their consumers: add Analytics without touching Orders.
- Resilience: a consumer that’s down catches up from the log when it returns.
- Real time at scale, for integration and analytics alike.
- A history you can replay to rebuild a view or feed a new system.
What it broke
- Eventual consistency: the order is accepted before payment goes through, so the business must handle “payment failed later”.
- Harder to follow: no single place shows the whole flow.
- Event schemas become contracts, and need versioning like any API.
- Ordering, duplicates and exactly-once processing need real care.
Calls or events: the same order, timed
The microservices order above, as a chain of calls, against the same work done with events. The service timings are illustrative.
Request and response
The customer waits 660 ms, and the order works only if all five services are up at once: about 99.5% of the time if each is up 99.9%.
Events
The customer waits 30 ms, and the order is accepted even while Payments is down. The price: the confirmation comes later, and a card that’s declined later has to be handled afterwards, by a saga that cancels the reservation and tells the customer.
Milestones
- 1987 Hector Garcia-Molina and Kenneth Salem describe sagas: long transactions as steps with compensations.
- 2005 Martin Fowler writes up event sourcing.
- c. 2010 Greg Young popularises CQRS: separate models for writing and reading.
- 2011 LinkedIn open-sources Kafka.
- 2017 AsyncAPI starts, to describe event-driven APIs as OpenAPI does for REST.
- 2019 CloudEvents 1.0: a common envelope for events from any source.
Go deeper: not all events are alike
Event notification says “something happened; ask me for the details”. Event-carried state transfer includes the data, so consumers don’t have to call back. Event sourcing makes the log itself the system of record. Change data capture (CDC) turns an existing database’s changes into events without touching the application, which is often how legacy systems join an event-driven estate.
Sagas coordinate a multi-step business transaction either by choreography, where each service reacts to the others’ events, or by orchestration, where a coordinator tells each step what to do. An event mesh links brokers across clouds and data centres so events can flow between them.
What came next: at the same time, the systems being integrated were moving out of the data centre altogether. Cloud and hybrid →
10 2010s onwards
Cloud, SaaS and hybrid integration
Half the systems now belong to someone else. Integration moves to the cloud too, and reaches back into the data centre.
Salesforce (1999) showed that business software could be rented, not installed. By the 2010s, companies ran dozens or hundreds of SaaS applications alongside their ERP and mainframe. You can’t install an adapter inside someone else’s SaaS, and connecting it to systems behind the firewall raised new security questions. Integration platform as a service (iPaaS) answered with hosted integration: ready-made connectors for SaaS APIs, visual flow designers, and small agents installed on-premises that dial out to the cloud, so no inbound firewall port has to be opened.
What it solved
- No middleware servers to run: the platform is someone else’s problem.
- Hundreds of ready-made connectors that keep up with SaaS API changes.
- Hybrid reach: on-premises agents connect safely through the firewall.
- Low-code tools let business technologists build simple flows themselves.
What it broke
- SaaS APIs have rate limits, quotas and changes you don’t control.
- Data residency and security: data crosses providers and borders.
- Sprawl: hundreds of flows built by many hands, and no one has the full list.
- The integration estate is now spread over several platforms and clouds.
Milestones
- 1999 Salesforce is founded on the pitch of “no software”.
- 2006 Amazon Web Services launches S3 and EC2.
- 2007 Webhooks get their name: a simple way for one web app to push events to another.
- 2010 Dell acquires Boomi, an early cloud integration platform.
- 2011 Gartner names the iPaaS category.
Go deeper: the hybrid integration platform
Analysts’ idea of a hybrid integration platform recognised that few organisations can pick just one style. They need API management, messaging, event streaming, B2B and EDI, managed file transfer and iPaaS together, across clouds and their own data centres.
Webhooks became the usual way for SaaS apps to announce changes, though they need retries, signatures and idempotent receivers to be dependable. The architectural lesson: put the connectivity where the systems are, but keep one place to see and govern all of it.
What came next: cloud providers then offered a different idea: don’t run integration servers at all. Serverless →
11 Mid-2010s onwards
Serverless
Small functions that run only when an event arrives. You pay per call and manage no servers.
AWS Lambda launched in 2014: upload a function, attach it to a trigger (an HTTP request, a file landing in storage, a message, a database change) and the cloud runs it on demand, from zero copies to thousands. Managed queues, event buses and workflow services followed. For integration, much of the glue that once ran on an ESB or a hand-built server became a handful of functions wired together by the cloud provider’s own services.
What it solved
- No servers to patch or size; capacity follows the load.
- Pay per use: an integration that runs twice a day costs almost nothing.
- A natural fit for event glue: react to a file, a message or a webhook.
- Managed workflows handle retries and long-running processes.
What it broke
- Cold starts and time limits: not every workload fits.
- Logic spread over many small functions and triggers is hard to see as a whole.
- Deep dependence on one cloud’s services and event model.
- Costs can surprise at high volume, and testing locally is awkward.
Milestones
- 2014 AWS Lambda.
- 2016 Azure Functions and Google Cloud Functions arrive; AWS Step Functions runs functions as workflows.
- 2019 Amazon EventBridge, a managed event bus that also takes events from SaaS partners.
Go deeper: four managed pieces
Serverless integration usually combines an API gateway, functions, queues or event buses, and a workflow engine. It favours choreography, with functions reacting to events, and makes per-step retries and dead-letter queues cheap.
The risks are those of microservices, magnified: dozens of tiny units, hidden dependencies, and observability that has to be designed in from the start. Infrastructure as code (Terraform, AWS CDK and similar) is what keeps it understandable.
What came next: which brings us to now, when all of these styles run side by side, and a new kind of client has arrived. Today →
12 2020s
Today: everything at once, plus AI agents
All of the above, side by side, and a new kind of client: AI agents that use your APIs on someone’s behalf.
A large organisation today may run a mainframe batch job, an ESB, an API gateway, Kafka, an iPaaS and serverless functions at the same time, across several clouds. The question has moved from “which middleware?” to “how do we find, govern and secure all of this?” API-first design, OpenAPI and AsyncAPI catalogues, GraphQL for flexible reads, platform engineering and data mesh all aim at the same thing: integration as self-service, with guardrails.
Then generative AI arrived. AI agents get things done by calling tools, which means calling your APIs. The Model Context Protocol (MCP, 2024) standardises how an agent finds and uses tools; Agent2Agent (A2A, 2025) standardises how agents from different organisations work together.
Why integration matters more now
- More systems than ever, owned by more vendors.
- Customers expect real time: stock, delivery, status.
- AI is only as useful as the data and actions it can reach, and only as safe as the controls on them.
- Privacy, data residency and audit rules apply at every hop.
What’s still hard
- Governing many platforms and many teams at once.
- Agents choose their calls at run time, so permissions, limits and audit trails matter more than ever.
- Untrusted text (a web page, an email) can steer an agent toward real systems.
- The cost and complexity of running every style at the same time.
Milestones
- 2015 Facebook open-sources GraphQL.
- 2019 Zhamak Dehghani describes data mesh: data owned and published by domain teams.
- Nov 2024 Anthropic introduces the Model Context Protocol.
- Apr 2025 Google announces Agent2Agent; in June it moves to the Linux Foundation.
- Dec 2025 MCP is donated to the Linux Foundation’s new Agentic AI Foundation.
Go deeper: agents are integration clients
An agent calling tools is integration with a twist: the sequence of calls isn’t designed in advance, it’s chosen at run time by a model. That puts weight on things integration teams already know well: clear contracts (an MCP tool is, in effect, an API with a description written for a model), least-privilege access, operations that are safe to retry, rate limits and audit trails.
Good APIs and clean events turn out to be the foundation AI needs. Organisations with a tangled integration estate find their agents inherit the tangle.
Side by side
All twelve, compared
Twelve styles on six points, and whether each is still relevant today. Choose a column heading to highlight it; choose an era’s name to jump back to it.
The pattern
There is no final architecture
Every style on this page is still running somewhere, often in the same company. Each new one didn’t replace the last; it took over the problems the last handled badly. Batch is still best for bulk. Queues are still best for reliable hand-offs. APIs suit questions that need an answer now; events suit news that anyone might care about.
The same tensions keep swinging back and forth:
- Central teamEvery team owns its own
- Smart pipesSmart endpoints
- Call and waitPublish and move on
- Vendor platformOpen standards
Good integration architecture is mostly a matter of choosing deliberately, for each flow, where on each of those lines it should sit.
Questions to ask before choosing a style
- Does the caller need an answer now, or only for it to happen? An answer now suggests an API; “make it happen” suggests a queue or an event.
- Who needs to know? One known system, or anyone who might care later? Call the one; publish for the many.
- What if the other side is down for an hour? If the business must carry on, don’t chain synchronous calls; put something durable in between.
- How much data, how often? Millions of rows once a night is still a job for bulk transfer; a steady flow is a stream.
- Who will own and change it? A central team, or the product team? Choose a style that team can run.
- Where do the systems live? Your data centre, someone’s SaaS, several clouds? That decides where the integration has to run.
- Who, or what, is calling, and what should it be allowed to do? A person’s app, a partner, an AI agent: the answer shapes the gateway, the permissions and the audit trail.
Sources and further reading
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (2003).
- Andrew Birrell and Bruce Nelson, “Implementing Remote Procedure Calls” (1984).
- Hector Garcia-Molina and Kenneth Salem, “Sagas” (1987).
- IBM MQ, on MQSeries’ 1993 release.
- Enterprise service bus, including Roy Schulte on where the term came from.
- TIBCO Software, annual report for fiscal 2002, on the release of BusinessWorks.
- IBM App Connect Enterprise, on the product’s names from MQSeries Integrator onwards.
- Microsoft, “Microsoft BizTalk Server product lifecycle update” (2026).
- DevClass, “Reduce costs by 90% by moving from microservices to monolith” (2023), on the Prime Video case study.
- Roy Fielding, Architectural Styles and the Design of Network-based Software Architectures (2000).
- MuleSoft, Explore API-led connectivity.
- James Lewis and Martin Fowler, “Microservices” (2014).
- Martin Fowler, “Event Sourcing” (2005) and “What do you mean by ‘Event-Driven’?” (2017).
- Jay Kreps, I ♥ Logs (2014), which grew from his 2013 essay “The Log”.
- Zhamak Dehghani, “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh” (2019).
- Anthropic, “Introducing the Model Context Protocol” (2024) and “Donating the Model Context Protocol” (2025).
- Linux Foundation, “Linux Foundation Launches the Agent2Agent Protocol Project” (2025).
Dates for products and terms are the earliest well-documented ones; several ideas here were in use under other names before they got the names we use now.