Event-Driven Architecture for IT ecosystems
Trying to avoid the italian cuisine problems
Modern businesses rely on a diverse ecosystem of applications, including custom-built systems and cloud-based services. Integrating these disparate systems can be challenging, often leading to tightly coupled (spaghetti) integrations that are difficult to maintain and scale. Event-Driven Architecture (EDA) offers a more flexible approach to interoperability. By leveraging events as the foundation for communication, EDA enables systems to exchange data more loosely, improving maintainability and facilitating easier adaptation to changing business requirements.
At the beginning there was messaging
There are various types of objects and information that pass through message brokers. To understand Event-Driven Architecture (EDA) it is crucial to be able to navigate easily between them:
Message - is the core object of communication, which can be literally anything. Following the examples given by Jacqui Read at DDD Europe 2024 in the talk “Every event everywhere all at once”, this might as well be “High five! Well done!”. A message can have more narrowly refined subtypes that serve specific interoperability function:
Command - is a subtype of a message where we tell the other side of the communication to do something, like “Create customer”, this literally means that the data set transferred between the systems is sent with the intent to invoke an action on the receiving end,
State - is another subtype, with a passive message, transferring a state of something, telling us for example “ticket 123qwe is in progress”, this is often reflected by a very light payload, sent periodically, most commonly containing an identifier, described state and some metadata,
Event - is another subtype of message that states a fact that occurred. This means that the message sent between the systems contains metadata and a payload (or identifier of one) that describe something that happened business or operations wise. But since we’ll be focusing on it a little bit more, we’ll get back to it in a moment,
Given the above, we can notice that a very common mistake happens, that there is a tendency to name something EDA, where it is not actually event-driven, so it uses message types that are not events. To make this clear and before we take a deep dive into EDA, we need to define what is an event and further explore the other types of “objects” that can go through a message broker.
Events vs the world
Let’s start with a generic definition. As we already established, the root of all things going through a message broker is a message. It is the most simple option and it has the least boundaries of what it can be. Simply put a string as payload and that is your message!
But events are not just some random messages. They are specialized in terms of the contents and meaning, so let’s look at a few definitions:
Event
1 a : something that happens : occurrence
b : a noteworthy happening
[...]
Essentially in programming an event is the fact that something happened. Which means it would be associated with verbs in the past tense like: created, started, completed, updated, failed, etc.. This is reflected in the messages metadata and payload. For example, if you use topic hierarchy the metadata would be hidden in the name of the topic, e.g.: /eshop/order/created/v1/{orderId}, and the payload would be the details of that order, even if it is just an ID of a record for later polling.
As mentioned earlier, there are other types of messages that are common to IT ecosystems. For comparison, the implementation philosophy of SOAP was commands and responses to them. The messages were sent to an API that was focused around command operations based on imperative verbs, like get, set, update, which resulted in names like e.g. getCustomer, updateOrder, setStatus. Similarly in REST, while the API represents the resource, the imperative verbs of HTTP Methods (GET, POST, PUT, PATCH, DELETE) represent the command nature of the invoked operation. Commands imply that the calling system is in control of the transactional behavior of the API provider.
Another option is to use messages to transfer state, where it can be described as statuses like those found on a common kanban board (e.g. to do, in progress, done). And lastly messages can transfer whole binaries of documents, which is a nice feature, but lands more in the spectrum of mass transfer topics, rather than real-time application integration.
What is Event-Driven Architecture?
As we now know what an event is, we can define what Event-Driven Architecture is. The most basic understanding would be that it is an architecture that detects, reacts to and processes events. It is a distributed architecture that is composed of event producers and event consumers, which are the components (applications, systems, modules of applications) that contain business logic. The communication between them, as mentioned, happens usually via a message broker, so a dedicated technical application that facilitates asynchronous message exchange.
Context: Integration vs. Distributed systems
As we have defined roughly what EDA is, there is one more distinction to make. Context matters! When using EDA as an architectural style, there are different considerations an architect must take into account when designing a distributed system and when designing application integration. While with a single system this is usually a single team that has a high level of control over development and the domain, for integration that would turn into dealing with cross-domain connections, coordinating between several different development teams and system architects. This brings additional complexity and certainly will influence several aspects of implementation. Furthermore it is worth noting that in distributed systems asynchronous communication may happen without the use of a specialized message broker, by using e.g. in memory queues.
Qualitative Analysis
As we did with Point-to-point, let’s explore the qualities of Event-Driven Architecture and look into a few pitfalls that can be crucial to consider when trying to apply this architectural style. For that we will be using a comparison table that was produced through a qualitative analysis of architectural styles taking several architectural characteristics into account. If you would like to learn more about this analysis or read how we define those characteristics, you can do so by reading this article, where we explain how this comparison was created.
Cost analysis
Let’s start with cost considerations of Event-Driven Architecture as an architectural style. With this approach to ecosystem interoperability a message or event broker is used as a mediator between different domain systems. This brings an implication of development and operational costs being higher than with Point-to-point. The use of a technical broker brings additional complexity. Depending on the choice of a broker, this means that all domain systems that wish to communicate need to learn the brokers protocol as well as learn to properly produce and consume events. This increases the cognitive strain on development teams. On the other side message brokers are incapable of building sophisticated processing logic as they only facilitate communication, so using them does not require a dedicated development team that would implement changes. This brings the development cost to be higher than Point-to-point, resulting in 2$ for EDA in this category. Operational cost is higher as well, as there is another component in the ecosystem, which often requires a license to run as well as a team to maintain and operate it. This also increases overall operational effort in all environments to provide analysis and fixes for any issues that might occur, as there are more components that need to be looked into. This gives the operational costs a score of 3$.
Architectural change cost, even if it is an unpredictable variable, is fairly low (2$). This is due to the fact that a message broker is providing a separation layer between domain systems, splitting a dynamic quantum, that would be common for a point-to-point connection, into two or more quanta, depending on the architectural pattern used. This makes the domain systems loosely coupled between each other, so in turn easier to modify or entirely replaced.
Architectural and design time analysis
Let’s first take a look at the design time qualities of EDA. Like with Point-to-point, the superpower of EDA, in this area, is its simplicity (4*). Implementing this architectural style is fairly simple in terms of architectural components. It is very clear and easy to understand. The reason the score is only 4* is that this style forces all the domain systems to learn how to work with events alongside making additional p2p calls while processing them. Secondly, the use of message brokers, especially patterns related to the use of topics as a communication medium, enables composability and by that also a certain level of reusability of events and related data models. However the downsides of EDA as an ecosystem architectural style, listed below, bring the composability score to 3*. While overall the scores are higher than with P2P, there are aspects that should be looked into. If we explore the capability of EDA to provide abstraction (2*), we can see that, like with P2P, we are bound by several types of coupling:
semantic coupling being the most basic and unavoidable,
contract coupling, which means being bound by the same data exchange contract
afferent, as in for producers, when using queues to communicate many to one,
efferent, as in for consumers, when using topics to communicate one to many or many to many,
conversation coupling, if the chosen message broker does not allow to be protocol agnostic,
This means there are many things to consider when putting this architectural style in a context of a specific implementation and choosing the right broker for a job. These would be the skills available in the organization or on the market, the protocol capabilities of systems that are to be integrated, etc.. Then the various degrees of coupling influence the contract resilience (2*). If the communication is other than one to one, there is a design consideration that each change in the data model (contract) may be a breaking change, depending on how producers and consumers parse payloads and what is the data format used. This impact needs to be assessed early on, because it will influence every integration.
Lastly we should take a look into extensibility (2*), and here’s where the context of integration vs. distributed systems comes into play leading to a lower score than often expected from EDA. This quality seems to be a lot stronger when designing a single domain distributed system, because it is a lot easier to extend the system with new modules or components that work within the same bounded context. When we’re dealing with ecosystem interoperability, while the same extensibility mechanisms apply (patterns to do many to one, one to many and many to many), they work cross-domain, with the cost of using various technologies, different bounded contexts or effort to adapt specific protocols. These cannot be disregarded just as implementation details and need to be considered beforehand.
Operational analysis
Moving on to the second group architectural characteristics, that describe the operational qualities of the architectural style. The foremost benefit of EDA in this area is its performance (5*). Some people might wonder why we scored it higher than P2P. There is one key difference here that is worth noting - responsiveness. Synchronous communication that is the key element of P2P is always a blocking communication, which means that it impacts responsiveness, because it creates a dynamic coupling between the caller and the provider of the API. As mentioned before, using a message broker splits this quantum into two quanta, which means that the event producer does not rely on the performance of the consuming system(s) and can provide data to a group of consumers in parallel instead of calling them in sequence. This provides a performance boost, freeing up resources as soon as the communication with the broker is done.
Looking into further characteristics we can see that scalability (4*) is the next characteristic that is scored high in the comparison. This architectural style makes it a lot easier architecturally to scale-out your systems, because it does not require additional components for load balancing! Using message broker queues enables the use of a competing consumers operational pattern, that by itself provides load balancing capabilities based on resource availability of consumers. Depending on the message broker chosen, this may be also available with topics and configuring consumer groups. It is also possible to provide auto-scaling capabilities in the ecosystem based on monitoring of queues or topics of the message broker and the load on those communication structures.
Another superpower of EDA that derives from its simplicity is testability (4*). The overall setup is fairly straightforward, as the only added component is a broker that technically facilitates communication and sometimes routing based on metadata, so it does not provide any additional logic (e.g. validation rules, content based routing, that needs to be specifically tested.
Looking further into operational characteristics, observability (3*) and auditability (2*) change slightly compared to P2P. While we can observe more within the ecosystem if proper monitoring is implemented to account for the message broker and its communication structures, it does not influence auditability. This is due to the fact that those characteristics still heavily depend on what can be extracted from business systems. The composition of the ecosystems and types of applications within it will have a great impact on what kind of data and metadata describing its operations can be extracted.
Lastly, what requires additional attention with EDA as an architectural style is security (2*). While all of the components can be separately secured, the concern is not with those components, but rather the implications of using message brokers. If we want to leverage the use of topics, it is important to understand the consequences of “contract coupling” mentioned before. The consideration is not only around the brittleness of the contract, but also the fact that if we are broadcasting data via topics to several consumers, it means that all of them receive the same scope of data, without consideration of the actual scope used by a particular system. This means that while the usage might be proper and as designed, full scope of data will be most likely at least logged somewhere. This means that the data can be accessed by someone who is not supposed to do so, resulting in a security breach.
Conclusions
Event-Driven Architecture is a very useful architectural style, especially for ecosystems that are required to be highly performant and at the same time remain simple, testable and scalable. While it has its downsides, if properly governed and mitigated, it is a viable architectural option for small to medium companies that rely on real-time interactions within their business processes.
Bibliography
Event definition; Merriam-Webster dictionary
Every Event Everywhere All at Once; Jacqui Read; DDD Europe 2024
I made everything loosely coupled - will my app fall apart?; Gregor Hohpe
Lesson 138 - Dynamic Quantum Entanglement; Mark Richards; DeveloperToArchitect.com
Well articulated 👍