Event-driven architecture and semantic coupling

Addressing the semantic coupling in an event-driven architecture is critical to truly achieving the loose coupling

Image from Pixabay under Pixabay license

Event-driven architecture (EDA) is key to building loosely coupled applications (microservices or not). It is an architectural style (see here and here) where components communicate asynchronously by emitting and reacting to the events.

Def. 1: An event is something that has happened in past. An event notification (or say event message) contains the description of the event. But in most cases, and also in this article, an event refers to the event message.

Def. 2: A producer is an application or application component that publishes events indicating some state change in it.

Def. 3: A consumer is an application or application component that listens to events and reacts to them.

Applications are loosely coupled when,

• Individual components can be developed and run without knowing the source or target. It is the interaction requirements on an information level that creates the basis for integration.
• Individual components are less constrained to the platform, programming language, and build and run environment. It is, therefore, possible for each application component to be built with the best-fit technology. And some even maybe a commercial-off-the-shelf (COTS) product.
• Individual components can be scaled, provisioned, audited, and checked for compliance independently.

As Martin Fowler explains here, components having to require the knowledge of an external system to complete an internal transaction creates dependency, leading to tight coupling. This knowledge could be for example if an API endpoint in a different component (e.g. another application) exists and is running and responding. This is avoided when applications are designed to work in a passive-aggressive manner where they only react to event messages/notifications.

Loosely coupled? Not yet!

However, there is still a subtle coupling on a logical/semantic level. And, if we are not careful, this can lead to even more complex problems.

• A consumer requires that the event messages have a specific schema. If a producer removes some fields, it can affect the consumers because they might have some logic built around exactly those fields. Likewise if the data type of the field changes, then also it may impact the consumers. Producers are therefore required to a pre-agreed schema.
• Due to a fault in the producer or message broker/queue, an event might get replayed or published more than once. This does not mean that event occurred twice. Consumers need to aware of such glitches and be able to word around them.
• A consumer also needs to be certain that it is the authentic producer that published the events and also that they have not been manipulated on the way.
• Both producers and consumers might be maintaining an event log that stores all the events published or consumed. Such an event log might even be a basis for application logic if the component utilizes event sourcing. Event sourcing extends the event-driven architecture where application states are stored completely as events in an event log. This log is the source of truth and any further updates are also stored as events. The log itself is append-only and replay is a method to restore the state. A component replaying the events then needs to accommodate the historic schema changes in the events representing state changes in the same data entity.

By removing the synchronous API calls we have achieved some degrees of decoupling. But the dependencies that affect the participating applications adversely have not vanished.

Addressing semantic coupling

Since semantic coupling is created by information exchange and therefore needs to be addressed on the information level.

Let us consider the following event.

{
  "header":{
    "application-id":"CRM-application-id",
    "schema-url":"json-schema-url",
    "timestamp":1625507375542,
    "message-id":"9aeb0fdf-c01e-0131-0922-9eb54906e209",
    "event-descriptor":"customer.added"
  },
  "customerid":"cust-id",
  "firstname":"some firstname",
  "lastname":"some lastname"
}

This event provides the following information.

1. The application that published it
2. Schema of the message (A URL link to the corresponding JSON-schema)
3. Timestamp of publishing as well as globally unique message-id.
4. Event descriptor that tells what the event was. In this case, a new customer was added. A combination of event descriptor and application-id must be globally unique.
5. The details of the customer data that was added.

This event composition can be visualized as below.

Event message composition

So how does this reduce the dependencies caused by semantic coupling?

Every event contains a header that provides several critical information.

• The schema URI points to the specific schema (eg. JSON schema) that this message adheres to. Any consumer then will have the possibility to dynamically parse the message as per the schema. This may require each message to be processed twice, first to read the header, pick the right parser as per the schema, and read the whole message. But the upside is that the consumer can parse the message as per the schema.
• The schema helps with addressing the historic schema changes while replaying an old event in an event log that might have a different schema than the current one.
• The globally unique message-id helps to identify the unintentional replays. A replayed message will have the same message-id as the previously processed one. Thus a consumer can be idempotent.
• The event descriptor is a domain-specific name of the event that is unique within that domain. The example above uses a dotted notation to separate the subject and verb (always in past tense).

The rest of the event is the data. In general, it should be fat so that each message is as complete on its own as possible. A fat event carries enough data to not warrant a callback to its originating application. But a fat event does not mean putting all the data in a single event. It can be done by just having ids/URIs to the other related data objects. The main idea is to make it complete for the context.

Can a specific technology or a choice of serialization help some of it? Sure, for example, Avro helps with the detection of schema changes. Schema registry provided by Confluent supports it fully. But relying on a particular technology will tie the solution to that specific technology. There might be some benefits to going that way, but I am more of an advocate of being technology independent to the extent possible.

Cloud events

Cloud events is a concrete and standard example that matches the above event composition pattern.

The same above event can be expressed following the cloud event specification as below.

{
  "specversion": "1.0",
  "type": "customer.added", 
  "source": "CRM-application-id",
  "id": "9aeb0fdf-c01e-0131–0922–9eb54906e209",
  "time": "2021–07–05T20:13:39.4589254Z",
  "dataschema": "json-schema-url",
  "data": {
    "customerid":"cust-id",
    "firstname":"some firstname",
    "lastname":"some lastname"
  }
}

This variation of the event carries the same information, but with some differences in composition. For example, there is no explicit header. Most of the fields I normally classify as header fields as part of the message. The data or the core state change is inside the “data” element. The “dataschema” then refers to the JSON schema of this “data” element and not the whole event. The specification for the whole event is instead provided by “specversion” just as a version number — 1.0 in this example.

The composition of events as in Cloud events can be visualized as below.

Event message composition as in “Cloud events”

The authenticity of the events

By attaching a signed hash to each message, it will be possible to verify the authenticity of each event.

{
 "header":{
 …
  "message-hash":{
   "alg": "some algorithm",
   "salt": "some salt",
   "signed-hash": "0xa3f20717a25…fcfc994cee1b"
  }
 },
 …
}

The cost of this is obviously that every message needs to be hash and signature verified every time they are processed. Therefore, this approach should be used only if message authenticity is critical.

Versioning

Just like REST API versions, events can also have versions. I consider events as also as APIs, because they trigger reactions in the consumers.

In the above example, the version of an event is implicit. The schema of an event encompasses the version change. The “schema-url” in my example or a combination of “dataschema” and “specversion” in Cloud events handle it.

But there might be cases where an explicit event version is necessary. A simple of doing it can be by appending the version to the event descriptor.

"event-descriptor":"customer.added.v1"

Or in Cloud events:

"type":"customer.added.v1"

It is important to note that a new version of an event must have the same semantics and meaning as the old version of it. It should indicate the state change for exactly the same data entity/subject. If not, it is not a new version of the event but rather a new event.

Conclusion

Any technology or architectural style can be used and misused. It can be understood and misunderstood or only partially understood. This is true for event-driven architecture also. To create applications and systems that are truly loosely coupled, it is essential to focus on the events, their composition, schema and embrace the fact that they will change.

The schema evolution is both a fact and necessary evil. A technology-independent solution will enable a long-lasting solution that can survive through changes in application landscape changes in an organization and the underlying technology stack.

And why a picture of nature? Well, I see nature also being event-driven. Action and reaction are everywhere and happens all the time. Actually, nature is a perfect example of an event-driven ecosystem.