What does it mean for developers to use Apache Kafka as a streaming platform ?

Software Applications Architecture

Let us step back and discuss on the current state of software development.

In the past, all the applications that were built was a giant monolith applications using the monolith architecture. That means all the functionalities will reside in one single application.

This kind of architecture was proven to fail under heavy load.

Things have changed and the current state of development uses a more modern architecture, which is the microservices architecture.

The application in itself is decomposed into micro services and each microservice has its own database.

But as a whole, in order to deliver the business functionality or value, multiple microservices interact with each other using some communication protocol.

–> The expectation of the apps that are built today have a new requirement, which is providing real time notifications and process the events as they occur.

In order to support that, we need to have a middleware in between.

The middleware that we are using here is an event streaming platform, in our case Apache Kafka.

The idea is, basically, at its core, each microservice will have an API which are going to generate a lot of events and other services, read those events and take necessary action as the events occur.

–> In a nutshell, each microservice will have an API and it will have an event producer and event consumer.

All other services communicate with each other through an event streaming platform. This fundamentally forms the basis for event driven microservices.

What is an Event Streaming Platform?

An event streaming platform allows the application to produce and consume a stream of records, as in like a messaging system.

You can think of this as a pub sub model.

3 foundation principles of event streaming platform

The producer and consumer here are independent of each other, meaning the producer has no clue about which consumer is going to read this message.

An event streaming platform also stores the stream of events so that it can be replayed if it’s necessary.

Events are generally retained in multiple servers for providing fault tolerance and availability.

An Event streaming platform allows the application to process the records as they occur.

Apache Kafka is built based on these principles.

Is Kafka an enterprise messaging system ?

• Traditional messaging systems have the transient message persistence, meaning once the records are read by the consumers, then the messages will be removed from the message broker.

In the case of Kafka, it is going to save the event in the file system where Kafka is installed and the events are retained for a certain time (retention time).

All the events in Kafka are immutable, meaning once the records are sent to Kafka, then it cannot be altered.

• In traditional messaging system, it is the brokers responsibility to keep track of messages consumed by the consumers and remove them from the broker when the messages are read.

But in Kafka, it’s the responsibility of the consumer to keep track of consumed messages.

• With traditional messaging system, you can target a specific consumer to read the message from the broker.

In case of Kafka, any consumer who have access to the broker can read the message.

• Traditional messaging system does not follow the principles of distributed systems.

But Kafka is built on top of the core principles of distributed system.

A distributed system in general is proven to handle the load very well and has the intelligence to distribute the load.

–> Kafka is a kind of streaming system, not just a mere messaging system.

Kafka Use cases

• Kafka can be used for many different use cases in the transportation domain.

Booking rides online through App is pretty common today. Kafka can be used for sending real time tracking of driver notification to the rider.

Ordering food online is also a pretty common scenario today, and Kafka can be used to provide real time tracking of the driver delivering the food.

• In retail world, Kafka can be used to provide a real time sale notifications and real time purchase recommendations based on the previous purchases and real time tracking of online orders.

• In banking world, Kafka can be used to alert on real time fraudulent transactions and provide new feature and product notifications and more.