⚡ Building Event-Driven Systems with WebSockets, Kafka, and Microservices – Anshuman Verma

🧭 Introduction

Modern systems are moving away from traditional REST APIs for real-time features. If you’re building applications like collaborative tools, IoT dashboards, or trading platforms, you need low-latency, event-driven architectures.

In this blog, we’ll explore how to combine WebSockets, Apache Kafka, and microservices to build a scalable and reactive architecture.

💡 Use Case

Let’s say you’re building a real-time analytics dashboard for a logistics company:

IoT devices send data every 3 seconds.

Data is aggregated and visualized in real-time.

Clients should see updates immediately.

Events should be durable and fault-tolerant.

🔌 Component Overview

Component	Role
WebSocket Gateway	Accepts client connections and sends data to/from clients.
Kafka Broker	Central messaging backbone for real-time event distribution.
Microservices	Handle device data, process events, store state.
Redis	Optional for pub/sub or caching recent updates.

🧱 Architecture Flow

IoT Device → Data Processor Service → Kafka Topic → WebSocket Publisher → Client Browser

Devices push data via HTTP/MQTT.
Microservice ingests and pushes it to Kafka.
Another service consumes Kafka topic and broadcasts via WebSocket.

🔧 Kafka Setup (Python Example with Confluent Kafka)

Producer (IoT Data Simulator):

from confluent_kafka import Producer
import json, time, random

p = Producer({'bootstrap.servers': 'localhost:9092'})

while True:
    data = {"device_id": 1, "temp": round(random.uniform(22.0, 30.0), 2)}
    p.produce('iot-events', json.dumps(data).encode('utf-8'))
    p.flush()
    time.sleep(2)

Consumer → WebSocket Broadcaster (Node.js):

const WebSocket = require('ws');
const { Kafka } = require('kafkajs');

const wss = new WebSocket.Server({ port: 8080 });
const kafka = new Kafka({ brokers: ['localhost:9092'] });
const consumer = kafka.consumer({ groupId: 'ws-publisher' });

(async () => {
  await consumer.connect();
  await consumer.subscribe({ topic: 'iot-events' });

  consumer.run({
    eachMessage: async ({ message }) => {
      const payload = message.value.toString();
      wss.clients.forEach(client => {
        if (client.readyState === WebSocket.OPEN) {
          client.send(payload);
        }
      });
    }
  });
})();

⚠️ Production Considerations

Scalability: Use NGINX or a gateway layer to load-balance WebSocket connections.
Security: Use TLS, authentication tokens per socket, and isolate Kafka consumers.
Persistence: Store Kafka events in ClickHouse or TimescaleDB for historical analytics.
Monitoring: Track latency between event and socket delivery via Prometheus.

🧪 Why This Pattern Works

Loosely Coupled: Services can scale independently.
Replayable: Kafka lets you reprocess old messages (time-travel debugging).
Reactive: Real-time user experience without polling.
Extendable: Plug in alert services, ML processors, or data pipelines without touching the WebSocket layer.

📦 Bonus: Deploying with Docker

version: '3.8'
services:
  kafka:
    image: bitnami/kafka:latest
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
  websocket-server:
    build: ./websocket-publisher
    ports:
      - "8080:8080"

🧠 Final Thoughts

Combining WebSockets + Kafka + Microservices lets you architect fast, reliable, real-time systems. This pattern is used by companies like Uber, Netflix, and Bloomberg for high-volume, low-latency data delivery.

If your project deals with live data, this architecture is a battle-tested blueprint to follow.

Further Reading:

Posted in Blog by anshumanverma

Write a comment