The Hitchikers Guide to Event Driven Architectures

with Node.js

DON'T PANIC!

It's Just Events

sforce.co/dont-panic-eda

Julián Duque

Principal Developer Advocate at Heroku

JSConf/NodeConf Colombia Organizer

Node.js Collaborator Emeritus

🦋 @julianduque.co

/in/juliandavidduque

X @julian_duque

Event Driven Architectures

Event-driven architecture (EDA), also known as message-driven architecture, is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.

Source: Wikipedia

Overview

Events ➡ Simple signals or messages that indicate something has happened, a state change, or a new request for action

EDA ➡ A way to communicate between components without tight coupling

Benefits ➡ EDA enables asynchronous, non-blocking interactions

Why Node.js?

JavaScript's non-blocking nature (Event loop)

Clicking a button on a website is an event. The browser listens to this event and sends a message to perform an action – this is essentially how EDA works at a small scale

Node.js Event Loop

The Heart of Gold

   ┌───────────────────────────┐
┌─>│           timers          │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │     pending callbacks     │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │       idle, prepare       │
│  └─────────────┬─────────────┘      ┌───────────────┐
│  ┌─────────────┴─────────────┐      │   incoming:   │
│  │           poll            │<─────┤  connections, │
│  └─────────────┬─────────────┘      │   data, etc.  │
│  ┌─────────────┴─────────────┐      └───────────────┘
│  │           check           │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
└──┤      close callbacks      │
   └───────────────────────────┘

Source: The Node.js Event Loop

EventEmitter

The Babel Fish

import { EventEmitter } from "node:events";

const emitter = new EventEmitter();

emitter.on("start", () => {
  console.log("started");
});

emitter.emit("start");

emitter.on("data", (data) => {
  console.log("received", data);
});

emitter.emit("data", { message: "Hello Node Congress" });

emitter.on("error", handleError);

EventEmitter

Handle events in Node.js

Emit events synchronously

Used widely in Node core and the ecosystem

```
http/https
```
```
fs
```
```
stream
```
```
dgram
```
```
process
```
etc...

import { EventEmitter } from "node:events";

const events = new EventEmitter();

events.on("emote", (data) => {
  const { emote } = data;
  sendToClient(emote);
});

events.on("votes", (data) => {
  const { votes } = data;
  sendToClient(votes);
});

events.emit("emote", { "emote": "dolphin" });
events.emit("votes", { "dolphin": 10, "applause": 5 });

Pub/Sub

The Restaurant at the End of the Universe

Pub/Sub with Redis

Synchronous communication

Fire & forget

Fan-out only

Non-persistent

import Redis from "redis";

const pub = new Redis();
const sub = new Redis();

sub.subscribe("emotes", (err, count) => {});
sub.on('message', (channel, message) => {
  if (channel === "emotes") {
    const data = JSON.parse(message);
    // process event
  }
});


setInterval(() => {
  pub.publish("emotes", JSON.stringify({ 
    emote: "dolphin" 
  }));
}, 1000);

$ npm install redis

Apache Kafka

The Infinite Improbability Drive of Events

Apache Kafka Core Concepts

Brokers

The instances running Kafka and managing streams of events in a cluster.

Producers + Consumers

Clients that write to or read from a Kafka cluster.

Topics

Streams of events that are replicated across the brokers. Configured with time based retention or log compaction.

Partitions

Discrete subsets of topics, and important tuning points for parallelism and ordering.

BROKER

TOPIC

PARTITION

PRODUCERS

CONSUMERS

Source: Building an Event-Driven Architecture with Managed Data Services

Kafka.js

100% implemented in JavaScript (no native dependencies)

Production-ready

Transactions, compression, batches

import { EventEmitter } from 'node:events';
import { Kafka } from 'kafkajs';
const kafka = new Kafka({
  clientId: 'emotes-api',
  brokers: ['host1:port', 'host2:port', 'host3:port'],
});

const producer = kafka.producer();
const consumer = kafka.consumer({ 
  groupId: KAFKA_PREFIX + 'emotes' 
});
const topic = KAFKA_PREFIX + 'emotes';

const events = new EventEmitter();
await consumer.subscribe({ topic });
await consumer.run({
  eachMessage: async ({ 
    topic: _topic, 
    partition, 
    message 
  }) => {
    try {
      const { event, data } = 
            JSON.parse(message.value.toString());
      events.emit(event, data);
    } catch (error) {
    }
  },
});
await producer.connect();

$ npm install kafkajs

One last thing...

Life, the Universe, and Everything

Server Sent Events

Unidirectional stream of events from the server to the browser

Content-Type: text/event-stream

id: 42
event: hitchhike
data: { "planet": "Betelgeuse V" }

// server.js
import FastifySSEPlugin from 'fastify-sse-v2';
// Register SSE plugin
fastify.register(FastifySSEPlugin);


// routes/index.js
import { EventIterator } from 'event-iterator';
// Route /event/:id
fastify.get('/events/:id', async (request, reply) => {
  // fastify-cors doesn't seem to work with fastify-sse-v2
  // so we need to add this header to this route manually
  reply.raw.setHeader('Access-Control-Allow-Origin', '*');
  const id = request.params.id;

  const eventIterator = new EventIterator(({ push }) => {
    // Get events from Kafka and send them through SSE
    events.on(`emote:${id}`, push);
    events.on(`votes:${id}`, push);
    // Cleanup event listeners to prevent memory leaks
    request.raw.on('close', () => cleanup(id, push));
    return () => cleanup(id, push);
  });

  reply.sse(eventIterator);
});

Server

$ npm install fastify-sse-v2 event-iterator

Server Sent Events

Unidirectional stream of events from the server to the browser

Content-Type: text/event-stream

id: 42
event: hitchhike
data: { "planet": "Betelgeuse V" }

EventSource is part of the Web API

const events = new EventSource("/api/events/nodecongress");

// Handle any message
events.onmessage = (message) => {};

// Handle named events
events.addEventListener("votes", (event) => {
  // Update votes on interface
});

events.addEventListener("emote", (event) => {
  // Trigger animation 
});

Client

It's a Wrap

Mostly Harmless (But Really Useful)

So Long, and Thanks for All the Fish

🐬

Thank you!