Transactional outbox

Saves a row and the event about it in one database transaction, then publishes the event. No lost events, and no events for rows that rolled back.

How it works

The bug it fixes: save the order, then publish to Kafka. If the process dies between the two, the order exists and nobody hears about it. Publish first instead, and a failed insert leaves an event for an order that doesn't exist. So insert the event into an outbox table in the same transaction as the order. A relay (a poller, or change data capture like Debezium reading the write ahead log) publishes unsent rows and marks them sent. The relay can die after publishing and before marking, so it publishes again: delivery is at least once. Consumers record processed event ids and skip repeats, which makes the effect exactly once.

Reach for it when

• A service must update its database and tell other services
• Webhooks, emails and search indexing after a write
• Anywhere you see a dual write to a database and a queue

Copy// The row and its event commit in ONE transaction. A relay publishes later. Consumers dedupe by event id.
function placeOrder(db, order) {
  db.transaction(tx => {
    tx.orders.push(order);
    tx.outbox.push({ id: `evt-${order.id}`, type: "order.placed", payload: order, sent: false });
  });
}
// May die after publish and before marking sent, so delivery is at least once.
function relay(db, publish) {
  for (const e of db.outbox)
    if (!e.sent) {
      publish(e);
      e.sent = true;
    }
}
class DedupConsumer {
  constructor(apply) {
    this.seen = new Set(); // production: a processed_events table, written in the same transaction as apply
    this.apply = apply;
  }
  handle(e) {
    if (this.seen.has(e.id)) return false;
    this.apply(e);
    this.seen.add(e.id);
    return true;
  }
}
// tiny in-memory "database" with rollback, so the pattern is testable
function memDb() {
  const db = { orders: [], outbox: [] };
  db.transaction = fn => {
    const snap = { orders: [...db.orders], outbox: [...db.outbox] };
    try {
      fn(db);
    } catch (err) {
      db.orders = snap.orders;
      db.outbox = snap.outbox;
      throw err;
    }
  };
  return db;
}

Traps

• Dual writes pass every test and lose events in production, on deploys and crashes.
• Consumers that aren't idempotent turn at least once into double charges.
• If order matters, publish per entity in order (partition by order id).
• Delete sent outbox rows, or the table grows forever.