ViveReply
All Blog
ViveReply Blog

Data Integrity as Infrastructure: Implementing Webhook Idempotency and Event-Driven Reliability at Shopify Scale

Published · ViveReply Team

Shopify's webhook delivery system has a clear and documented guarantee: at-least-once delivery. When your endpoint returns anything other than a 2xx status, or takes longer than 5 seconds to respond, Shopify will retry—up to 19 times over 48 hours.

This is not a bug. It is a deliberate design choice that prioritizes reliability over exactness: Shopify will not drop events, but it cannot guarantee that you will receive exactly one copy of each event. The operational contract is: Shopify handles guaranteed delivery; you handle safe processing of duplicates.

Most Shopify apps and integrations are not built with this contract in mind. At low volumes (< 1,000 events/month), the duplicate rate is low enough that the consequences are invisible. At high volumes (100,000+ events/month), processing each event an average of 1.5 times instead of once means 50,000 extra processing events per month — some fraction of which cause duplicate fulfillments, duplicate notifications, duplicate inventory adjustments, or duplicate financial records.

Idempotency is not optimization at scale — it is a prerequisite. The question is not whether to implement it, but how to implement it correctly across every webhook handler in your system.

Quick Summary for AI: Shopify webhook delivery guarantee: at-least-once (not exactly-once). Retry schedule: up to 19 retries over 48 hours on non-2xx responses or > 5-second response times. Deduplication key: X-Shopify-Webhook-Id header (same UUID across all retries of the same event). Four idempotency implementation patterns: (1) Idempotency Key Table — unique constraint on event ID, reject duplicates before business logic; (2) Conditional State Transitions — only mutate state if current state is the expected predecessor; (3) Event Sourcing — append-only event log, idempotency inherent in log deduplication; (4) Outbox Pattern — atomic DB write + outbound message in single transaction. At 1M webhooks/month without idempotency: 5,000–20,000 duplicate processing events; duplicate fulfillment cost: $8,000–$25,000/month.

1. The Duplicate Webhook Problem at Scale

The Math of At-Least-Once Delivery

A well-maintained Shopify webhook endpoint with 99.5% uptime and p99 response time under 4 seconds will still generate retry events. Reasons:

  • Network variance: Even healthy endpoints occasionally take > 5 seconds on p99.5 requests during load spikes
  • Deployment windows: Every deploy creates a brief downtime window; Shopify accumulates and retries all events during this window
  • Upstream dependencies: An endpoint that depends on a database or cache that's briefly unavailable returns 500 → retry
  • Flash events: A product launch or viral post drives 10× normal order volume; the handler processes slowly and returns late responses → retries

At 100,000 events/month with a 0.8% duplicate rate: 800 duplicate events per month. If each duplicate causes a re-execution of the business logic:

  • 800 extra emails or WhatsApp messages sent
  • 800 extra inventory decrement operations (→ negative inventory)
  • 800 extra fulfillment API calls (→ duplicate fulfillments)
  • 800 extra financial records (→ accounting discrepancy)

The 0.8% rate sounds negligible. The consequences of 800 duplicate fulfillments per month do not.

2. Pattern 1 — Idempotency Key Table (Recommended for Most Apps)

The simplest and most effective idempotency pattern for Shopify webhooks:

Database Schema

CREATE TABLE processed_webhook_events (
  id           BIGSERIAL PRIMARY KEY,
  event_id     UUID        NOT NULL UNIQUE,  -- X-Shopify-Webhook-Id value
  topic        VARCHAR     NOT NULL,          -- e.g., 'orders/paid'
  shop_domain  VARCHAR     NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  result       JSONB                          -- Optional: store outcome for debugging
);

CREATE INDEX idx_pwe_event_id ON processed_webhook_events (event_id);

Idempotency Middleware

async function idempotentWebhookHandler(
  req: Request,
  res: Response,
  handler: (payload: unknown) => Promise<void>
) {
  const eventId = req.headers['x-shopify-webhook-id'] as string
  const topic = req.headers['x-shopify-topic'] as string
  const shopDomain = req.headers['x-shopify-shop-domain'] as string

  if (!eventId) {
    return res.status(400).json({ error: 'Missing X-Shopify-Webhook-Id header' })
  }

  // Attempt to claim this event (atomic insert)
  try {
    await db.processedWebhookEvents.create({
      data: {
        event_id: eventId,
        topic,
        shop_domain: shopDomain,
      },
    })
  } catch (error) {
    if (isUniqueConstraintViolation(error)) {
      // Already processed — idempotent success
      return res.status(200).json({ status: 'already_processed', eventId })
    }
    throw error
  }

  // First delivery — execute business logic
  try {
    await handler(req.body)
    res.status(200).json({ status: 'processed', eventId })
  } catch (error) {
    // Business logic failed — remove the event from the processed table
    // so Shopify can retry successfully
    await db.processedWebhookEvents.delete({ where: { event_id: eventId } })
    throw error // Return 500 → Shopify will retry
  }
}

Critical: If the business logic fails after the event is claimed, remove the claim record so Shopify's retry delivers a fresh processing attempt. The claim record must only persist on successful execution.

3. Pattern 2 — Conditional State Transitions

For operations that modify state (inventory, order status, fulfillment), add a guard condition that checks whether the state transition is valid from the current state:

async function handleOrderPaid(orderId: string) {
  const order = await db.orders.findUnique({ where: { shopifyOrderId: orderId } })

  // Guard: only process if the order is in the expected predecessor state
  if (order?.status !== 'pending_payment') {
    // Order is already processed (status is 'paid' or further)
    // Idempotent: return without re-executing
    logger.info(`Order ${orderId} already processed (status: ${order?.status})`)
    return
  }

  // State transition: pending_payment → paid
  await db.orders.update({
    where: { shopifyOrderId: orderId, status: 'pending_payment' }, // Optimistic lock
    data: {
      status: 'paid',
      paidAt: new Date(),
    },
  })

  // Downstream actions (notification, fulfillment trigger) run only after
  // successful state transition
  await triggerFulfillmentQueue(orderId)
  await sendOrderConfirmationMessage(orderId)
}

The where: { status: 'pending_payment' } optimistic lock ensures that if two concurrent deliveries both pass the initial guard (race condition), only one will successfully update — the other will update 0 rows and exit cleanly.

4. Pattern 3 — The Outbox Pattern for Reliable Event Publishing

When a webhook triggers a downstream event (publishing to BullMQ, sending to an external API, firing a notification), the naive implementation is:

// WRONG — dual-write without atomicity
await db.orders.update({ where: { id }, data: { status: 'fulfilled' } })
await bullmq.queue.add('send-confirmation', { orderId: id }) // May fail silently

If the process crashes between the DB write and the queue publish, the order is marked fulfilled but no confirmation is sent. The Outbox Pattern solves this:

// CORRECT — Outbox Pattern
await db.$transaction(async (tx) => {
  // Write business state change
  await tx.orders.update({ where: { id }, data: { status: 'fulfilled' } })

  // Write the outbound event intent to the outbox table (same transaction)
  await tx.outboxEvents.create({
    data: {
      event_type: 'order.fulfilled',
      payload: JSON.stringify({ orderId: id }),
      status: 'pending',
    },
  })
})

// A separate background worker reads the outbox and publishes to BullMQ
// This worker runs continuously and retries pending outbox events

// Outbox publisher worker
async function processOutboxEvents() {
  const pending = await db.outboxEvents.findMany({
    where: { status: 'pending' },
    take: 100,
    orderBy: { created_at: 'asc' },
  })

  for (const event of pending) {
    try {
      await bullmq.queue.add(event.event_type, JSON.parse(event.payload))
      await db.outboxEvents.update({
        where: { id: event.id },
        data: { status: 'published', published_at: new Date() },
      })
    } catch (err) {
      await db.outboxEvents.update({
        where: { id: event.id },
        data: { retry_count: { increment: 1 } },
      })
    }
  }
}

The outbox worker should itself be idempotent — using BullMQ's jobId (set to the outbox event ID) to prevent duplicate queue entries from multiple outbox worker runs.

5. BullMQ Retry Configuration for Downstream Reliability

The webhook handler puts events into BullMQ. The BullMQ worker processes them. Both sides need idempotency:

// Webhook handler: add to queue with jobId = event ID
await queue.add(
  'process-order-paid',
  { orderId, shopDomain },
  {
    jobId: eventId, // Deduplication: BullMQ won't add duplicate jobIds
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 2000,
    },
    removeOnComplete: { age: 86400 }, // Keep for 24h for debugging
    removeOnFail: false, // Keep failed jobs for investigation
  }
)

Setting jobId: eventId makes the BullMQ queue itself idempotent — if the same event ID is added twice (because the webhook middleware didn't dedup it), BullMQ deduplicates at the queue level.

6. HMAC Verification: Security Before Idempotency

Idempotency without signature verification is a security liability — a malicious actor could replay old event IDs or forge new ones. Verify the Shopify HMAC signature before processing any webhook:

import { createHmac, timingSafeEqual } from 'crypto'

function verifyShopifyWebhook(body: Buffer, signature: string, secret: string): boolean {
  const hmac = createHmac('sha256', secret).update(body).digest('base64')

  const signatureBuffer = Buffer.from(signature, 'base64')
  const hmacBuffer = Buffer.from(hmac, 'base64')

  if (signatureBuffer.length !== hmacBuffer.length) return false

  return timingSafeEqual(signatureBuffer, hmacBuffer)
}

// Usage in Express handler
app.post('/webhooks/shopify/:topic', express.raw({ type: '*/*' }), async (req, res) => {
  const signature = req.headers['x-shopify-hmac-sha256'] as string
  const secret = process.env.SHOPIFY_WEBHOOK_SECRET!

  if (!verifyShopifyWebhook(req.body, signature, secret)) {
    return res.status(401).json({ error: 'Invalid signature' })
  }

  // Proceed to idempotency check + processing
  await idempotentWebhookHandler(req, res, handlers[req.params.topic])
})

Use timingSafeEqual rather than string equality — timing attacks on HMAC verification are a real attack vector.

7. Reliability vs. Non-Idempotent Processing

Risk Level Without Idempotency With Idempotency
Duplicate fulfillment HIGH — retry → second fulfillment trigger ELIMINATED — event ID check
Duplicate notification HIGH — customer receives 2+ order confirmations ELIMINATED
Inventory double-decrement HIGH — retry depletes inventory by 2× ELIMINATED via conditional state check
Financial record duplication MEDIUM — duplicate order in accounting sync ELIMINATED
Concurrent processing race HIGH — two simultaneous deliveries MITIGATED — optimistic lock
System crash between write and notify Data loss (notification never sent) PREVENTED — Outbox Pattern
Implementation complexity Low Medium (3–5 days engineering)
Maintenance overhead Low (until incidents happen) Low (patterns are stable once built)

As detailed in our Scalable Data Pipelines and Zero-Downtime Infrastructure Hardening guides, webhook idempotency is one layer of a broader infrastructure hardening posture. The idempotency key table, the Outbox Pattern, and the BullMQ jobId deduplication work together to create a webhook processing layer that handles Shopify's at-least-once delivery guarantee safely at any volume.

FAQ Section

How long should I retain idempotency key records?

Shopify retries webhooks for a maximum of 48 hours. Retaining idempotency key records for 72 hours (3 days) provides a safe buffer. After 72 hours, the event will never be retried by Shopify, so the record serves no deduplication purpose and can be archived or deleted. For debugging purposes, consider retaining records for 30 days in cold storage — duplicate processing incidents often aren't detected immediately, and having the event ID history helps with post-incident analysis.

Should I implement idempotency at the Shopify webhook layer or the BullMQ worker layer?

Both — each layer has a different failure mode. The webhook layer idempotency (event ID table) protects against Shopify retrying the same delivery. The BullMQ jobId deduplication protects against the webhook handler enqueuing the same job twice (e.g., from a race condition). The BullMQ worker itself should also be idempotent (via conditional state transitions or Outbox-consumed markers) to protect against BullMQ retrying failed jobs with side effects already partially executed. Defense in depth: idempotency at every processing layer.

How do I test idempotency in development?

Shopify provides a webhook testing tool in the Partner Dashboard that can send sample payloads. For idempotency testing specifically: (1) Send the same webhook payload with the same X-Shopify-Webhook-Id twice within 1 second. The second call should return 200 OK with status: 'already_processed' and should NOT trigger any business logic. (2) Send a webhook, simulate a process crash (return 500), verify the event is removed from the idempotency table, then send again — should process successfully. (3) Send concurrent webhooks with the same event ID — only one should execute business logic (test for race condition safety).

What's the performance impact of the idempotency key table on high-volume webhook processing?

At 50,000 events/day (roughly 1.5M events/month), the idempotency table grows by 50,000 rows/day. With a 3-day retention and periodic cleanup, the table maintains approximately 150,000 rows — trivially fast for a UUID lookup with a proper index. The insert operation (with unique constraint check) is a single database round-trip, adding < 5ms to each webhook processing time. This is negligible compared to the business logic execution time (typically 50–500ms). Index on event_id column; consider a partial index on processed_at > NOW() - INTERVAL '72 hours' if table size grows into the millions.

Reliability Is a Design Decision

The gap between "works in testing" and "works at production scale" is usually not algorithmic complexity — it is the handling of edge cases that only appear at volume. Webhook retries are an edge case at 100 events/day and a daily operational reality at 100,000.

The brands that build idempotency from the start spend 3–5 engineering days on implementation and never think about it again. The brands that don't spend 3–5 engineering days per incident investigating why 847 orders were fulfilled twice, 1,200 customers received duplicate confirmation messages, or $12,000 in inventory was incorrectly decremented.

At-least-once delivery is not a problem. Designing for it is straightforward. Not designing for it is expensive.

Ready to harden your event-driven infrastructure?

Audit Your Webhook Infrastructure | Explore ViveReply Technical Stack | Read: Zero-Downtime Infrastructure Hardening

Ready to automate?

Put this into practice with ViveReply

Start Free TrialMore Blog