Deduplication

Webhook providers sometimes send the same event multiple times — due to retries, network issues, or provider bugs. Hookbase's deduplication system detects and silently drops duplicate events before they reach your destinations.

How It Works

When deduplication is enabled on a source, each incoming event is assigned a dedup key based on the configured strategy. This key is checked against a sliding time window. If a matching key is found, the event is dropped and returns 200 OK to the provider (so it stops retrying).

Enabling Deduplication

Enable dedup on a source via the API or dashboard:

curl -X PATCH https://api.hookbase.app/api/sources/src_abc123 \
  -H "Authorization: Bearer whr_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "provider_id",
    "dedupWindowHours": 24
  }'

Strategies

auto (Default)

Automatically detects the best strategy based on the source's provider setting:

provider_id

Extracts the event ID from provider-specific headers:

If the header is missing, the event is accepted (not dropped).

payload_hash

Computes a SHA-256 hash of the entire request body. Two events with identical payloads within the window are considered duplicates.

idempotency_key

Uses a custom header (specified in dedupCustomHeader) as the dedup key. This gives you full control over what constitutes a duplicate.

curl -X PATCH .../sources/src_abc123 \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "idempotency_key",
    "dedupCustomHeader": "X-Idempotency-Key"
  }'

Your webhook sender must include this header:

curl -X POST https://api.hookbase.app/ingest/myorg/mysource \
  -H "X-Idempotency-Key: order-12345-payment-confirmed" \
  -H "Content-Type: application/json" \
  -d '{"event": "payment.confirmed", "order_id": "12345"}'

none

Explicitly disables deduplication. Every event is accepted regardless of duplicates. This is the default when dedupEnabled is false.

Window Configuration

The dedupWindowHours controls how long dedup keys are remembered:

Monitoring Duplicates

Dropped duplicates are tracked in the source's metrics. View them in the dashboard or via the API:

curl https://api.hookbase.app/api/analytics?sourceId=src_abc123 \
  -H "Authorization: Bearer whr_your_api_key"

The response includes duplicatesDropped in the source breakdown.

Examples

GitHub — Auto Detection

curl -X POST https://api.hookbase.app/api/sources \
  -H "Authorization: Bearer whr_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GitHub Webhooks",
    "slug": "github",
    "provider": "github",
    "dedupEnabled": true,
    "dedupStrategy": "auto",
    "dedupWindowHours": 24
  }'

Stripe — Provider ID

curl -X PATCH .../sources/src_stripe \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "provider_id",
    "dedupWindowHours": 48
  }'

Custom API — Idempotency Key

curl -X PATCH .../sources/src_custom \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "idempotency_key",
    "dedupCustomHeader": "X-Request-ID",
    "dedupWindowHours": 24
  }'

Related

• Sources API — Configure dedup fields
• Pipeline Architecture — Where dedup fits in the processing flow

See Also

• Sources — Configure source-level deduplication
• Events API — View event history
• Best Practices — Idempotency patterns
• Filters — Additional event filtering