Best Pattern for Handling Duplicate and Out-of-Order GitHub Webhook Events?

[daisygrace475-spec]

🏷️ Discussion Type

Bug

💬 Feature/Topic Area

API

Body

We are building an internal integration that uses GitHub webhooks to update deployment and project-status records in another system.

During testing, we noticed that the same webhook can occasionally be delivered more than once. Events may also arrive late or in a different order from the actions shown in GitHub. The integration must process valid events without creating duplicate records or overwriting newer data.

Our development team at AppVerticals, a mobile app company, is considering this approach:

Store each GitHub delivery ID for idempotency.

Verify the webhook signature before processing.

Add accepted events to a queue.

Retry temporary failures with exponential backoff.

Record event timestamps before applying updates.

Send repeatedly failing events to a dead-letter queue.

For teams operating GitHub Apps at scale, is the delivery ID sufficient for deduplication, or should we create an internal idempotency key using the repository, event type, and resource ID?

I would also appreciate guidance on handling out-of-order events, particularly for pull-request, workflow-run, and deployment-status webhooks.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[BhuwanJoshi-01]

Your design is solid, and the delivery ID is the right key for deduplication. It just can't handle ordering, so I'd add a second check when you update each record.

An internal key of repo + event type + resource ID won't work as a deduplication key on its own, because one pull request or run legitimately sends many events. It would drop real updates.

1. Receive the event

1. Verify X-Hub-Signature-256 against the raw request body.
2. Save X-GitHub-Delivery and add the event to your queue in one transaction.
3. Return a 2XX. GitHub treats a response that takes over 10 seconds as a failure.

2. Skip repeat deliveries

CREATE TABLE webhook_deliveries (
  delivery_id  text PRIMARY KEY,
  status       text NOT NULL DEFAULT 'queued',
  received_at  timestamptz NOT NULL DEFAULT now()
);

INSERT INTO webhook_deliveries (delivery_id)
VALUES ($1)
ON CONFLICT (delivery_id) DO NOTHING;

If the insert affects 0 rows, return a 2XX and don't queue it again, unless that delivery is marked as failed. A redelivery keeps the same X-GitHub-Delivery value, so skipping failed IDs would block your own recovery.

3. Never let an older event overwrite newer data

Give each resource a version you can compare, and only update when the incoming version is higher. For workflow_run, use the attempt number plus a status rank (requested = 1, in_progress = 2, completed = 3):

INSERT INTO workflow_runs (run_id, run_attempt, status_rank, status, conclusion)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (run_id) DO UPDATE SET
  run_attempt = EXCLUDED.run_attempt,
  status_rank = EXCLUDED.status_rank,
  status      = EXCLUDED.status,
  conclusion  = EXCLUDED.conclusion
WHERE (EXCLUDED.run_attempt, EXCLUDED.status_rank)
    > (workflow_runs.run_attempt, workflow_runs.status_rank);

A late in_progress can't undo completed, and a re-run still replaces the earlier attempt. Add any other status values you handle, such as queued, to the rank.

4. Choose a version for each event type

Event	Record key	How to compare
workflow_run	workflow_run.id	run_attempt, then status rank
deployment_status	deployment.id	deployment_status.created_at, with deployment_status.id as a tiebreaker
pull_request	pull_request.id	pull_request.updated_at, or refetch (see below)

5. Refetch when timestamps can't settle it

Timestamps only have one-second precision, so two pull request events can tie. For fields that matter, treat the webhook as a signal that something changed, then get the current state from the REST API. You'll use more API requests, but you won't have to work out the order yourself.

6. Plan for missed events

GitHub doesn't automatically redeliver failed deliveries, so run a scheduled job that lists recent deliveries through the REST API and redelivers the failed ones.

Your dead-letter queue covers events that fail on your side, and this job covers events that never reached you. Keep delivery IDs for at least as long as you might replay them.

Drafted with AI assistance.

[fynchfries]

I'd recommend using the GitHub delivery ID (X-GitHub-Delivery) as the primary deduplication key. However, I wouldn't rely on it alone for maintaining data consistency.

A good approach would be:

• Deduplication: Store delivery IDs and enforce a unique constraint to prevent processing the same delivery twice.
• Signature verification: Validate X-Hub-Signature-256 before accepting the payload.
• Out-of-order events: Don't rely solely on webhook timestamps. Fetch the latest resource state from the GitHub API when event ordering matters.
• Retries: Use exponential backoff for temporary failures and a dead-letter queue for events that repeatedly fail.
• Idempotency: Use an internal resource-level key when applying updates, so processing an event more than once doesn't create duplicate records.
  For pull requests, workflow runs, and deployments, I'd also recommend tracking their GitHub IDs and checking the current resource state before applying updates.

My suggestion: Use delivery IDs for duplicate delivery detection and resource IDs for maintaining state consistency. They solve two different problems.