Skip to content

docs(webhooks): which variables are filled, escaping, retries and dedup keys - #817

Merged
hlenke merged 9 commits into
mainfrom
docs/webhook-payload-variables
Nov 11, 2026
Merged

hlenke merged 9 commits into
mainfrom
docs/webhook-payload-variables

Conversation

@hlenke

@hlenke hlenke commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

What changed and why

The webhooks page lists every variable but doesn't say which ones have a value for which check type, how double brackets escape values outside the body, how a failure relates to its recovery and reminders, or how to recognize a retried webhook. Those are the questions people building incident integrations on top of webhooks keep running into. This PR extends integrations/alerts/webhooks.mdx only:

  • Escaping: the existing triple-bracket note now says that {{NAME}} HTML-escapes values in the URL, headers and query parameters too (& → &, = → =), and when to use triple brackets. A short line after the secrets callout covers a URL promoted with the lock icon.
  • Variable table: API_CHECK_RESPONSE_STATUS_* are filled for URL monitors too (the old text said API checks only). It also adds the missing STATE variable, points CHECK_ERROR_MESSAGE to the new availability table, notes that TAGS includes group tags, and gives the heartbeat STARTED_AT format.
  • New "Which variables are filled per check type": a two-column table (variable, when it has a value) for API checks, URL, TCP, DNS and heartbeat monitors, when CHECK_ERROR_MESSAGE is empty, and the fact that there is no variable for the request URL or method. It now also has a Browser checks column, how the Browser check error message is cut, and a missed-ping heartbeat example (STATE FAILING).
  • GROUP_NAME: empty for heartbeat monitors, even in a group (code only: the heartbeat branch of tokenizeCheckResult returns before GROUP_NAME is set).
  • Opsgenie example: tags use triple braces, so env=prod isn't sent as env=prod.
  • Using variables: a check's own environment variable (Browser checks and Playwright Check Suites) wins over an account variable with the same name.
  • New "Body format": which Content-Type we send for valid JSON, for other text, and when you set the header yourself.
  • New "Render tags as an array or a string" (under the tagValue helper).
  • Retries: a retry resends the same body, timestamp and signature, and a response slower than 5 seconds is retried.
  • New "Failures, recoveries and reminders": matching a recovery to its failure, which ALERT_TYPE each state change sends, what a reminder carries, and a deduplication key.
  • Group variables: the intro note now says a webhook can use a group's variables when the check that triggers the alert belongs to that group, whether the webhook is attached to the check or to the group.

No overlap with open PRs: #811 edits slack.mdx and #812 edits communicate/alerts/overview.mdx; this PR doesn't touch either.

How each claim was verified

Tested end to end on a local Checkly stack built from the latest product code. Real check runs triggered real alerts, and the webhooks went to a local HTTP listener that logged every request's method, URL, headers and raw body. The channel template used every variable. Checks were driven between passing, failing and degraded by changing what the target returned.

Claim How verified
{{NAME}} HTML-escapes in URL, headers, query params and body (&→&, =→=); {{{NAME}}} doesn't A test secret containing & and = referenced in URL, header, query param and body; raw requests captured
A variable or secret holding a whole URL with a query string breaks with {{NAME}} and works with {{{NAME}}} Secret URL …/hook/x?a=1&b=2: request arrived at ?a= (rest lost) with double, intact with triple
Lock icon promotion writes {{NAME}}; a promoted URL with a query string then breaks Web app: typed a URL, clicked the lock icon, saved the "Save as secret" dialog. The field became {{WEBHOOK_URL}}, the account secret was created, and the next alert went to ?token= with the rest dropped
Triple brackets are accepted in the webhook URL field Saved in the web app and through the API; delivered
API_CHECK_RESPONSE_STATUS_CODE/_TEXT filled for API checks and URL monitors; TCP 200/SUCCESS or 500/ERROR; empty for DNS and heartbeat Captured payloads per type
CHECK_ERROR_MESSAGE: filled on request errors (refused connection) for API/URL, empty on assertion failures, empty for TCP, DNS, heartbeat; HTML-escaped Captured payloads: API/URL with status 500 vs refused connection, TCP refused, DNS NXDOMAIN, heartbeat recovery
RESPONSE_TIME, RUN_LOCATION, REGION empty for heartbeats; STATE only for heartbeats (e.g. LATE) Captured heartbeat recovery payload
Heartbeat STARTED_AT is UTC YYYY-MM-DD HH:mm:ss Captured heartbeat payload
TAGS includes the group's tags; {{#each}} with {{{this}}} gives a JSON array, {{{TAGS}}} a comma-separated string, {{this}} escapes = Captured payloads from a check in a group with tags team:payments, env=prod
No variable for the request URL or method Webhook variable list in the product code, plus captured payloads
Valid JSON is sent as application/json; other text as application/x-www-form-urlencoded; a Content-Type: text/plain header sends plain text; Content-Type: application/json on invalid JSON sends one JSON string Four channels with those templates and headers; captured headers and raw bodies
A retry resends the same body, x-checkly-timestamp and x-checkly-signature Listener answering 500: 6 deliveries per notification (10/20/40/80/160 s apart), byte-identical body, same timestamp and signature
A response slower than 5 s is retried even though the request was received Listener answering 200 after 6 s: same 6 deliveries
Failure and recovery share CHECK_ID; CHECK_RESULT_ID/STARTED_AT belong to the triggering run (recovery = first passing run) Captured for API, URL, TCP and DNS
ALERT_TYPE per state change, including degraded→failing = ALERT_FAILURE, failing→degraded = ALERT_DEGRADED, failing→degraded→passing = ALERT_RECOVERY, passing→degraded→passing = ALERT_DEGRADED_RECOVERY Captured payloads; matches the alerting code
Degraded notifications only reach channels with "a check degrades" on Channels with it off received no ALERT_DEGRADED
Reminders: same ALERT_TYPE, IS_REMINDER true, REMINDER_SEQUENCE 1, 2…, result ID and STARTED_AT of the latest failing run Check with 2 reminders every 5 minutes; captured both for URL and TCP monitors
CHECK_RESULT_ID + ALERT_TYPE + REMINDER_SEQUENCE identify one notification Unique across all 29 captured notifications; identical across retries
Browser checks: CHECK_ERROR_MESSAGE is the first error, cut to 300 characters, then line breaks removed; status code/text empty; RESPONSE_TIME, RUN_LOCATION, REGION filled; STATE empty Production test account: a scheduled failing Browser check (runtime 2026.04) sending to a webhook on a public echo service; payload read back from the notification result (GET /v1/alert-notifications). Two failure alerts: the 318-character multi-line error arrived as 290 characters, lines joined, cut before its last words; status fields ""; RESPONSE_TIME 5735, RUN_LOCATION Frankfurt, REGION eu-central-1 (also on the recovery)
Missed heartbeat ping: ALERT_FAILURE, STATE FAILING, STARTED_AT like 2026-11-09 00:30:21 Production test account: heartbeat with a 30 s period and no grace, pinged once, then not again. Failure alert 39 s after the ping, payload as in the example
A check-level variable overrides an account variable in the template (Browser checks and Playwright Check Suites) Production test account: account variable ZZ_DOCS_VAR=account, the same key =check on a Browser check and on a Playwright Check Suite. Both checks' alerts rendered check; the heartbeat (no check variables) rendered account
Group variables render when the alerting check belongs to the group, whether the webhook is attached to the check or to the group; a check outside the group gets none, even through a channel that is also attached to the group Local stack, scheduled API checks, template with a group variable: check in the group with the channel on the check (group alert override off) → value; same check with the override on and the channel on the group → value on its recovery and its next failure; a check outside the group on a channel also attached to the group → empty

What couldn't be tested, and what was left out

  • Multistep, Playwright and ICMP checks can't run on this local stack (ICMP needs raw sockets on macOS). Browser checks were tested on a production test account instead. Multistep and ICMP are left out of the availability table rather than described from code alone.
  • Playwright Check Suites are left out of the availability table: on a production test account, two failure alerts from a suite with two failing tests had an empty CHECK_ERROR_MESSAGE, although the code joins the failures with ; . Reported as a possible product bug.
  • Heartbeat failure alerts come from a scheduled job that doesn't run on the local stack. The failure payload was captured on a production test account instead; the local recovery payload is unchanged.
  • SSL alerts and AI analysis variables weren't exercised. Their table rows are unchanged.
  • Test setup: one local-only setting was relaxed so webhooks could reach the listener on this machine. Payload rendering, queueing and retries were unchanged.
  • Left out pending product decisions: the full alert event sent when a webhook has no template, and a few behaviors that may be bugs.

🤖 Generated with Claude Code

…tries and dedup keys

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
checkly-422f444a 🟢 Ready View Preview Nov 11, 2026, 3:15 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

… variables

Captured real webhook payloads: Browser CHECK_ERROR_MESSAGE (first error, 300
chars, line breaks removed), a missed heartbeat ping (STATE FAILING), and
check-level variables overriding account variables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nnel

Tested on a local stack with API checks: a check in a group renders the
group's variables whether the webhook is subscribed on the check or through
the group; a check outside the group renders them empty, even through a
channel that is also attached to the group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@hlenke hlenke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs changes · The variable, escaping, retry, and deduplication claims match the webhook code. Three Important items close gaps the new text opens: the Opsgenie example now contradicts the new triple-bracket advice, CHECK_ERROR_MESSAGE has a second source, and GROUP_NAME is empty for heartbeats.

CI is green, the PR is mergeable (CLEAN), and main hasn't touched webhooks.mdx since the merge base.

Overlap with #811. The description says there is no overlap with #811, but #811 also edits integrations/alerts/webhooks.mdx (it adds a Slack incoming webhook example between Pushover and Trello). The hunks don't touch each other and git merge-tree of the two heads is clean, so either can land first. #811's new example uses triple braces for the same reason this PR gives, so the two agree.

Section placement. The new subsections sit under Using Variables, next to the variable table; Render tags sits under the tagValue helper; and Failures, recoveries and reminders comes between Webhook Retries and Webhook Examples. All of these read in a sensible order.

Inline: 3 Important (one is on an unchanged line, so GitHub shows it below), 3 Wording.

What I checked
  • Read the full diff from the merge base e0e0277f, and integrations/alerts/webhooks.mdx at the PR head.
  • gh pr checks 817: all pass. gh pr view: MERGEABLE/CLEAN. git log <base>..origin/main -- integrations/alerts/webhooks.mdx: no commits.
  • pr_impact.py: no broken anchors; webhooks.mdx is a hub (26 inbound pages), and no existing heading changes. The new in-page anchors match their headings.
  • Overlap: gh pr list --json number,files, and git merge-tree --write-tree of PRs 811 and 817 (clean).
  • Local app unavailable (BUSY: someone's own Tilt is on the ports), so I verified against source instead. Monorepo origin/main@84921d2637:
    • apps/backend/api/src/common/services/alert-notification-service/channels/webhook/index.ts:
      • The URL, headers, and query parameters go through the same Handlebars compile as the body, so double brackets HTML-escape there too.
      • Valid JSON is sent as a parsed object; otherwise the rendered string is sent.
      • The request timeout is 5 seconds.
      • The signature is computed once and stored on the queued request.
      • Variable tokens: heartbeats return only the common tokens plus STATE. API and URL checks fill CHECK_ERROR_MESSAGE from the request error. Browser and Multistep checks use the first error, trimmed, cut to 300 characters, with line breaks removed. CHECK_ERROR_MESSAGE is always HTML-escaped. The run's schedule error, when present, takes precedence for every type. ALERT_TYPE mapping: degraded→failing is ALERT_FAILURE, failing→degraded is ALERT_DEGRADED.
    • apps/backend/api/daemons/alerting-daemon/src/WebhooksDaemon.ts: a failed request is re-queued with the same config (same body, x-checkly-timestamp, x-checkly-signature), up to 5 retries with exponential backoff.
    • apps/backend/api/src/common/services/alert-notification-service/AlertNotificationService.ts: check environment variables override account variables on a key collision, for every check type.
    • a check degrades label: apps/webapp/src/pages/alert-channel-builder/PageAlertChannelBuilderNotificationsEvents.vue.
  • The Content-Type for a non-JSON body and for Content-Type: application/json on invalid JSON comes from axios's default request transform. That's consistent with the code path above and with the PR's captured requests; I didn't run it.
  • Could not verify from source: the heartbeat STARTED_AT format (a ping timestamp in that format exists in apps/backend/api/src/modules/heartbeats/HeartbeatsController.ts, but I didn't trace it into the alert), the TCP 200/SUCCESS or 500/ERROR values, and the degraded→passing case that yields ALERT_RECOVERY. The PR's captured payloads cover all three.

Outside this PR's diff

  • integrations/alerts/webhooks.mdx:387: Important (unchanged line, integrations/alerts/webhooks.mdx line 387 at the PR head) The Opsgenie example still renders tags with "{{this}}", which line 193 of this PR now says turns env=prod into env&#x3D;prod. Update it in this PR so the page doesn't contradict itself:
  "tags": [{{#each TAGS}} "{{{this}}}" {{#unless @last}},{{/unless}} {{/each}}]

Comment thread integrations/alerts/webhooks.mdx Resolved
Comment thread integrations/alerts/webhooks.mdx Resolved
Comment thread integrations/alerts/webhooks.mdx Outdated
| `CHECK_ERROR_MESSAGE` | Request errors | Request errors | Empty | Empty | Empty | First error |
| `API_CHECK_RESPONSE_STATUS_CODE`, `API_CHECK_RESPONSE_STATUS_TEXT` | Response status | Response status | `200` / `SUCCESS` or `500` / `ERROR` | Empty | Empty | Empty |
| `RESPONSE_TIME`, `RUN_LOCATION`, `REGION` | Filled | Filled | Filled | Filled | Empty | Filled |
| `STATE` | Empty | Empty | Empty | Empty | Filled | Empty |

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important Consider a GROUP_NAME row. The heartbeat branch of the webhook token builder returns before GROUP_NAME (and the SSL variables) are set, so GROUP_NAME is empty for a heartbeat monitor even when it belongs to a group. For the other types it's the group name, or empty outside a group (tokenizeCheckResult in monorepo apps/backend/api/src/common/services/alert-notification-service/channels/webhook/index.ts). Readers who route alerts by group are the ones this table is for. I verified this from source only, since the local app wasn't available.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Comment thread integrations/alerts/webhooks.mdx Resolved
Comment thread integrations/alerts/webhooks.mdx Resolved
…UP_NAME

Two columns instead of seven, a GROUP_NAME row (empty for heartbeat
monitors), Multistep checks next to Browser checks, and triple braces
for tags in the Opsgenie example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hlenke
hlenke merged commit 6f164b4 into main Nov 11, 2026
7 checks passed
@hlenke
hlenke deleted the docs/webhook-payload-variables branch November 11, 2026 03:20

This branch was successfully deployed

1 active deployment
staging — 440718c9 Deployed Nov 11, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant