Skip to content

Alerts: troubleshooting checklist for a missing alert - #812

Merged
hlenke merged 3 commits into
mainfrom
docs/no-alert-troubleshooting
Nov 10, 2026
Merged

hlenke merged 3 commits into
mainfrom
docs/no-alert-troubleshooting

Conversation

@hlenke

@hlenke hlenke commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

What and why

Users whose check failed without a notification have no single place that tells them what to look at. The alerts overview explains states, thresholds and testing, the configuration page explains settings, and the notification log page explains deliveries, but nothing walks through "why didn't I get this alert?".

This adds a Troubleshooting missing alerts section to communicate/alerts/overview.mdx, right after "Testing your alerts". It starts from the check's Alerts table (which shows whether Checkly generated an alert and how many notifications went out) and splits the causes into three cases:

  • no alert generated: no state change (already failing), threshold not reached (retries count as one run), which settings level applies;
  • alert generated but no notifications (-): not subscribed (and group override uses the group's channels), the channel's notification events, muted check or group, a maintenance window silencing alerts, a channel subscribed after the failure started;
  • notifications sent: read the delivery status and result in the notification log.

It links to the existing configuration, maintenance window and notification log pages instead of repeating them. No existing text changed.

Two follow-ups:

  • the checklist names the channel types the Hobby plan sends alerts to; other subscribed channels are skipped;
  • platform/groups.mdx gets a warning: turning on Group override for alert settings selects Use the global account notification settings, but the group's alert channels still replace the checks' own, so a group without channels alerts no one.

How each claim was verified

Local stack at monorepo origin/main (e086a77755): API, webapp, scheduler, results daemon, alerting daemon and the Go runner, so checks really ran. Each scenario used its own API check and webhook channels pointing at a local listener; the check's target returned 200 or 500 on demand. "Alerts table" = check_alerts rows and the check overview page; "notifications" = notification log rows, the overview's Notifications column and requests the listener received.

Claim How it was verified
The check overview has an Alerts table with Failed / Recovered rows and a Notifications column; it links to the notification log entry UI on the check overview page; the link opens alerts/notifications?alertNotificationId=…
Notifications column shows "1 sent", "1 sent, 1 failed", or - when nothing was sent UI: delivered webhook, webhook to a closed port (after retries), muted check
A still-failing check sends nothing new 3 failing runs in a row with threshold 1: one ALERT_FAILURE, one delivery
With "alert after 2 failures", the first failed run sends nothing Run 1 failing: no alert; run 2: alert delivered
A run and its retries count as one failed run Fixed retry strategy with 2 retries, threshold 2: 3 failed attempts, failed-run counter 1, no alert
Not subscribed: the alert is generated, no channel notified Alert row, no notification rows, nothing at the listener
Group with Group override for alert settings: the group's channels are used, the check's own ignored Check subscribed to channel A, group to channel B: only B notified. The check editor shows the group's channels and "Controlled by" the group
Group without override: the check's own channels are used Same setup with override off (alert settings and global flag null, as the UI writes them): only the check's channel notified
A channel only sends the selected Notification events (a check fails, a check degrades, a check recovers) Channel with a check fails off: no failure notification; labels from the channel editor
A recovery from degraded only goes to channels with both a check degrades and a check recovers Degraded check subscribed to two channels: the one without degraded got neither the degraded alert nor the recovery; the other got both
A muted check sends no notifications; muting a group mutes every check in it Muted check, and unmuted check in a muted group: alert rows, no notifications. Muted label in check and group editors
A maintenance window that silences alerts suppresses notifications; a failure that started during it and still fails after it ends sends no failure alert Window with silence-alerts tag: alert row, no notification. Completed the maintenance, ran the still-failing check: nothing sent
A channel subscribed while the check is failing gets nothing for that failure, only for the next state change Subscribed during the failure, ran failing again: nothing; on recovery the channel got the recovery
On Hobby, only email, Slack, Microsoft Teams, Discord, Telegram and generic webhook channels get alerts; other subscribed channels (e.g. SMS, phone call, PagerDuty, Opsgenie, FireHydrant) are skipped and get no notification log entry Hobby fixture account, failing API check subscribed to a generic webhook, an MS Teams webhook and a FireHydrant webhook (all to a local listener), a PagerDuty channel and an SMS channel (SMS created while the account was on Team, then moved back to Hobby, as after a downgrade; Hobby can't create SMS channels, 402). Failure alert: notification log rows and listener requests for the generic and MS Teams webhooks only; nothing for FireHydrant, PagerDuty or SMS. Control: same check after moving the account to Team (PagerDuty/SMS subscriptions off): the recovery also reached the FireHydrant webhook. Email, Slack, Discord, Telegram and Opsgenie: same allowlist in the alerting daemon (code)
Turning on Group override for alert settings selects Use the global account notification settings; the group's alert channels still replace the check's own; with none selected, nothing is sent Web app: on a group without the override, ticking it showed Use the global account notification settings selected and every alert channel off; saved (DB: useGlobalAlertSettings true, no group subscriptions). A failing check in that group, subscribed to its own webhook channel: failure alert generated, no notification log rows, nothing at the listener
The notification log shows Delivery status (Successful / Failed) and Notification result UI: a 200 response shown as Successful with the response; a closed port shown as Failed, "Server never responded"

What couldn't be tested

  • Third-party destinations (email, Slack, SMS, incident tools) can't receive notifications from a local stack, so delivery was tested with webhook channels only. The section only says that a Successful status means the receiving service accepted it.
  • The local stack refuses webhooks to loopback addresses, so the scratch build allowed 127.0.0.1 for the local listener. Nothing else was changed.
  • Time-based escalation isn't tested here; the section links to the existing configuration docs for it.
  • Hobby channel types: delivery was tested with webhook channels; SMS and PagerDuty were checked by the absence of any send attempt. No docs page lists alert channels per plan, so only the checklist names them.

Left out on purpose: how recoveries reach channels that didn't get the failure notification (treated as a bug).

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
checkly-422f444a 🟢 Ready View Preview Nov 10, 2026, 1:19 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

Add a missing-alerts checklist item for channel types the Hobby plan
doesn't send to, and a warning on the groups page that a group alert
override with no alert channels alerts no one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@hlenke hlenke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to merge · Every claim in the new troubleshooting section and the group warning matches the alerting and webapp source; one optional wording tweak inline.

Nothing blocks this PR. CI passes (docs-gate, preview-checks, static-docs-checks, Mintlify link-rot and spellcheck), it's mergeable, and main hasn't changed either file since the merge base.

Placement: the new ## Troubleshooting missing alerts section sits at the end of communicate/alerts/overview.mdx, after "Testing your alerts". That's the right spot: it doesn't push down the page's main content.

Overlap: #814 also edits platform/groups.mdx, but in a different hunk ("Mixing Checks and Monitors in a Group", around line 167; this PR adds the Warning under "Retries & Alerting", around line 128). They don't conflict and can land in either order.

Inline: 0 blockers, 0 important, 1 wording (optional).

What I checked
  • Read the whole diff from merge base e0e0277f: communicate/alerts/overview.mdx (+26) and platform/groups.mdx (+4). Checked the heading outline of both pages at the PR head.
  • gh pr checks 812: all pass. gh pr view: MERGEABLE / CLEAN. git log e0e0277f..origin/main: no new commits on main.
  • pr_impact.py: only an info note that /communicate/alerts/overview is a hub page. The PR changes no slugs or anchors.
  • Anchors exist on main: configuration#escalation-configuration ("## Escalation Configuration"), configuration#configuration-inheritance ("### Configuration Inheritance"), and maintenance-windows/overview#silence-alerts ("#### Silence alerts").
  • Hobby channel allowlist: monorepo apps/backend/api/daemons/alerting-daemon/src/helpers.ts (origin/main@84921d2637). On Hobby, only email and Slack/Slack App channels, plain webhooks, and Microsoft Teams, Discord and Telegram webhooks pass the plan filter. Every other channel is dropped before notifications are created. This matches line 126.
  • Notification events and degraded recovery: the same file sends a recovery from degraded only to channels with both degraded and recovery turned on. This matches line 125.
  • Channel selection: AlertingDaemon.ts applies the plan filter, then the event filter, to the check's and group's subscriptions.
  • UI labels in apps/webapp/src: "Notification events" (PageAlertChannelBuilder.i18n.ts); "a check degrades" (channel builder); "Delivery status" and "Notification result" (AlertNotificationsDetail.vue); "Retries & Alerting" (group builder tabs and check builders); "Group override" + "for" + section title, which renders as "Group override for alert settings" (OverrideByGroupCheckbox); "Use the global account notification settings" (ResourceAlertSettingsGlobalSelector.vue); "Muted" (i18n/en.ts); and the "N sent" count in the check overview alerts row (CheckOverviewAlertsRowLogsLink.vue).
  • Group override defaults: the webapp's alert-settings unit tests cover the reset to defaults when the group override is turned on. The PR's own end-to-end run (described in its body) showed that this selects the global settings with no group channels.
  • Local app unavailable (BUSY), so I verified against source. I didn't rerun the PR author's end-to-end scenarios.

Comment thread communicate/alerts/overview.mdx Resolved
@hlenke
hlenke enabled auto-merge (squash) November 10, 2026 01:20
@hlenke
hlenke merged commit ed94f26 into main Nov 10, 2026
7 checks passed
@hlenke
hlenke deleted the docs/no-alert-troubleshooting branch November 10, 2026 01:21

This branch was successfully deployed

1 active deployment
staging — 14b90375 Deployed Nov 10, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant