Alert Routing, Tier Mode, Policy Mode, and On-Call

The three systems#

When an incident opens, Reliable first waits for the AI classifier to decide whether it pages at all (see Classify before paging). To page it, Reliable answers three questions in order:

1. Audience: who is a candidate to be paged? (Decided by your project's routing mode: tier or policy.)
2. On-call filter: of those candidates, who is actually on shift right now?
3. Severity filter: of those on-shift engineers, who has opted in to alerts at this severity level?

Each project picks one routing mode. The on-call schedule and severity filter then sit on top of whichever mode you chose, they apply to both.

Why two systems? Why not just policy?#

The routing mode and the on-call schedule answer different questions, which is why they're separate:

• Routing mode answers "who is the right group of people to handle this kind of incident?" That's a property of the alert (payment errors → payments team, DB errors → infra team).
• On-call schedule answers "of that group, who is awake / on shift right now?" That's a property of time (the payments team has 4 engineers; only one is the 3am on-call).

A hospital is the cleanest parallel. The department (cardiology, ER) is the policy, it decides which specialists could handle a case. The shift roster decides which one of them is in the building right now. You need both: routing without scheduling pages your CTO at 3am for every transient error; scheduling without routing pages whoever happens to be on shift, even if they have nothing to do with the failing service.

Collapsing them into one config would force every policy step to carry its own rotation, multiplying complexity, and would break the severity bypass: "for SEV1, page everyone in the audience regardless of who's on shift" only makes sense when the two layers are independent. Keeping them separate is the same model PagerDuty and Opsgenie use, for the same reason.

Tier mode#

Tier mode is a straight escalation ladder. Each engineer is tagged with a tier, tier_1, tier_2, tier_3, or lead. When an incident fires:

1. Page all tier_1 engineers (step 1).
2. If nobody acks within the timeout (default 20 min), escalate to tier_2.
3. Then tier_3, then lead.
4. A tier with nobody to page (no active engineers in it, or none whose min_severity the incident meets) is skipped straight away, without waiting out a timeout. A team whose only engineer is a lead gets paged at once, as long as the incident meets the lead's min_severity (medium by default).
5. Past lead, the engine stops escalating, the incident stays open and unacked until someone manually intervenes.

There's no config beyond setting tiers on engineers. Best fit: small teams, one rotation, you want it to just work.

Policy mode#

Policy mode replaces the fixed tier ladder with named policies. A policy has ordered steps; each step targets either a tag (e.g. all engineers tagged backend) or a specific engineer. Each step has its own timeout and minimum severity.

One policy per project is marked as the default policy: all incidents use it unless overridden. Steps can also be flagged notify_only (observers, paged in parallel, never block escalation, no ack required) for stakeholders who want awareness without responsibility.

Example: "Payment Service" policy

  Step 1 (5 min)   → tag: payments-oncall
  Step 2 (10 min)  → tag: backend-leads
  Step 3 (15 min)  → engineer: cto@company.com
  Step 4 (observer) → tag: managers      ← always paged, never blocks

Best fit: multiple rotations, route by service / team, observer notifications, custom escalation timelines per service.

The on-call schedule#

The on-call schedule is separate from routing mode. It works the same way whether you're in tier or policy mode: after the audience is resolved, the schedule narrows it to whoever is actually on shift right now.

• Schedule disabled (project default): no filtering. The whole audience gets paged regardless of time of day.
• Schedule enabled, someone on shift: only engineers currently on the rotation receive the page.
• Schedule enabled, nobody on shift: everyone in the original audience gets paged, the system never silently drops an alert because of a roster gap.

A bypass threshold lets you say "page everyone regardless of schedule when severity is at least X". Default bypass is critical: SEV1s wake the whole team.

The severity filter#

Severity is checked while the audience is chosen, before the on-call schedule narrows it, per engineer (tier mode) or per step (policy mode):

• Tier mode: each engineer's min_severity preference (defaulting to project-wide setting, defaulting to medium). Incidents below that severity skip the engineer.
• Policy mode: each step has its own min_severity. Steps below threshold are skipped (the engine tries the next step in the same call, no wasted timeout cycle).

Classify before paging#

A new incident does not page the moment it opens. It first waits for the AI severity classifier, for up to 30 seconds (TRIAGE_TIMEOUT_SECONDS on the backend). The severity it ends up with decides what happens:

• Low: a quiet push notice goes to the people paging would have picked (same routing mode and on-call schedule), with no alarm. The min_severity filters are skipped for the notice, because with them a low incident would have nobody to tell. Nobody is paged and nothing escalates.
• Medium or higher: paged exactly as before, through the three systems above.
• The classifier fails or doesn't answer in time: paged. A broken or slow classifier never means silence.

The final severity is the classifier's rating when it is confident, and otherwise the severity the incident was reported with. If an incident that only got a notice later rises to medium or higher, it is paged then.

The dashboard shows the outcome on every new incident: a Paged or Notified only chip on the incident page with the reason in plain words, a Notified only marker in System Activity, and a triage entry (plus one entry per notice sent) on the activity trail. There is no chip while the classifier is still deciding, or on incidents from before this existed.

A walkthrough: error → page#

Suppose your frontend fires an error. Here's the trace:

1. The SDK sends the error to /api/v1/ingest/errors with severity medium (the SDK's default).
2. The backend opens an incident. If your project is in policy mode AND has a default policy set, the incident is attached to that policy. Otherwise it goes through tier mode.
3. Nobody is paged yet: the AI classifier rates the incident first (up to 30 seconds). If the final severity is low, the routing below picks who to tell (without the min_severity filter) and they get a quiet notice instead of a page; the trace ends there. If the classifier fails or is too slow, the incident pages anyway.
4. The audience resolver picks candidates: the lowest tier with someone whose min_severity is at or below medium (tier mode), or the first policy step whose min_severity the incident meets and that has targets (policy mode).
5. If on-call schedules are enabled, only currently-rostered candidates remain (or everyone, if nobody is on shift, see above).
6. What's left receives the page, full-screen takeover on the mobile app, plus every other enabled channel (Slack, email, webhook).
7. If the incident's severity rises later AND nobody has been paged for it yet (it only got a notice, or the severity filter left nobody to page), it is paged at the new severity. That's the safety net for the under-reported case.

Repeats and auto-resolve#

An error with the same fingerprint as an incident that is still open or acknowledged does not open a new incident. It raises that incident's occurrence count and does not page again, so an error storm pages once.

Frontend incidents resolve themselves after 60 minutes with no new occurrences. If the error comes back after that, it opens a fresh incident, which goes through triage again: it pages, or only notifies if it is low. To clear everything at once, use Resolve all on System Activity: it resolves every open and acknowledged incident in the project after a confirmation, and pages no one.

Not an issue#

Some frontend errors are expected: a browser extension, a third-party script, a WebSocket that reconnects on its own. Mark them as not an issue and matching errors are still recorded and counted on the Errors & Network page, but they never open an incident, so they never page or notify anyone.

• From Errors & Network: expand a JavaScript error and choose Mark as not an issue. Pick Just this error (that exact error) or All similar errors (every error whose message contains a pattern, prefilled with the stable part of this message and editable). For similar errors you can also limit the rule to one source, such as the error's own (js, console, ...) or the SDK's websocket tag. Choose whether it covers this frontend project or all of them, add a reason, and let it expire after 7 or 30 days or never.
• From an incident: Not an issue on a frontend incident's page (or mark_not_an_issue from an AI assistant with a write key). Just this error mutes that exact error; All similar errors mutes errors containing the stable part of its message, from the same source.

The stable part of a message: URL query strings are dropped and the message is cut before the first run of 3 or more digits or 8 or more hex characters (ids, hashes, status codes), so Request failed with status code 502 also covers 503. If the cut would leave fewer than 12 characters, the whole message is used instead. Matching is not case-sensitive.

Creating a rule also closes the open and acknowledged incidents it matches, with status Not an issue, and stops their pending pages. The dialog says how many it closed.

Rules are managed in Frontend Project → Settings → Muted Errors, which lists what each one matches, how often it has matched and when it expires. Unmute deletes a rule: matching errors open incidents again from then on, and the incidents it already closed stay closed. A rule that has expired stops muting on its own.

Common confusions#

"Why did nobody get paged?" First check the incident page: Notified only means the classifier rated it low and the team got a quiet notice instead (see Classify before paging). Otherwise, most often: severity filter dropped everyone. Check each engineer's min_severity against the incident's severity. Another common cause is a repeat: if the error's incident is still open, the new occurrence is counted on it and does not page (see above). And if no incident opened at all, look for a Not an issue badge on the error in Errors & Network. Run a manual page from On-Call → Page an Engineer to confirm channels work, manual pages bypass all three filters above, so success there isolates the issue to the routing system.

"I'm in policy mode but errors page tier_1 anyway." Likely no default policy is set. If a project is in policy mode but has no default policy, incidents fall through to tier mode (the safe fallback). Mark one policy as Default from Alert Policies.

"Schedule says nobody is on call but I got paged." That's the roster-gap fallback, the system pages everyone in the audience rather than silently dropping the alert. Or your project is using the severity bypass, which pages everyone regardless of schedule above a threshold (default: critical).

"Manual page works but errors don't." Classic symptom of the severity filter. Manual pages skip the audience resolver, on-call filter, and severity filter entirely, they just dispatch the channels for one chosen engineer. If real errors don't page, the gap is upstream in one of those three filters.

When to use what#

• Tier mode + schedule off: tiny team, everyone paged on every alert. Zero config.
• Tier mode + schedule on: small team with a shift rotation. Tag your folks tier_1, set up the schedule, you're done.
• Policy mode + schedule on: multiple services / teams, custom escalation per service, observers. Worth the configuration overhead once you have more than one rotation.
• Policy mode + schedule off: rare, you have multiple services but no rotation. Usually a transitional state.