Problem
In IT Alerting schedules, on-call notifications may not reach the intended primary contact. Instead, notifications can route directly to a secondary contact or higher escalation layer, even when the on-call calendar appears correctly configured in the Scheduling Admin user interface.
When this occurs, incidents that should notify the primary on-call staff can skip the primary staff escalation layer and immediately notify a secondary or escalation layer. Delivery results can therefore show attempts only to secondary or escalation contacts instead of contacting the primary first.
This behavior can persist across incidents for the same shift until the shift configuration is corrected.
Root Cause
One cause of this behavior is an incorrect ordering of the staff escalation layers in the on-call calendar. This can occur after edits to the schedule or calendar, such as changing escalation options, where the escalation order saves incorrectly in the database while still displaying correctly in the Scheduling Admin UI.
When the saved escalation order is incorrect, the notification system uses that incorrect order when routing notifications. As a result, the system can route directly to a secondary contact instead of contacting the primary first.
This situation can be associated with recent changes to escalation settings, including switching between options such as “Use custom escalation time” and “Use notification escalation settings,” or reordering staff layers. If a staff layer was reordered or edited recently, that change can trigger an incorrect staff escalation layer order being stored.
Solution
To resolve issues where the primary on-call is skipped, review and correct the escalation configuration for the affected calendar and shift in Scheduling Admin.
First, open the affected shift in the calendar and review the shift layers and their order. Verify the following:
The primary contact is placed in the first staff escalation layer or first position in a sequenced layer.
The staff escalation layer ordering and sequencing for the shift are correct.
Sequencing is enabled where required.
Escalation times between staff and to the next layer are set appropriately.
If the staff layer order or sequencing is incorrect, adjust the layer order and save the calendar.
Next, inspect the calendar-level escalation configuration. Verify:
The calendar’s Escalation Option setting, including whether it is set to “Use custom escalation time” or “Use notification escalation settings.”
That recent changes to escalation options are understood, as a calendar change can affect which escalation settings are applied to incidents.
After adjusting the layers and escalation options, save the changes and then test the configuration by launching an incident that uses the affected calendar. Confirm that notifications now follow the intended escalation sequence and that the primary contact is reached before escalation to secondary layers.
If on-call notifications are not reaching the correct contacts, the following additional steps can help validate the configuration:
Check the order of layers in the on-call staff calendar.
Verify that layers are in the correct sequence, with the primary layer first followed by escalation layers.
Drag and reorder the layers so the primary layer is first.
Save the changes.
Test the configuration by launching an incident to confirm proper notification routing.
To further confirm the escalation ordering behavior, delivery details for affected incidents and for test incidents can be collected and reviewed. In supporting logs, entries that describe the retrieved on-call staff structure, including shift identifiers, calendar identifiers, staff layer contacts and their ordering, sequencing flags, and escalation time settings, can indicate whether the stored staff escalation order is unexpected.
Workaround
If correcting the layer ordering and escalation options in the existing shift does not resolve the issue, a practical remediation is to delete and recreate the affected shift within the calendar.
In Scheduling Admin, delete the affected shift or the affected staffing configuration and then recreate the shift. Recreating the shift restores the shift configuration and forces a correct staff escalation order to be stored, which can restore normal notification routing.
After recreating the shift, test by launching an incident that uses the updated calendar and shift. Monitor the delivery details to confirm that the primary contact is now contacted first and that escalation to secondary layers occurs only after the primary is evaluated.