Alarms¶

Overview¶

Alarm configuration is a global setting that allows an organization to subscribe to various Alerts and/or Notifications and forward them to supported notification channels. Notification settings can be modified in the UI under "Manage Account" -> "Alarm Settings". Each channel offers four types of filters:

• Environment type - allows filtering based on environment type. Example: PROD / QA / DEV
• Alert Event Type - allows filtering based on type of alert. Example: Alert / Notification
• Alert Event Priority - allows filtering based on priority of alert. Example: Low / Medium / High / Critical
• Event Category - allows filtering based on category. Example: InstanceEvents / UserEvents / DeploymentCellEvents, etc.
• Event Type - allows filtering based on specific event type. Example: UnhealthyInstance / UserSignUp / DeploymentCellDeleteStarted, etc.
• Payload - a json structure with properties of the event. Example: instance_id, subscription_id

Each organization can configure multiple channels with different event filters.

Interpreting instance health alerts¶

Alerts complement, but do not replace, the health state shown on an instance.

In practice:

• UnhealthyInstance means one or more required health signals failed.
• A running pod does not automatically mean the instance is healthy.
• Endpoint reachability, readiness, custom health checks, and dependent integrations can all affect alerting and health state.

When you receive an instance health alert, check:

1. The instance health details in the Operations Center
2. The workflow that last changed the instance
3. Instance debug output for rendered Helm or Terraform artifacts

For more detail on health-state semantics, see Monitoring and Debugging and Troubleshooting.

Supported notification channels¶

Email¶

Email channel sends notification as email message to email address provided in email channel configuration.

PagerDuty¶

PagerDuty channels sends notification as PagerDuty alert. Recipient is identified by PagerDuty integration key that needs to be provided during configuration.

For more details on how to generate integration key, refer to this PagerDuty documentation.

Webhook¶

Webhook calls HTTP-based callback function with configurable payload. By default, Omnistrate will include following details as request body:

{
   "eventID": "{{ $var.id }}",
   "serviceID": "{{ $var.ServiceID }}",
   "eventName": "{{ $var.Name }}",
   "eventDescription": "{{ $var.Description }}",
   "eventType": "{{ $var.Type }}", 
   "payload": "{{ $var.Payload }}"
}

Webhook channel allows method, endpoint and payload to be provided. If webhook action fails, we will retry few times before failing to notify this channel of the event.

Signing webhook deliveries¶

An endpoint that accepts any POST it receives will act on anything that reaches it. Give the channel a signing secret and every delivery arrives with an HMAC-SHA256 signature your receiver can check, so it can tell a real alarm from a request that merely knows the URL.

Set it when you create or edit a Webhook channel:

The secret must be at least 32 bytes. That is not an arbitrary minimum: deliveries are signed with HMAC-SHA256, and RFC 2104 discourages a key shorter than the hash output because it weakens the function. A short key still produces a perfectly valid signature — it is simply cheaper to forge than the algorithm's name suggests.

It is stored encrypted and never returned by any read, so keep your copy. A configured channel shows that signing is on, and an identifier for the active secret, so you can confirm a rotation took effect without the secret itself ever being displayed.

Headers on every delivery¶

Five headers accompany a signed delivery, alongside Content-Type: application/json and any static headers you configured:

A complete request looks like this:

POST /omnistrate/alarms HTTP/1.1
Host: hooks.example.com
Content-Type: application/json
X-Omnistrate-Event-Id: evt-3c3d521bd975
X-Omnistrate-Event-Type: FailedBackup
X-Omnistrate-Timestamp: 2026-08-25T07:58:45Z
X-Omnistrate-Delivery-Attempt: 1
X-Omnistrate-Signature-256: sha256=24be8a3ec41a191588bb4a28cf5cef833f8fcef9050eada0fbd2d2b140b292b8

{"eventID":"evt-3c3d521bd975","serviceID":"s-Xy2ZaBcDeF","eventName":"FailedBackup", ...}

Your own configured headers are applied first, so a header you set cannot displace the signature.

Verifying the signature¶

const timestamp = req.headers['x-omnistrate-timestamp'];
const signature = req.headers['x-omnistrate-signature-256'];
if (typeof timestamp !== 'string' || typeof signature !== 'string') {
  return res.status(401).end();
}

// rawBody is the unparsed request body. In Express, use express.raw() or capture it in a verify hook.
const signed = `${timestamp}.${rawBody}`;
const expected = 'sha256=' + crypto
  .createHmac('sha256', process.env.OMNISTRATE_WEBHOOK_SECRET)
  .update(signed, 'utf8')
  .digest('hex');

// Constant time, so a mismatch cannot be found one character at a time.
const expectedBuf = Buffer.from(expected, 'utf8');
const signatureBuf = Buffer.from(signature, 'utf8');
if (expectedBuf.length !== signatureBuf.length || !crypto.timingSafeEqual(expectedBuf, signatureBuf)) {
  return res.status(401).end();
}

// Then reject anything too old to be a live alarm. Five minutes is a reasonable window.
if (Date.now() - Date.parse(timestamp) > 5 * 60 * 1000) {
  return res.status(401).end();
}

What your receiver should do¶

• Answer quickly with any 2xx, and do the work asynchronously. A receiver that blocks holds the delivery open.
• Deduplicate on X-Omnistrate-Event-Id. It is stable across retries and across an operator re-sending an event, which is deliberate: re-sending exercises your idempotency rather than bypassing it.
• Use HTTPS. Redirects are not followed — following one on a signed POST would either strip the body and signature or replay them at a host you never registered.

The same signing scheme is used for marketplace fulfillment webhooks. See Marketplace fulfillment API if you integrate with those as well.

Channels configuration¶

Omnistrate generates a variety of Notifications and Alerts whenever a corresponding event occurs. Each notification channel can be configured to receive only specific types (based on event type, environment, priority, etc.).

To add new channel, open "Manage account" -> "Notifications" page on UI where you will be able to add new channel. There are 2 configuration options available when adding a channel:

1. Basic - subscription is created based on environment type, alert event type and event priority. This option offers less control, but makes it easier to create a channel based on fewer inputs. All of basic dimensions (environment, type and priority) have limited set of options that are unlikely to change as we add new types of alerts. An example of a basic notification rule: "High and critical priority Alerts from prod environments".

2. Advanced - subscription is created based on environment type and specific category and/or event type. This option offers more control, but is based on event categories.

Alarm Event Categories and Type¶

The following table lists all available alarm event categories and the specific event types within each. For the payload delivered with each event, see Webhook Alarm Event Payloads.