Set up webhook notifications

Send instance events to an HTTPS endpoint you control. CosmicAC posts a payload when a job fails, degrades, or recovers, and when a served model's health changes. One webhook covers the whole instance.

Job lifecycle events also depend on the settings of the job that raises them. See What controls delivery.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
An HTTPS endpoint that accepts POST requests. For Slack, create an incoming webhook, then copy its URL.

Steps

Open the Notifications page

In the left navigation, click Settings. On the Settings page, under Instance, click Notifications.

Save the webhook URL

In Webhook URL, enter the HTTPS URL that receives the events, then click Save. CosmicAC sends nothing until you save a URL.

The first save also generates the signing secret and shows it once. Copy it now if you plan to verify signatures.

Choose the payload format

Under Format, select one of the following.

Generic JSON: a flat event object carrying id, type, schema, timestamp, instance, and the fields for that event type.
Slack-compatible: a Slack incoming webhook body carrying text and a color-coded attachments block. Red marks a failure, green a recovery, and orange a degraded state.

Select Slack-compatible when the URL points at Slack. CosmicAC posts the body straight to the incoming webhook URL, so it needs no Slack workspace credentials, bot token, or OAuth 2.0 authorization.

Select the events to send

Under Events, turn on each event you want delivered. Delivery is opt-in, so CosmicAC never sends an event you leave off.

The four job.* events also need the matching preference on the job that raises them. See What controls delivery.

Event	Fires when
`job.failed`	A job transitions to Failed. Includes the failure reason.
`job.degraded`	Healthy replicas drop below desired. The endpoint stays live.
`job.recovered`	A job returns to Active from Degraded or Failed.
`job.restart_storm`	Any replica restarts three times within 10 minutes.
`model.health.down`	A served model's health enters Down.
`model.health.recovered`	A served model's health leaves Down.

Click Save to apply the selection.

Send a test event

Click Send test event. CosmicAC posts a test.ping payload to the saved URL. Repeated tests in quick succession are rate limited.

Confirm the delivery

Check Recent Deliveries. Each row lists the event type, the target URL, the response status, and how long ago CosmicAC sent it. A successful delivery shows the response code, such as 200 OK.

A delivery counts as failed when the receiver answers with a status outside the 200 range or doesn't answer in time.

Verify the signature on your receiver

This step applies when your receiver is a service you run. Skip it when the webhook URL points at Slack, because a Slack incoming webhook can't run verification code and ignores the signature header.

Your receiver needs the signing secret to confirm that a payload came from CosmicAC. CosmicAC shows the secret once, when you first save the webhook URL. If you didn't copy it then, click Regenerate to issue a new one.

Every request carries these headers.

X-Cosmic-Event: the event type, such as job.failed.
X-Cosmic-Delivery: the unique event identifier. Use it to discard duplicates.
X-Cosmic-Timestamp: the send time in milliseconds.
X-Cosmic-Signature: sha256= followed by the HMAC-SHA256 digest of the raw request body, keyed with the signing secret.

Recompute the digest over the raw body and compare it before you trust a payload.

const crypto = require('crypto')

const expected = 'sha256=' + crypto
  .createHmac('sha256', process.env.COSMICAC_WEBHOOK_SECRET)
  .update(rawBody)
  .digest('hex')

const signature = req.headers['x-cosmic-signature']
const valid = expected.length === signature.length &&
  crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(signature))

What controls delivery

Every job has four notification preferences, one for each lifecycle event. CosmicAC turns on all four when you create the job. To set them, see Create a GPU Container Job, Create a vLLM Managed Inference Job, and Create a Parakeet Managed Inference Job.

An event reaches your webhook only if both of these are on:

The job's notification preference, which controls whether CosmicAC creates the event.
The event under Events on the Settings → Notifications page, which controls whether CosmicAC posts it.

If either one is off, the event doesn't reach your webhook. Turning it on later doesn't deliver the events that CosmicAC already skipped.

CosmicAC delivers each event it creates to two destinations.

Destination	Requirements
The notification bell in the web interface	None. CosmicAC shows the event to the job's owner.
Your webhook	A saved webhook URL, and the event turned on under Events on the Settings → Notifications page.

The Events selection applies only to the webhook. If you turn off an event under Events, it still appears in the notification bell.

The CLI and the web interface spell the same event differently. Pass the underscore form, such as job_failed, to cosmicac jobs create --notify. The web interface shows job.failed.

The model.health.down and model.health.recovered events have no per-job preference, because no job raises them. The Events selection is the only setting that controls them.