Alert Channels

Alert channels define where notifications are sent when alerts trigger. Set up multiple channels for redundancy and team coverage.

Available Channels

Channel	Best For	Latency
Slack	Team collaboration, quick response	Instant
Email	Formal records, async notification	Minutes
Webhook	Custom integrations, PagerDuty	Instant
MS Teams	Microsoft ecosystem teams	Instant

Slack

Send alerts to Slack channels for real-time team visibility.

Add Slack Integration

Go to Settings > Alert Channels
Click Add Channel > Slack
Click Connect to Slack
Authorize tracer in your Slack workspace

Select Channel

Choose which Slack channel receives alerts:

Public channels (recommended for ops)
Private channels (you must invite the bot)

Configure Notifications

Channel: #ops-alerts
Notify On:
  - Alert triggered
  - Recovery
Include:
  - Monitor name
  - Error details
  - Run link

Slack Message Format

Alert messages include:

Monitor/journey name
Failure reason
Location that detected issue
Link to run details
Action buttons (acknowledge, view)

Example message:

🔴 Alert: Production API Health Check
Status: Down (3 consecutive failures)
Location: US East
Error: Connection timeout after 30s
Run: https://app.tracer/runs/abc123

Multiple Slack Channels

Send different alerts to different channels:

# Critical alerts
Channel: #critical-alerts
Rules: Production monitors, consecutive failures

# Non-critical
Channel: #monitoring
Rules: Staging monitors, all alerts

Create dedicated channels for alerts to avoid noise in general channels. Use #ops-alerts or #monitoring instead of #general.

Email

Send alerts via email for formal notification and records.

Add Email Channel

Go to Settings > Alert Channels
Click Add Channel > Email
Enter recipient email addresses

Configure Recipients

Add multiple recipients:

Recipients:
  - [email protected]
  - [email protected]
  - [email protected]

Email Format

Alert emails include:

Subject with monitor name and status
Detailed error information
Timeline of events
Direct link to dashboard

Email Delivery

Emails sent via SendGrid for reliability
SPF/DKIM signed for deliverability
From: alerts@tracer

Add alerts@tracer to your allowlist to ensure delivery.

Webhooks

Send alerts to any HTTP endpoint for custom integrations.

Add Webhook Channel

Go to Settings > Alert Channels
Click Add Channel > Webhook
Enter your webhook URL

Configure Webhook

URL: https://your-service.com/webhook/alerts
Method: POST
Headers:
  Authorization: Bearer your-secret-token
  Content-Type: application/json

Webhook Payload

{
  "event": "alert.triggered",
  "alert": {
    "id": "alert_abc123",
    "name": "Production API Alert",
    "status": "triggered",
    "triggered_at": "2024-01-15T10:30:00Z"
  },
  "monitor": {
    "id": "mon_xyz789",
    "name": "Production API Health",
    "type": "url"
  },
  "run": {
    "id": "run_def456",
    "status": "failed",
    "error": "Connection timeout",
    "location": "us-east",
    "duration_ms": 30000
  },
  "incident": {
    "id": "inc_ghi012",
    "started_at": "2024-01-15T10:25:00Z",
    "failure_count": 3
  }
}

Webhook Events

Event	Description
`alert.triggered`	Alert condition met
`alert.recovered`	Monitor back to healthy
`incident.opened`	New incident created
`incident.acknowledged`	Someone acknowledged
`incident.resolved`	Incident closed

Webhook Security

Verify webhooks are from tracer:

Headers:
  X-Webhook-Signature: sha256=abc123...
  X-Webhook-Timestamp: 1705315800

Verify signature:

const crypto = require('crypto');

function verifySignature(payload, signature, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
  return `sha256=${expected}` === signature;
}

PagerDuty Integration

Send to PagerDuty using webhook:

URL: https://events.pagerduty.com/v2/enqueue
Method: POST
Headers:
  Content-Type: application/json
Body Transform:
  routing_key: "your-pagerduty-integration-key"
  event_action: "trigger"
  payload:
    summary: "{{monitor.name}} is down"
    source: "tracer"
    severity: "critical"

Opsgenie Integration

URL: https://api.opsgenie.com/v2/alerts
Method: POST
Headers:
  Authorization: GenieKey your-api-key
Body Transform:
  message: "{{monitor.name}} is down"
  priority: "P1"
  source: "tracer"

Microsoft Teams

Send alerts to Microsoft Teams channels.

Create Incoming Webhook

In Teams, go to your channel
Click ... > Connectors
Add Incoming Webhook
Copy the webhook URL

Add Teams Channel

Go to Settings > Alert Channels
Click Add Channel > Microsoft Teams
Paste the webhook URL

Teams Message Format

Alerts appear as adaptive cards with:

Status indicator (color-coded)
Monitor name and type
Error details
Action buttons

Testing Channels

Verify your channels work before incidents:

Go to the channel settings
Click Send Test Alert
Verify you received the notification

Always test channels after setup. A misconfigured channel means missed alerts.

Channel Best Practices

Multiple channels - Use at least 2 channels for redundancy
Escalation path - Slack for first response, email for records
Team channels - Avoid DMs, use team channels
Test regularly - Verify channels work monthly
Document ownership - Who monitors which channel?

Troubleshooting

Slack Not Receiving

Check bot is in the channel
Verify integration is active
Test with Send Test Alert

Email Not Arriving

Check spam folder
Add alerts@tracer to allowlist
Verify email address is correct

Webhook Failing

Check URL is accessible
Verify authentication
Check response logs in tracer

Test with curl:

curl -X POST https://your-webhook.com/alerts \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

Alert Channels

On this page