Home/Docs/Monitoring

Monitoring contract

Contract version 2.0.0 · payload schema_version 2.0 · Trial/Paid plans

Monitoring puts your extension under constant watch: a live traffic light on its card in the list, a base monitoring tab in Detail and a separate Telegram status message with edge alerts. It combines two independently enabled components: a push heartbeat (an agent on your service's host reports on a declared schedule; silence is itself a signal) and uptime pings (the panel checks your public URLs for availability).

1. How it works (two components, one light)

• Push heartbeat: a cron script on your host (e.g. an adapted monitor.sh) POSTs a v2 metrics report to the panel at the interval you declared. Every report must carry the reserved overall verdict. Missed reports escalate the light (dead-man's switch): if the host, the network or cron dies, you see it precisely because the calls stop.
• Uptime pings: the panel itself periodically requests the public URLs you configure (site, api, docs) and checks only availability, never the content. No agent needed, a URL is enough.

An extension is "on monitoring" when monitoring is enabled and at least one component is configured. Otherwise its card shows a neutral gray dot.

1b. Heartbeats and the slot pool

Monitoring lives at the extension level as two independent flat lists that share one slot pool: heartbeats (push producers, formerly "stands") and uptime pings (pull checks). An extension can have several heartbeats – for example a main host and a paid worker. Each heartbeat pushes reports with its own bearer token and declares its own presentation (tabs, sections and Telegram lines) via its label. Uptime pings are a separate flat list owned by the extension, not nested under any heartbeat. The panel merges everything into one aggregate light (worst-of), one Detail surface and one summary Telegram message.

Heartbeats and pings share a single per-extension slot pool: one heartbeat costs one slot, one active ping costs one slot, and their sum must not exceed the plan's max_slots (Trial 3 · Studio 10 · Fleet 25). Exceeding the pool returns 409 slot_limit; after a downgrade the excess is suspended (pings first, heartbeats last). The response carries slots_used and max_slots so the UI can show the budget.

2. The traffic light

The aggregate is the worst-of roll-up across every enabled heartbeat's component and every active ping's light, where severities order as ok < warn < crit. A heartbeat's component is max(reported, watchdog): reported is the severity of the overall metric from the last accepted report; watchdog is a freshness floor computed from the panel's own receive clock (agent clock skew is harmless).

Severity is canonical ok | warn | crit. Friendly aliases are normalized on intake: warning becomes warn; error and critical become crit.

3. Watchdog (missed heartbeats)

You declare the expected report interval (60s to 24h) when enabling the heartbeat. The watchdog then floors the light by freshness, with grace = max(60s, 25% of the interval) to absorb cron and network jitter:

A rejected report (401 / 422 / 429) never refreshes the watchdog, so a broken agent honestly escalates; the last rejection reason is visible in the monitoring settings. Recovery is instant: the first accepted report recomputes the light immediately.

4. Uptime pings

A ping is a named availability check of one public https URL. The panel performs GET with a 10s timeout, follows at most 5 redirects, reads no body beyond a minimal limit and treats a final 2xx-class status as success (204 and 206 count too). A final 3xx (redirect loop or limit), 4xx, 5xx, timeout or DNS/TLS/network error is a failure. A failed check is confirmed by one retry after ~15s before it counts, which absorbs single network blips.

Ping results appear as first-party metrics ping:<name> in the pings section (Uptime) of the base monitoring tab and as the reserved Telegram line pings: 🟢 api 210ms · 🟢 site 90ms · 🔴 docs (timeout). The number of pings per extension and the minimum interval depend on the plan:

Monitoring as a whole is Trial/Paid only; the minimum heartbeat interval is 60s on all plans.

5. Putting an extension on monitoring

In the extension's Settings, monitoring block:

• Turn monitoring on (Trial/Paid plans).
• Heartbeat: enable it and declare the report interval; the panel issues a bearer monitoring token, shown exactly once (rotate it any time; the old token immediately gets 401). Put the token and the endpoint URL into your agent's environment.
• Pings: add named public URLs (e.g. site, api, docs) with a per-ping interval, up to the plan limit.
• Telegram: enable monitoring delivery and optionally set a separate group/chat.

6. The report format

The report is the same metadata-driven v2 payload as the custom-metrics contract (tabs, sections, Telegram lines, metrics), pushed to POST /api/v1/monitoring/report with Authorization: Bearer <monitoring token>. Monitoring adds reserved ids:

Limits: payload at most 64 KB, at most 200 metrics, at most 2 reports per minute per source (excess gets 429 and does not count as a heartbeat).

6.1. Example report

{
  "schema_version": "2.0",
  "generated_at": "2026-07-12T10:19:58Z",
  "tabs":     [ { "id": "stand", "title": "Stand", "order": 5, "icon": "server" } ],
  "sections": [ { "tab": "stand",      "id": "docker", "title": "Docker", "order": 0 },
                { "tab": "monitoring", "id": "host",   "title": "Host",   "order": 10 } ],
  "telegram": { "header": "{sev} {name} monitoring",
                "lines": [ { "id": "host",   "order": 0, "prefix": "🖥 " },
                           { "id": "docker", "order": 1, "prefix": "📦 " } ] },
  "metrics": [
    { "id": "overall", "kind": "status", "state": "ok", "severity": "ok" },

    { "id": "issues", "kind": "list",
      "value": { "items": [ { "text": "RAM 91% (7.1/7.8G)", "severity": "warn" } ] },
      "dashboard": { "show": true, "tab": "monitoring", "section": "host", "order": 0,
                     "render": "list", "name": "Issues" },
      "telegram":  { "show": false } },

    { "id": "ram", "kind": "percent", "label": "RAM", "value": 91, "severity": "warn",
      "dashboard": { "show": true, "tab": "monitoring", "section": "host", "order": 1,
                     "render": "kpi" },
      "telegram":  { "show": true, "line": "host", "pos": 10,
                     "format": "RAM {value}%", "sep": " · " } }
  ]
}

Successful response:

{ "accepted": true, "effective": "green", "next_expected_by": "2026-07-12T10:26:13Z" }

7. Integrating a monitor script (monitor.sh style)

If you already run a cron health-check script, add one best-effort POST at the end of its tick. A delivery failure must not break the tick: the watchdog will catch prolonged silence anyway.

# monitor.env
EXTOPS_REPORT_URL="https://extops.dev/api/v1/monitoring/report"
EXTOPS_REPORT_TOKEN="<monitoring token from Settings>"

# end of the tick: build the payload with jq from the values you already computed,
# then push it best-effort (never let delivery break the tick itself)
payload="$(jq -n --arg sev "$SEVERITY" '{
  schema_version: "2.0",
  generated_at: (now | todate),
  metrics: [ { id: "overall", kind: "status", state: $sev, severity: $sev } ]
}')"
curl -sS -m 10 -X POST "$EXTOPS_REPORT_URL" \
  -H "Authorization: Bearer $EXTOPS_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$payload" || true

Map your script's severities to the canonical set (critical is accepted and normalized to crit) and feed your existing issue list into the issues metric. Set the declared interval equal to the cron period.

8. Telegram: separate status message and alerts

• Monitoring data goes to Telegram as a separate editable status message, distinct from the custom-metrics status: header + the reserved pings line + the lines your report declares.
• Optionally route monitoring to a separate group/chat; when not set, the extension's channel chat is used. The bot transport is shared with the extension's channel.
• Edge alerts fire on every change of the aggregate color, in both directions, including recovery back to green (with incident duration). While the color is stable there is silence; config changes (pausing, editing pings) do not alert.

9. The machine-readable contract

The formal description of the report endpoint, the payload schema, the reserved ids and the traffic-light semantics is the OpenAPI 3.1 file. It carries the same contract version (2.0.0).