Home/Docs/Monitoring

Monitoring contract

Contract version 2.1.0 · payload schema_version 2.0

Two kinds of heartbeat. This page covers monitoring of a Chrome extension registration: your agent sends its report to POST /api/v1/monitoring/report. Web project environments in the panel use a separate POST /api/v1/heartbeat/report with a different token. The minimal environment report is {"overall":"ok"}; a custom metrics v2 payload is accepted too. Do not swap one endpoint for the other when you set up an agent. Section 8b describes the environment heartbeat.

Monitoring puts your extension under constant watch: a live traffic light on its card in the list, a base monitoring tab in Detail and a separate Telegram status message with edge alerts. It combines two independently enabled components: a push heartbeat (an agent on your service's host reports on a declared schedule; silence is itself a signal) and uptime pings (the panel checks your public URLs for availability).

1. How it works (two components, one light)

• Push heartbeat: a cron script on your host (e.g. an adapted monitor.sh) POSTs a v2 metrics report to the panel at the interval you declared. Every report must carry the reserved overall verdict. Missed reports escalate the light (dead-man's switch): if the host, the network or cron dies, you see it precisely because the calls stop.
• Uptime pings: the panel itself periodically requests the public URLs you configure (site, api, docs) and checks only availability, never the content. No agent needed, a URL is enough.

An extension is "on monitoring" when monitoring is enabled and at least one component is configured. Otherwise its card shows a neutral gray dot.

1b. Heartbeats and the slot pool

Monitoring lives at the extension level as two independent flat lists that share one slot pool: heartbeats (push producers, formerly "stands") and uptime pings (pull checks). An extension can have several heartbeats – for example a main host and a paid worker. Each heartbeat pushes reports with its own bearer token and declares its own presentation (tabs, sections and Telegram lines) via its label. Uptime pings are a separate flat list owned by the extension, not nested under any heartbeat. The panel merges everything into one aggregate light (worst-of), one Detail surface and one summary Telegram message.

Heartbeats and pings share a single per-extension slot pool: one heartbeat costs one slot, one active ping costs one slot, and their sum must not exceed the plan's max_slots (Trial 3 · Studio 10 · Fleet 25). Exceeding the pool returns 409 slot_limit; after a downgrade the excess is suspended (pings first, heartbeats last). The response carries slots_used and max_slots so the UI can show the budget.

2. The traffic light

The aggregate is the worst-of roll-up across every enabled heartbeat's component and every active ping's light, where severities order as ok < warn < crit. A heartbeat's component is max(reported, watchdog): reported is the worst of the overall metric and the breached thresholds (6.2) from the last accepted report; watchdog is a freshness floor computed from the panel's own receive clock (agent clock skew is harmless).

Severity is canonical ok | warn | crit. Friendly aliases are normalized on intake: warning becomes warn; error and critical become crit.

3. Watchdog (missed heartbeats)

You declare the expected report interval (60s to 24h) when enabling the heartbeat. The watchdog then floors the light by freshness, with grace = max(60s, 25% of the interval) to absorb cron and network jitter:

A rejected report (401 / 413 / 422 / 429) never refreshes the watchdog, so a broken agent honestly escalates; the last rejection reason is visible in the monitoring settings. Recovery is instant: the first accepted report recomputes the light immediately.

4. Uptime pings

A ping is a named availability check of one public https URL. The panel performs GET with a 10s timeout, follows at most 5 redirects, reads no body beyond a minimal limit and treats a final 2xx-class status as success (204 and 206 count too). A final 3xx (redirect loop or limit), 4xx, 5xx, timeout or DNS/TLS/network error is a failure. A failed check is confirmed by one retry after ~15s before it counts, which absorbs single network blips.

Ping results appear as first-party metrics ping:<name> in the pings section (Uptime) of the base monitoring tab and as the reserved Telegram line pings: 🟢 api 210ms · 🟢 site 90ms · 🔴 docs (timeout). The number of pings per extension and the minimum interval depend on the plan:

Monitoring availability and the number of slots depend on your plan; the minimum heartbeat interval is 60s.

5. Putting an extension on monitoring

In the extension's Settings, monitoring block:

• Turn monitoring on, if your plan includes it.
• Heartbeat: enable it and declare the report interval; the panel issues a bearer monitoring token, shown exactly once (rotate it any time; the old token immediately gets 401). Put the token and the endpoint URL into your agent's environment.
• Pings: add named public URLs (e.g. site, api, docs) with a per-ping interval, up to the plan limit.
• Telegram: enable monitoring delivery and optionally set a separate group/chat.

6. The report format

The report is the same metadata-driven v2 payload as the custom-metrics contract (tabs, sections, Telegram lines, metrics), pushed to POST /api/v1/monitoring/report with Authorization: Bearer <monitoring token>. Monitoring adds reserved ids:

Limits: payload at most 64 KB, at most 200 metrics, at most 2 reports per minute per source (excess gets 429 and does not count as a heartbeat).

A body over 64 KB gets 413 payload_too_large and is never cut and read as a shorter report. A report that breaks the rules above gets 422 invalid_report with every problem listed. Both reasons are kept as the heartbeat's last error:

HTTP/1.1 413 Payload Too Large
{ "code": "payload_too_large", "message": "report is over 64 KB; send fewer metrics or split them across heartbeats",
  "limit_bytes": 65536, "size_bytes": 71680 }

HTTP/1.1 422 Unprocessable Entity
{ "code": "invalid_report", "message": "report rejected",
  "errors": [ "metric \"overall\" must have kind \"status\" (got \"text\")",
              "metric id \"ping:x\" uses the reserved \"ping:\" namespace" ] }

6.1. Example report

{
  "schema_version": "2.0",
  "generated_at": "2026-07-12T10:19:58Z",
  "tabs":     [ { "id": "stand", "title": "Stand", "order": 5, "icon": "server" } ],
  "sections": [ { "tab": "stand",      "id": "docker", "title": "Docker", "order": 0 },
                { "tab": "monitoring", "id": "host",   "title": "Host",   "order": 10 } ],
  "telegram": { "header": "{sev} {name} monitoring",
                "lines": [ { "id": "host",   "order": 0, "prefix": "🖥 " },
                           { "id": "docker", "order": 1, "prefix": "📦 " } ] },
  "metrics": [
    { "id": "overall", "kind": "status", "state": "ok", "severity": "ok" },

    { "id": "issues", "kind": "list",
      "items": [ { "text": "RAM 91% (7.1/7.8G)", "severity": "warn" } ],
      "dashboard": { "show": true, "tab": "monitoring", "section": "host", "order": 0,
                     "render": "list", "name": "Issues" },
      "telegram":  { "show": false } },

    { "id": "ram", "kind": "percent", "label": "RAM", "value": 91, "severity": "warn",
      "dashboard": { "show": true, "tab": "monitoring", "section": "host", "order": 1,
                     "render": "kpi" },
      "telegram":  { "show": true, "line": "host", "pos": 10,
                     "format": "RAM {value}%", "sep": " · " } }
  ]
}

Successful response:

{ "accepted": true, "effective": "green", "next_expected_by": "2026-07-12T10:26:13Z" }

6.2. Thresholds on report metrics

Any numeric metric of a report, except overall, may carry thresholds (the block of the custom-metrics contract, section 5.4). The panel computes the severity of such a metric at intake and the heartbeat's light is the worst of overall and the breached thresholds: a metric over crit makes the heartbeat red even when overall is ok. recover works against the previous accepted report of the same heartbeat. A broken block gets 422 invalid_report with the reason in the list. A report without thresholds behaves as before.

{ "id": "queue", "kind": "number", "label": "Mail queue", "value": 41,
  "thresholds": { "direction": "above", "warn": 20, "crit": 100, "unit": "msgs", "recover": 15 },
  "dashboard": { "show": true, "tab": "monitoring", "section": "host", "order": 2, "render": "kpi" },
  "telegram": { "show": true, "line": "host", "pos": 20, "format": "queue {value}" } }

The panel draws it as 41 (threshold 20/100 msgs) on a yellow tile, and the heartbeat is yellow.

7. Integrating a monitor script (monitor.sh style)

If you already run a cron health-check script, add one best-effort POST at the end of its tick. A delivery failure must not break the tick: the watchdog will catch prolonged silence anyway.

# monitor.env
EXTOPS_REPORT_URL="https://extops.dev/api/v1/monitoring/report"
EXTOPS_REPORT_TOKEN="<monitoring token from Settings>"

# end of the tick: build the payload with jq from the values you already computed,
# then push it best-effort (never let delivery break the tick itself)
payload="$(jq -n --arg sev "$SEVERITY" '{
  schema_version: "2.0",
  generated_at: (now | todate),
  metrics: [ { id: "overall", kind: "status", state: $sev, severity: $sev } ]
}')"
code="$(curl -sS -m 10 -o /tmp/extops-report.json -w '%{http_code}' -X POST "$EXTOPS_REPORT_URL" \
  -H "Authorization: Bearer $EXTOPS_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$payload" || echo 000)"
# 413 payload_too_large: over 64 KB; 422 invalid_report: .errors lists every problem
case "$code" in 413|422) logger -t monitor "panel: $code $(jq -c . /tmp/extops-report.json)" ;; esac

Map your script's severities to the canonical set (critical is accepted and normalized to crit) and feed your existing issue list into the issues metric. Set the declared interval equal to the cron period.

8. Telegram: separate status message and alerts

• Monitoring data goes to Telegram as a separate editable status message, distinct from the custom-metrics status: header + the reserved pings line + the lines your report declares.
• Optionally route monitoring to a separate group/chat; when not set, the extension's channel chat is used. The bot transport is shared with the extension's channel.
• Edge alerts fire on every change of the aggregate color, in both directions, including recovery back to green (with incident duration). While the color is stable there is silence; config changes (pausing, editing pings) do not alert.

8b. Web project heartbeats

A web project has heartbeats of its own. Each one belongs to an environment (for example Production) and has its own hb_ token. The report rules are those of section 6; below is what differs.

POST /api/v1/heartbeat/report
Authorization: Bearer hb_<64 hex>
Content-Type: application/json

{"overall":"ok"}

• Body: a v2 report with the required overall status metric, or the thin {"overall":"ok"} with ok, warn or crit. The panel turns the thin body into a v2 report and stores the normalized body.
• Limits: at most 64 KB per report, at most 4 reports per minute per token and at most 200 metrics. A report over 200 metrics gets 422, as for an extension.
• Answers: 200 {"accepted":true,"light":"green"}, 401 for an unknown token, 413 payload_too_large, 422 invalid_report, 429 with Retry-After: 60. The 413 and 422 reasons show as Last error in Setup, Monitoring.
• Covers: a heartbeat covers zero or more surfaces of its environment. Its light colours each covered surface; a heartbeat that covers none colours the environment. Covers is set in the panel, not in the report.
• Light: green up to interval + grace after the last accepted report, yellow up to 2 × interval + grace, red after that, with grace = max(60s, 25% of the interval). The interval is 60s to 24h.
• Waiting for first report: a new heartbeat stays gray until its first accepted report. After 24 hours without one it turns yellow and sends one notification; it is never red before its first report.
• Thresholds: as in section 6.2. The light in the 200 answer and in the panel is the worst of freshness, overall and the breached thresholds, and through Covers it reaches the covered surfaces.
• Tabs: metrics with tab:"monitoring" appear on the first tab of the project's Monitoring section, and base is reserved too. Tabs you declare get tabs of their own.

8c. Who connects to your services and how

A web project can add port checks to a Service whose address is a host, for example a mail server. These checks are made by Ext Ops Panel itself. This section says who connects, what is sent, how often, and how a host can opt out. The agent contract above does not change.

• Where from: one address, 89.117.144.124, with the name probe.extops.dev. The name resolves to this address, and the address resolves back to the name. There is no other vantage point.
• Only verified hosts: a check runs only on a host under a domain that the owner of the project has verified with a DNS TXT record or a well-known file. When the proof goes away, the checks pause.
• Ports: TCP 25, 110, 143, 465, 587, 993, 995; TLS 443, 465, 993, 995; SMTP 25 and 587; SMTPS 465; IMAP 143; IMAPS 993. No other port is ever checked.
• TCP and TLS: connect and read the greeting the server sends, or complete a TLS handshake and read the certificate. Nothing is sent.
• SMTP: wait for the 220 greeting, send EHLO probe.extops.dev, send STARTTLS when the server offers it and complete the handshake, then send QUIT. SMTPS on port 465 does the same inside TLS, without STARTTLS.
• IMAP: wait for * OK, send a1 CAPABILITY, send STARTTLS on port 143 when it is offered, send ID ("name" "ExtOpsPanel-Probe" "support-url" "https://extops.dev/docs/monitoring#probes") only when the server announces ID, then send LOGOUT. IMAPS on port 993 does the same inside TLS.
• Never sent: AUTH, LOGIN, AUTHENTICATE, MAIL FROM, RCPT TO, DATA, VRFY, EXPN, SELECT. The probe never logs in, never sends mail and never opens a mailbox. Every line passes a fixed list of allowed commands before it is written, and a test fails if any other command reaches a server.
• How often: a mail check runs at most every 5 minutes on any plan, with a spread of 10% either way. One host gets at most one connection at a time and at most 40 connections per hour, for all projects together. Past that, the host is left alone until the next hour.
• When a server says no: an SMTP 421 stops every check of that host for one hour. An SMTP 554 or an IMAP * BYE is recorded as a refusal and tried again only at the next interval.
• Limits of one check: 5 seconds to connect, 10 seconds for the whole dialog, at most 4 KB read.
• What is kept: the result, the reply code, the latency, the first line of the greeting cut to 120 characters with every e-mail address removed, a short list of capabilities such as STARTTLS and IMAP4rev1, and the dates of the certificate. Not the full greeting, not the answers to EHLO or ID, and no addresses.
• Opting out: if you run a host and do not want these checks, write to support@extops.dev with the host name. We exclude the host or its whole domain, and nothing connects to it after that. You can also answer 421 or block 89.117.144.124. The owner of the project can pause or delete a check at any time in Setup, Monitoring.

9. The machine-readable contract

The formal description of the report endpoint, the payload schema, the reserved ids and the traffic-light semantics is the OpenAPI 3.1 file. It carries the same contract version (2.1.0).