Self-diagnostics
The Morse SDK absorbs its own internal errors so they can never break your application. When it does, it reports that error to us — so we find out our SDK is broken inside your process without you having to file a ticket.
This page is the complete list of what that report contains. It is checked against the wire schema by an automated test: if we add a field to the payload without adding it here, our build fails.
Everything we receive, you also see. Every diagnostic is written to your own morse_ai logger
before it is transmitted — including when you have transmission switched off entirely. There is no
category of self-reported information we get that your engineers cannot.
On by default
Self-diagnostics is on by default, at level error — the SDK transmits an event when it
absorbs one of its own internal errors. You do not switch it on, and upgrading the SDK does not
ask: if you are on a version that has this feature, it is already active. To switch it off, see
Turning it off.
The level is off, error, or debug. It starts at error and we can lower or raise it from our
side via a response header on traffic the SDK is already sending — used to mute a noisy release, or
to ask for more detail while chasing a bug. debug relaxes how many reports may be sent; it never
widens what may be sent, and it cannot override your opt-out.
When it sends
Only on failure. There is no heartbeat, no periodic health check, no startup ping. A process that runs without an internal SDK error transmits zero diagnostic events, and never even starts the background sender.
Two kinds of report exist:
- Errors — one absorbed exception from inside the SDK.
- Delivery counters — aggregate counts of telemetry the SDK had to discard (a full queue, an exhausted retry, a quota rejection). These are counts only, and they are how you and we both learn that telemetry is being silently lost.
What is never sent
This list is the point of the page, so it comes before the field table:
- Your prompts and completions. No LLM input or output, ever, at any verbosity level.
- Your span, trace, or log content. A diagnostic is about the SDK, not about your data.
- Local variables. Stack frames are captured with
traceback.extract_tb(Python) and by parsingError.stack(TypeScript) — neither reads frame locals, so there is nothing to strip. - Your application’s stack frames. Frames outside the Morse package are replaced by a count. We learn “three of your frames were here”; we never learn which files or functions.
- Your environment variables, config values, API keys, or credentials of any kind.
Exception messages get a second, unconditional pass for credential assignments
(
password=,client_secret:,authorization: Bearer …,PGPASSWORD=) — even if your organization has switched that pattern off for its own telemetry. We would rather lose a word of our own error text than receive one of your secrets. - Your file paths. SDK frames are reported relative to the Morse package root, so no absolute path from your machine or container is transmitted.
- Your users’ data, IDs, or identifiers of any kind.
Every field we transmit
| Field | What it is |
|---|---|
kind | error or counters |
fingerprint | Stable identifier for the failing code path, e.g. morse_ai.tracing._send_trace |
error_type | Exception class name, e.g. KeyError |
message | Exception message, passed through the same PII redaction as your telemetry, then through the secret-assignment scrub a second time unconditionally, and byte-capped |
stack | Morse-scoped frames: our file/line/function, plus a count of your elided frames |
counters | Delivery-loss counts, {reason, category, quantity} — see below |
sdk_language | python or typescript |
sdk_version | The SDK version installed |
runtime_version | Python or Node version |
platform | OS and architecture, e.g. Linux-x86_64 |
extra | An allow-listed set only: adapter, module, function, operation, endpoint_host (hostname only, never a path or query) |
Your organization is identified by the API key the request is authenticated with. It is never read from the payload — the schema has no field for it.
Delivery-loss reasons
Each is a closed value, paired with a category (trace, span, log, record, diagnostic)
and a count. No free text is ever sent.
| Reason | What it means |
|---|---|
queue_full | The in-memory queue was full and the oldest events were evicted. |
encode_failed | The batch could not be serialized to JSON. A payload problem, not a network one. |
auth_rejected | The server returned 401 or 403 — the API key is revoked, expired, or lacks the scope. |
endpoint_rejected | Another non-429 4xx (400, 404, 413, 422) — usually a misconfigured endpoint URL. |
quota_rejected | The server returned 429 and the batch was still being rate-limited after the SDK had waited out the server’s Retry-After and retried. Seeing this means you were over your ingest limit for long enough that redelivery could not catch up, not that a single 429 was discarded. |
send_failed | A send failed and no retry was possible — the flush budget had already elapsed. |
retry_exhausted | A retryable failure (5xx or a network error) that used up all three attempts. A rate limit that outlasts our retries is reported as quota_rejected, not here. |
shutdown_drop | The process was shutting down and the batch could not be flushed in time. |
auth_rejected, endpoint_rejected and send_failed were added in 2026-08. Before that, the
Python SDK reported every non-429 4xx as encode_failed and the TypeScript SDK reported every
drop as retry_exhausted, so these failures were indistinguishable in the counters. The
TypeScript SDK gained a retry loop shortly after, which is what finally made its
retry_exhausted truthful.
Since 2026-08, these counters also cover the trace pipeline’s OTLP exporter, not only the
legacy JSON transport. A batch the exporter gives up on — the common case being a burst that
trips your plan’s ingest rate limit — previously produced a single line on your own stdout and
nothing else: no exception, no counter, no signal here. It is now counted under span, and
logged locally at WARNING with the count, whichever way you have diagnostics configured.
When Morse can’t be reached
The SDK holds telemetry in memory only. If the Morse API cannot be reached (no network, DNS
failure, a firewall, an outage), the SDK retries each batch for a few seconds and then drops it,
counted as retry_exhausted. Spans are not buffered across outages: nothing is written to disk,
and nothing is resent when the network comes back. Runs that happen during an outage are missing
from Morse for good.
If you need every run through an outage, keep your own record of the outcome (your database or
logs) and send it again later with record().
Messages you may see
These are the lines the TypeScript SDK prints for the common setup failures, from SDK 0.7.1.
Version 0.7.0 prints the older dropped 1 span (auth_rejected) style line plus a raw
morse internal error line for the same causes.
| Message | What it means and what to do |
|---|---|
Morse API key looks malformed (...). Check MORSE_API_KEY. | Printed at start-up. The key does not look like a Morse key: cut short, wrapped in quotes, or with spaces. Copy the key again. In Command Prompt, write set MORSE_API_KEY=... without quotes. |
Morse rejected the API key (401): N span(s) dropped. Check MORSE_API_KEY or get a new key in Settings → API keys. | The server refused the key (wrong, revoked, or expired; 403 for a read-only personal key). Those spans are gone. Fix the key; rotate it in Settings → API keys if you no longer have it. |
Morse unreachable (api.morsehq.dev), retrying… | A send failed at the network level and the SDK is retrying. Printed once per outage. Morse not responding is the same thing on the tracing layer. |
Morse unreachable (api.morsehq.dev): N span(s) dropped after retries. Spans are not buffered across outages. | The retries ran out. Those spans are gone (see above). Check your network, proxy or firewall, and MORSE_ENDPOINT if you set one. |
The raw error behind the last three (for example OTLPExporterError: Unauthorized or
getaddrinfo ENOTFOUND) is printed only when MORSE_SDK_DEBUG=1 is set.
Retention
Diagnostic events are deleted after 30 days, on a fixed schedule that does not depend on your plan.
Turning it off
Set MORSE_SDK_DISABLE_TELEMETRY=1 in your environment. Nothing is transmitted.
Local logging continues. Opting out stops data leaving your process; it does not stop your own
engineers from seeing what the SDK is doing. Delivery-loss counters in particular keep reaching
your logger at WARNING, because silent telemetry loss is exactly the thing you want your on-call
to know about.
Turning this off means we cannot see SDK failures happening in your environment. If the SDK breaks for you, we will only find out when you tell us.
Rate limits and volume
Diagnostics are bounded on every axis so they can never become a load source:
- At most 100 reports per failing code path per minute leave the SDK.
- Local log lines are capped separately, so a tight failure loop cannot flood your logs.
- Pending diagnostics are held in a small bounded buffer; if it fills, the oldest are dropped.
- If our endpoint is unreachable, a circuit breaker stops attempts for five minutes rather than retrying. Diagnostics are never retried — a retry loop would amplify the exact failure being reported.