# Error tracking How Cohorly turns `$exception` events into error issues - fingerprints, the error worker, sampling, the monthly exception quota, triage statuses, and where to set up each SDK and upload symbols. Errors in Cohorly are ordinary events named `$exception`. An exception that reaches Cohorly can be broken down, filtered, put in a funnel or a cohort, and shown on a user's timeline like any other event, with the same [Query IR](/api/queries) and no second query language. On top of that, Cohorly groups them into **error issues** so you can triage "what is breaking" instead of reading one event at a time. ## When to use it Use error tracking when you need to: - see which errors and crashes are hitting your users, grouped, ranked and tied to a release; - ask product questions about errors, such as "users who hit this error and did not check out"; - triage: mark an error resolved in the release that fixes it, ignore noise, and get it reopened automatically when it comes back. ## Core terms | Term | Meaning | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Exception event | An event named `$exception` sent by an SDK, with the error type, message, stack, release and breadcrumbs. | | Error issue | One logical error: a group of exception events that Cohorly treats as the same problem. It has a short id like `SHOP-12`, a status and first/last seen. Always called an "error issue", never a bare "issue". | | Fingerprint | The grouping key of an exception: a versioned hash string starting with `v1:`. Events with the same fingerprint belong to the same error issue. One error issue can own several fingerprints over time. | | Release / dist | The build an error came from, sent as `$exception_release` and the optional `$exception_dist`. They are also the key that finds uploaded source maps and R8 mappings without a map id; iOS dSYM symcaches are matched by the binary's debug id (UUID) instead. | | Error worker | A separate process that is the only writer of error issues. It groups, symbolicates and records exceptions. | | Staging | The table `exception_payloads`, where an exception waits, with its full stack and breadcrumbs, until the worker handles it. | | Sampled | A mark on a number that can undercount because the worker discarded some payloads. See [Sampling](#sampling-and-the-sampled-mark). | ## How an exception becomes an error issue 1. **An SDK sends a `$exception` event** to `POST /track` like any other event, with the project token. See [Set up your SDK](#set-up-your-sdk). 2. **`/track` stages it, it does not group it.** An `$exception` is never inserted into `events` by `/track`. It is scrubbed and truncated, then inserted into the staging table `exception_payloads`. The `inserted` count in the response includes exceptions that reached staging. Events in the same batch that are not exceptions go to `events` as usual. 3. **The error worker picks it up.** It symbolicates the stack when a map exists, computes the fingerprint, creates the error issue or attaches the payload to an existing one, updates the counters and `last_seen`, reopens a resolved error issue that came back, and only then inserts the final event into `events` with the same `$insert_id`. The worker is the **only** writer of error issues, fingerprints, counters and the only path from staging into `events`. Doing the grouping in one place keeps a crash loop, where thousands of exceptions share a handful of fingerprints, from contending on the same rows inside every `/track` request. The cost is that an exception becomes visible in an analysis only after the worker has handled it. ### Scrubbing and truncation Before an exception is staged, `/track` removes sensitive text and caps its size: | What | Rule | | ----------------------------------------------------------- | ------------------------------------------------------------------ | | E-mail addresses | Replaced with `[email]`. | | Bearer tokens, JWTs and `token=` / `api_key=` style secrets | Replaced with `[token]`. | | 13 to 19 digit card numbers that pass the Luhn check | Replaced with `[card]`. | | Where it applies | Exception values, `error_title`, and breadcrumb messages and data. | | Exception value | Cut to 1024 characters. | | `error_title` | Cut to 255 characters. | | Exceptions per event | At most 10 (the reported exception and its cause chain). | | Frames per stack | The last 100 are kept. | | Breadcrumbs | The newest 100 are kept. | Property names that your project's data dictionary marks as dropped are removed from the staged payload as well. SDKs can also scrub before sending, see `beforeSendException` on the [Web](/sdks/web#error-tracking) and [React Native](/sdks/react-native#error-tracking) pages. ### What the stored event keeps The event that ends up in `events` is immutable and small. It keeps the ordinary properties the client sent, the final `$exception_fingerprint`, and a flattened summary: `$exception_types`, `$exception_values`, and `$exception_level` (default `error`). `$exception_release` and `error_title` are kept as the SDK sent them. It never carries the stack (`$exception_list`), the breadcrumbs (`$exception_breadcrumbs`) or any symbolication output. Those live only in the staged payload, which is kept for the exception retention of your plan (90 days on the free plan). It also never carries an error issue id. `$exception_issue_id` is a reserved key: `/track` strips it from every incoming event. Which error issue an event belongs to can change when issues are merged, so Cohorly answers it at read time through the fingerprint map. In a Query IR you can still filter or break down by `$exception_issue_id`. A retry of the same exception (same `$insert_id`) is a no-op, whether the first copy is still in staging or already in `events`. ## Fingerprints and grouping A fingerprint is `v1:` followed by the SHA-256 of a canonical input: - the `error_title`, when the SDK sent one, as a prefix; - then either the client-supplied `$exception_fingerprint` (it groups, but is never stored as sent), or the exception types plus the **in-app frames** as function plus module or file name; - a raw native frame with no function name uses its module plus its offset inside the binary, which is stable across launches (this is why uploading a dSYM can change the fingerprint); - line and column numbers are never part of it, because they change on every unrelated edit; - with no in-app frame, the exception types plus the normalized message (numbers, hex values, UUIDs and quoted ids collapsed) stand in. Because the stored event keeps the fingerprint computed at ingest, one logical error can own several fingerprints, for example before and after its source map was uploaded. The fingerprint map is what makes them one error issue. A future algorithm version never rewrites `v1:` fingerprints or the ids people already bookmarked. ### Unsymbolicated error issues If no map has been uploaded for a release yet, the error issue is created right away and marked `unsymbolicated`, showing raw frames. When the source map index, R8 mapping or dSYM symcache arrives, the worker re-symbolicates and either maps the new fingerprint onto the existing error issue or, if that fingerprint already belongs to another error issue, merges the two and keeps the older one. No new-issue alert fires for the symbolicated twin. Right after a map is uploaded, while the worker is still moving earlier payloads of that release onto the symbolicated error issue, new payloads for that release wait (at most 10 minutes after they were received) and then proceed unsymbolicated. System frames (for example UIKit or the Android framework) stay unsymbolicated. ## Sampling and the Sampled mark To protect storage and the worker during a crash loop, the worker records at most: - **20 payloads per fingerprint per hour**, and **200 per day**; - except the **first event of each user who is new to that fingerprint that day**, which is always recorded, up to **1,000 new users per fingerprint per day**. Hours and days are the time `/track` received the exception (UTC), not the event time, so an SDK flushing a day-old offline queue cannot reopen a closed window. Everything over the limits is counted and discarded: the staging row keeps its key (so a retry is not counted twice) and loses its body. The consequence is that the number of **recorded** events can be lower than what **arrived**. Cohorly keeps the two apart: - **Analysis numbers** (events and affected users in a period, trends, breakdowns) are Query IR results and count recorded events only. - **Operational counters** (`received`, `recorded`, `discarded`) are kept by the worker per fingerprint and per hour, include what was discarded, and never feed an analysis number. A result is marked **Sampled** when it can undercount for that reason. In the API, a query result carries a `sampled` field (`{ errorIssues: n }`, the number of error issues in scope that had payloads discarded) and the error issue endpoints return two flags: | Flag | True when | What it means | | ---------------- | ------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | `sampled_events` | Any payload of the error issue was discarded in the period. | `events` undercounts. | | `sampled_users` | The first event of a new user was discarded in the period, by the 1,000 new users cap or by the monthly exception quota. | `users` may undercount. | A daily unique-users trend is exempt from the first kind: since the first event of every new user is always recorded, it is marked Sampled only when the 1,000 new users cap (or the quota) hit. Event totals, hourly unique users, funnels, retention, flows, and any filter or breakdown on an event property other than `$exception_issue_id` are marked whenever anything was discarded in scope. The check matches counters by the hour they were received, so a crash that is sent on the next launch does not mark the range it happened in. Sampled is a warning, not an error: the numbers are correct for the events that were recorded. Read the `received` counter on the error issue to see how many exceptions actually arrived. ## Volume limits that never reject the batch Exceptions have their own limits, separate from the analytics ones. The rate limit is per project; the monthly quota is per organization, shared by all its projects. Both apply only to `$exception` events. | Limit | Default | Over it | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | Exception rate limit | 1000 exceptions per 60 seconds, one token per exception. Self-hosted servers can change it with `COHORLY_EXCEPTION_RATE_CAPACITY` and `COHORLY_EXCEPTION_RATE_WINDOW_MS`. | The exception is refused. | | Monthly exception quota (`exceptionsPerMonth`) | 100,000 recorded exceptions per month on the free plan. | The exception is refused, except the first one per fingerprint per day. | A refused exception is never written anywhere: no staging row, no dedup key, and it is not counted in `inserted`. Cohorly counts it per project and per day, by reason (`rate_limit` or `quota`), so you can see how many never arrived. What matters is what these limits do **not** do: - **They never reject the batch.** `/track` still answers success and the analytics events in the same batch still land. This is the deliberate opposite of the monthly event quota, which rejects a whole batch with a 429. An error storm must never cost you your analytics. The SDK sees the batch as accepted because retrying a quota-refused exception would only be refused again. - **They never count toward the event quota.** `$exception` does not count toward `eventsPerMonth`, and the analytics rate limit and quota only look at non-exception events. The quota counts recorded exceptions, meaning what the worker promoted into `events`. Over the quota, the worker also re-checks at promotion time: a payload is recorded only if its fingerprint has recorded nothing yet that day, otherwise it is discarded and counted. Otherwise a backlog staged while you were under the quota would all be recorded above it. `GET /api/usage` reports `exceptions` (recorded this month), `exceptionLimit`, `exceptionsRefused` (this month, both reasons) and `exceptionRetentionDays`. ## Triage: status, resolve and merge An error issue is `active`, `resolved` or `ignored`. | Action | Who | Behavior | | ---------- | ------------------ | --------------------------------------------------------------------------------------------------------- | | Resolve | Owners and members | Sets `resolved` and records `resolvedInRelease`, by default the release the error issue was last seen in. | | Ignore | Owners and members | Sets `ignored`. An ignored error issue stays silent. | | Reactivate | Owners and members | Sets `active` again. | | Merge | **Owners only** | Folds one error issue into another. **Final: there is no unmerge.** | A `resolved` error issue that receives a new event is reopened as `active` by the worker (a regression). An `ignored` one does not reopen. **Merging** repoints every fingerprint of the merged error issue, and of anything already merged into it, at the survivor. The merged id keeps resolving to the survivor, and no `events` row is touched. Because analyses resolve error issues through the fingerprint map, a merge changes past numbers on purpose: the survivor counts the merged issue's earlier occurrences too. The same actions are available over the API and to AI agents: | Endpoint | Purpose | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /api/errors/issues` | List error issues for a period, with `events`, `users`, `received`, `recorded` and the sampled flags. Filters: `status`, `release`, `search`, `from`, `to`; sort by `last_seen`, `events` or `users`. | | `GET /api/errors/issues/:id` | One error issue by numeric id or short id: fingerprints, counters and the latest recorded occurrence with its stack and breadcrumbs. `eventOffset` steps back to older recorded occurrences. | | `GET /api/errors/fingerprints` | The error issue each `fingerprint` (repeat the parameter, up to 100) resolves to, merges included. | | `PATCH /api/errors/issues/:id` | `{ status, resolvedInRelease? }`. Owners and members. | | `POST /api/errors/issues/:id/merge` | `{ into }`. Owners only, final. | Agents use the same operations through the [MCP server](/api/mcp#error-issues). ## The Errors screen **Errors** in the dashboard's sidebar is where you triage. It has two pages. The **list** shows one row per error: its id, title, level, status (with an `Unsymbolicated` badge when the stack has no symbols yet), events and affected users in the period (14 days by default, up to 90), a daily trend, first and last seen, and the release it was last seen in. Filter by status (Active, Resolved, Ignored), release or a search on the title, and sort by last seen, events or users. Select rows to resolve or ignore several at once. The **detail** page, at `/errors/`, shows: - The header with the triage actions: **Resolve**, **Ignore**, **Reopen**, and **Merge** for owners. **Copy link** copies a link to the error. - First and last seen, first and last release, and the events **received** and **recorded** in the period. - **Occurrences**: events and users per day, broken down by release, app version, OS, browser or device, with a table of counts per value. **Open in Explore** opens the same query in the trend builder, and **Explore affected users** opens it counting unique users. - **Stack trace** of a recorded event, in-app frames first and library frames collapsed. **Newer** and **Older** step through recorded events. - **Breadcrumbs** of that event, oldest first, and its **context**: device, OS, app version, OTA update, browser, URL and the user, linking to their profile. Every count on these pages comes from a query over `$exception` events, so it counts recorded events and shows **Sampled** when payloads were discarded (see above). Received and recorded are counters, labelled as such. In a user's profile, `$exception` events are marked **Error** and link to their error. ## Alerts Owners set up alert rules per project in **Settings, Alerts**; members can see them. A rule picks one or more triggers: | Trigger | When it fires | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | New error issue | The first time a fingerprint creates an error issue. A second event of the same error issue never alerts. | | Regression | A resolved error issue receives an exception again. Ignored error issues stay silent. | | Ingestion stopped | The project received data before and has received no event, `$exception` included, for longer than its threshold (60 minutes by default, 0 turns it off). | | Worker stalled | Exceptions of the project have waited more than 15 minutes for the error worker, so new errors are not visible yet. | Level, release and platform filters (empty means all) narrow the first two triggers. Each rule sends to every member of the organization by email, to a webhook, or both, and **Send test** tries the destinations right away. To keep a burst from flooding you, a project sends at most 10 new error issue alerts per UTC hour (an owner can change it from 1 to 1000). The rest are summed up in one message after the hour ends. Regressions are never held back. ### Webhooks Cohorly sends a `POST` with a JSON body whose `text` field is one line, so a Slack incoming webhook URL works as is. Every request carries an `x-cohorly-event` header with the alert type and an `x-cohorly-signature` header, `t=,v1=`, where the signature is the hex HMAC-SHA256 of `.` keyed by the rule's signing secret (shown to owners in the rule's Edit dialog). Check it before trusting the request: ```js import { createHmac, timingSafeEqual } from "node:crypto"; function verify(secret, header, rawBody, toleranceSec = 300) { const parts = Object.fromEntries(header.split(",").map((p) => p.split("="))); const t = Number(parts.t); if (!t || Math.abs(Date.now() / 1000 - t) > toleranceSec) return false; const expected = createHmac("sha256", secret).update(`${t}.${rawBody}`).digest(); const got = Buffer.from(parts.v1 ?? "", "hex"); return got.length === expected.length && timingSafeEqual(got, expected); } ``` A webhook that fails (no 2xx within 5 seconds, or a redirect) is retried after 1, 5 and 30 minutes and 2 hours, then given up. Delivery is at least once: the `x-cohorly-delivery` header stays the same on every retry, so drop a request whose value you have already handled. On the hosted service the URL must be https and public; a self-hosted instance can allow private addresses with `COHORLY_ALERT_WEBHOOK_ALLOW_PRIVATE=1`. ## Set up your SDK Each SDK page has its own error tracking section with the options and code, so this page does not repeat them. | Platform | Page | What it captures | | ------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------- | | Browser | [Web](/sdks/web#error-tracking) | Unhandled errors and promise rejections, plus `captureException()`. | | React Native | [React Native](/sdks/react-native#error-tracking) | JavaScript errors, an error boundary, `captureException()` and opt-in native crashes. | | Node.js | [Node.js](/sdks/node#error-tracking) | Server-side errors and uncaught exceptions. | | Next.js | [Next.js](/sdks/nextjs#error-tracking) | Server errors through `onRequestError`, and client errors through an error boundary. | | iOS | [iOS](/sdks/ios#crashes) | Native crashes and MetricKit diagnostics, sent on the next launch. | | Android | [Android](/sdks/android#crashes) | Uncaught exceptions, ANRs and native crashes. | Two things every SDK must agree on, or grouping and symbolication suffer: - **Release.** `$exception_release` is the release at crash time, never the release of the launch that sends it. A native crash sent on the next launch, possibly after an update, still belongs to the build that crashed. - **Release and dist match your uploads.** Source maps and R8 mappings are found by release and dist, so the upload must use the same values. iOS dSYMs are matched by the binary's debug id, so upload the dSYM of that exact build. ## Upload symbols Frames from minified or obfuscated builds are symbolicated from files you upload at build time: | Platform | File | | --------------------------------------------- | ----------------------------------------- | | Web, Next.js, React Native (including Hermes) | Source maps, turned into a compact index. | | Android | R8 / ProGuard `mapping.txt`. | | iOS | dSYMs, turned into symcaches (on macOS). | You upload them with the `@cohorly/cli` command line, or with its Vite, Next.js and Expo plugins, using a **project upload token** (`chrly_upl_...`). The token can only upload symbols: every other API route refuses it. Owners mint and revoke upload tokens in Settings > Symbols, where they also see the uploaded artifacts per release and the storage used; members can list them. Commands, options, CI examples and the release and dist rules are on the [CLI page](/cli), so they are not repeated here. Your source code (`sourcesContent`) never leaves the build machine, and the server only looks frames up in the uploaded index. Uploads count against an organization ceiling of 1 GiB of symbol storage on the free plan. ## Privacy The staged payload holds the raw stack and breadcrumbs, which are more identifying than the flattened event. Deleting a user by `distinct_id` removes their `events` and their staged exception payloads. Error issues and counters carry no `distinct_id` and stay, so an error issue can survive even when the deleted user was the only one who hit it. SDK opt-out (`optOutTracking()`) also stops `$exception` events, and `captureExceptions: false` stops them without opting out of everything else. ## Troubleshooting ### My exceptions do not show up Exceptions appear only after the worker has handled them. If analytics events arrive but exceptions never do, the error worker may not be running. On a self-hosted instance it is the `error-worker` service, started separately from the API. ### The stack shows raw, minified frames The error issue is `unsymbolicated`: no map was found. For source maps and R8 mappings, check that the upload used exactly the `$exception_release` and `$exception_dist` the SDK sends; for iOS, upload the dSYM of that exact build. Then upload again. Once the map arrives the worker re-symbolicates and merges the error issue into its symbolicated twin. ### The counts look lower than expected Look for the Sampled mark and compare `received` with `recorded` on the error issue. If the organization went over its monthly exception quota, `exceptionsRefused` in `GET /api/usage` says how many were refused at `/track`. Payloads the worker discarded at promotion are not refusals: they show up in `received` versus `recorded` and in the Sampled mark.