# Webhook retries

> TimeTracking webhook retry schedule (7 attempts over about 21 hours), what counts as failure, how to inspect and replay deliveries, and how to build an idempotent receiver.

URL: https://timetracking.tech/developers/webhooks/retries
Last updated: 2026-10-24

## What counts as success

| Outcome | Result |
|---|---|
| HTTP `2xx` within 10 s | `delivered` |
| HTTP `3xx` | **failure** (redirects are never followed) |
| HTTP `4xx` / `5xx` | failure |
| Timeout (10 s), DNS error, TLS error, connection refused | failure |
| URL now resolves to a private/loopback address | failure (`last_error` explains why) |
| Webhook deleted or inactive | `failed` immediately, `last_error: "webhook inactive"` |

## Schedule: 7 attempts in total

The first attempt is made by the next scheduler run (every 5 minutes). After each failure the next attempt is scheduled with this backoff:

| Attempt | Delay after the previous failure | Cumulative (approx.) |
|---|---|---|
| 1 | — (next scheduler run, ≤ 5 min) | 0 |
| 2 | 1 min | 1 min |
| 3 | 5 min | 6 min |
| 4 | 30 min | 36 min |
| 5 | 2 h | 2 h 36 min |
| 6 | 6 h | 8 h 36 min |
| 7 | 12 h | 20 h 36 min |

After the 7th failed attempt the delivery is marked `failed` and is not retried again. Actual times can be up to 5 minutes later, because the scheduler runs every 5 minutes.

Each delivery records `attempts`, `last_status_code`, `last_error` (max 500 chars) and `next_attempt_at`.

## Inspect and replay

Admin key or admin user token required.

```sh
# Last 50 deliveries of a webhook, newest first
curl -s "$TT/webhooks/$HOOK_ID/deliveries?limit=50" -H "Authorization: Bearer $ADMIN_USER_TOKEN" \
  | jq '.data[] | {id, event, status, attempts, last_status_code, last_error}'

# Re-queue a failed (or delivered) delivery: status → pending, attempts → 0, sent at the next scheduler run
curl -s -X POST "$TT/webhook-deliveries/21/replay" -H "Authorization: Bearer $ADMIN_USER_TOKEN"
# → {"data":{"requeued":true}}
```

The payload of a replayed delivery is identical, including the event `id`. The signature is computed fresh with the **current** secret and a new timestamp.

Recovery after an outage:

```sh
curl -s "$TT/webhooks/$HOOK_ID/deliveries?limit=200" -H "Authorization: Bearer $ADMIN_USER_TOKEN" \
  | jq -r '.data[] | select(.status=="failed") | .id' \
  | while read id; do curl -s -X POST "$TT/webhook-deliveries/$id/replay" -H "Authorization: Bearer $ADMIN_USER_TOKEN"; echo; done
```

Only the most recent 200 deliveries are visible per call. For longer outages, reconcile from the API instead. For example, `GET /days?from=…&to=…&updated_since=<outage start>` and `GET /exports/runs?updated_since=…`.

## Building an idempotent receiver

- **Dedupe on `event.id`.** Store processed IDs for at least 7 days. A replay or a timeout followed by a retry delivers the same `id` again.
- **Acknowledge, then process.** Return `200` as soon as the signature is valid and the event is queued. Slow handlers cause timeouts and therefore duplicates.
- **Don't trust ordering.** Events can arrive out of order, for example `timesheet.approved` before `timesheet.submitted` after a retry. Re-read the current state from the API (`GET /days/...`, `GET /exports/runs/{id}`) instead of applying `data` as a delta.
- **Return 2xx for events you ignore.** Otherwise they are retried for about 21 hours.
- **Alert on `failed`.** Poll `GET /webhooks/{id}/deliveries` daily, or watch your own endpoint's error rate.

## Testing your endpoint

```sh
curl -s -X POST "$TT/webhooks/$HOOK_ID/test" -H "Authorization: Bearer $ADMIN_USER_TOKEN"   # queues webhook.test
sleep 300   # wait for the scheduler
curl -s "$TT/webhooks/$HOOK_ID/deliveries?limit=1" -H "Authorization: Bearer $ADMIN_USER_TOKEN" | jq '.data[0] | {status, last_status_code, last_error}'
```

For local development, expose your receiver with a public HTTPS tunnel (e.g. `cloudflared`, `ngrok`). `localhost` and private addresses are rejected.