Retry and backoff policies

Repetitions (retries) help to survive temporary failures, but if incorrectly configured, they cause avalanches of traffic, duplicates of operations and cascading falls. A reliable retray policy always starts with deadlines/timeouts, takes into account idempotency and uses backoff + jitter.

1) Basic principles

1. First timeout/deadline, then retreat. Repeating without a time limit only lengthens the failure.
2. Retray - only for safe/idempotent operations. For insecure ones - through idempotency-keys and transaction guarantees.
3. Backoff is mandatory. Exponential or GCRA-like with jitter to desynchronize waves.
4. Limit the number of attempts and the total budget time. Do not leave the user's SLO.
5. Respect infrastructure signals. '429/503' + 'Retry-After', circuit breaker status, queue limits.

2) Timeouts and deadlines (deadline propagation)

Request timeout <SLO service, deadline propagates down the chain (HTTP headers/gRPC context).
Timeout composition: total (first attempt + backoff + subsequent) ≤ user deadline.
Different timeouts for reads/records: records are shorter and stricter; reads allow hedging.

3) Idempotency and safe repeats

Reads (GET/idempotent RPC): safely repeated with '5xx', 'UNAVAILABLE', network timeouts.

• Use'Idempotency-Key '(HTTP) or request-ID on the border; the server must deduplicate.
• Write idempotent handlers: "upsert," "at-least-once" + compensation (saga).
• External payments/settlements - only with idempotent keys and transaction log.

4) Backoff algorithms

Exponential: 'base 2 ^ attempt', bounded by 'max _ backoff'.
Decorated Jitter (full/equal jitter) - randomness in the range to desynchronize clients.
GCRA/Token-Bucket-like delays: consistent with rate limits.
Borders: 'initial _ backoff' (50-200 ms reads; 200-500 ms records), 'max _ backoff' (1-5 s), 'max _ elapsed' (for example, 3-10 s).

The recommended pattern is exponential backoff + full jitter.

5) Repeat/Do Not Repeat Decision Policies

• Network errors/timeouts, '429' (respectfully 'Retry-After'), "soft" '5xx' ('502/503/504'), gRPC 'UNAVAILABLE/DEADLINE _ EXCEEDED'.

• '4xx '(except' 409/429/408 'in some scenarios), business errors,' 401/403 ', validation errors, explicit' DoNotRetry'flag.

• '409 Conflict '- sometimes repeated with a delay after consensus/locks.
• '404'for permanently consistent reads - one or two times retray with a small backoff.

6) Concurrency caps and "retray storm"

Limit simultaneous retrays per-client/per-tenant/per-endpoint.
Total limit of attempts per request (e.g. 2-3).
Slow down hot endpoints through admission control so as not to inflate queues.

7) Interaction with Circuit Breaker and limits

If CB is open, do not perform retrays directly - go to fallback or wait for 'half-open' samples.
At '429' - respect 'Retry-After'; if not, use "soft" backoff.
Retrays can increase strain; apply adaptive thresholds (lower 'max _ attempts' for an incident).

8) Protocols and contracts

HTTP

Codes: '408/429/5xx'.
Headers: 'Retry-After', family 'RateLimit-', 'Idempotency-Key', 'Request-Id'.
The client must send'X-Request-Timeout '/' Deadline-At '(if so).

gRPC

Use context with a deadline; respect 'UNAVAILABLE', 'DEADLINE _ EXCEEDED', retray policies per method.
For idempotent RPCs, include retrays; for mutations - only with idempotency support.

9) Queues, background tasks and integrations

At-least-once handlers → idempotent actions, key deduplication.
Delay queue for backoff between attempts (for example, 5s/30s/2m).
Dead-letter queue (DLQ) with a limit of attempts and manual processing.
Outbox/CDC - so that replays do not compromise transactional integrity.

10) Hedging vs Retries

Hedging is useful for highly critical reads in tail latency.
Limit: no more than X% of requests, start-grace delay (for example, p95 latency), cancellation of losers.
Do not apply hedging to write operations without strong idempotency.

11) Telemetry and observability

Теги: `tenant_id`, `endpoint`, `attempt`, `decision` (retry/skip), `reason`, `backoff_ms`, `deadline_ms`, `idempotency_key`.
Metrics: share of retreats, success after retreats, p95/p99 "end-to-end," number of exceeded deadlines, CB triggering.
Audit logs: upper N "noisy" keys/endpoints, correlation with 429/503.

12) Testing and chaos

Profiles: "saw" (burst-lull), "storm" (mass timeouts), "sticky" errors (every Nth), latency tails.
Torus limiter/cache/queue failures, clock-skew.
Checks that the total duration (attempts + backoff) fits into the SLO.

13) Policy pseudocode

pseudo handle(req, deadline):
attempt = 0 backoff = initial()
while attempt < MAX_ATTEMPTS and now() < deadline:
attempt += 1 with timeout(per_attempt_timeout(deadline, attempt)):
try:
resp = call(req)
if isRetryableStatus(resp): raise Retryable(resp. status)
return resp except Retryable as e:
if circuit. isOpen(dep) or! isIdempotent(req): break sleep(jitter(backoff))
backoff = min(exp(backoff), MAX_BACKOFF)
except NonRetryable:
break return fail_or_fallback(req)

14) Configuration template (example)

yaml retries:
default:
max_attempts: 3 initial_backoff_ms: 150 max_backoff_ms: 2000 strategy: exponential_full_jitter respect_retry_after: true per_attempt_timeout_fraction: 0. 4 # 40% of remaining deadline hedging:
enabled: false read_heavy:
max_attempts: 4 initial_backoff_ms: 80 max_backoff_ms: 1200 hedging:
enabled: true start_after_p95_ms: 300 max_extra_requests_ratio: 0. 05 write_strict:
max_attempts: 2 initial_backoff_ms: 250 max_backoff_ms: 1000 idempotency_required: true

limits:
concurrent_retries_per_tenant: 100 concurrent_retries_per_endpoint: 20

15) Pre-sale checklist

• Deadlines propagate through calls; total duration of SLO ≤.
• Backoff algorithm with jitter; parameters validated by load test.
• IDempotence: keys/logs/compensations for records.
• Retray policies differ for reads and records; respect `Retry-After`.
• Retray competitiveness and total number of attempts are limited.
• Integration with circuit breaker and rate limits is configured.
• Telemetry: tags, metrics, reason logs; dashboards p95/p99, share of success after retrays.
• Tests of "storms" and latency tails, DLQ for tasks.
• Customer documentation: codes/titles, backoff and cancellation examples.

16) Typical errors

Replays without timeouts/deadlines are "eternal expectations."

Fixed pauses without jitter - synchronous waves and DDoS self-sabotage.
Retrays of unsafe records without idempotency - duplicates and desynchronization.
Ignoring 'Retry-After' and CB signals - escalating incident.
Bottlenecks in queues due to lack of caps and admission control.

Lack of telemetry of reasons/solutions of retrays - "blind flight."

17) Quick recipes

Public API reads: 3 attempts, 'initial = 100ms', 'max = 1s', full jitter, hedging up to 5% of traffic.
Critical records (payment): 1-2 max attempts, strict timeout, mandatory 'Idempotency-Key', no hedging.
External integrations: respect '429/Retry-After', 'max _ attempts = 3', 'max _ backoff = 2-5s', outgoing flow limits.
Background tasks: delay-backoff (5s → 30s → 2m), DLQ, idempotent handlers.

Conclusion

A good repetition policy is a balance between rapid recovery and controlled failure. Deadlines, idempotence, exponential backoff with jitter, limiting competition and respecting infrastructure signals turn retrays from a "storm" into a tool for improving SLO reliability and retention.