Retry, budgets and breakers
Retry, budgets and breakers
Purpose. Defines how many times and until when one destination may be attempted, who decides that, how waiting is handled without blocking a worker, how rate limits are counted, and which breakers exist. Sources: docs/PLAN.md §4.5, §6.2 and §6.6, and the code in src/Core/Retry and src/Core/Guard. The inputs come from Decision (retryable, minDelaySeconds, throttleProvider); the reason codes are RetryBudgetExhausted, CircuitOpen and Expired.
Rules
- One budget per destination. The absolute deadline is
min(logical expiry, retry.max_elapsed). The budget also holdsmax_attemptsandnext_attempt_at. - One planner.
RetryPlanneris the only component that plans attempts. Transport, sender, job and failover all consume the same counter (plan §4.5). TheDecisionsays whether a retry is allowed at all; the planner decides when. - No long sleep. Library code may not sleep for more than a few milliseconds (
AGENTS.md). A wait longer thanretry.inline_threshold_ms(default0: only immediate retries, such as the one right after a token refresh, run inline) becomesDeferredwithnextAttemptAt. - Durable profile:
next_attempt_atis persisted. Light profile: a new job carries the remaining list of destinations.release()is not used, because it would replay a rewritten payload. - A deferred attempt whose time is past the deadline does not touch the network. It becomes
Expired. - Rate limits count messages, not HTTP requests. The key is the Firebase project or the Apple team, never the connection alias. Per topic or per device is added only when the provider signals it. Transactional traffic has a reserved quota.
rate.fcm_per_minuteis an application ceiling, not Google’s quota (plan §4.5).- Three breakers, kept separate:
- Authentication: per credential. Decides failover to the secondary credential once, for everyone.
- Quota: per project.
- Destination quality: per binding, with a minimum sample and a controlled probe before it reopens.
- Time budget chain (plan §6.6), checked by
doctor(QueueTimeouts): transport timeout < chunk deadline < job timeout < worker timeout <retry_after(or the SQS visibility timeout), and in the durable profile chunk deadline < ledger lease, each link with a margin ofmax(5 s, 10 %). The shipped defaults (30 < 40 < 50 < 55 < 90, lease 120) fit a stock Laravel application; seedocs/queues-and-horizon.md. - Defaults (
RetryPolicy, configretry.*):max_attempts5 (first attempt included),max_elapsed3600 s, backoff base 1000 ms doubled per retry up to 300000 ms,jitter0.2, unknown-outcome policyreplaceable. - Jitter (
Backoff): for retry n the exponential part ismin(max, base * 2^(n-1)). Above the provider minimum the delay is spread down by at mostjitterbut never below the minimum ([max(min, exp * (1 - jitter)), exp]); otherwise the minimum dominates and the delay is spread up from it ([min, min * (1 + jitter)]), so destinations throttled together do not return together. - Plans that give up end as
FailedwithRetryBudgetExhausted(attempts ormax_elapsedran out) orExpiredwithExpired(the next attempt would fall at or after the logical expiry); the raw cause is kept inReason::$rawReason. If any attempt of the budget ended unknown the status staysOutcomeUnknown, because the provider may already have accepted the message.
Provider signals
| Signal | Decision (plan §4.5) |
|---|---|
APNs ExpiredProviderToken |
One coordinated refresh of the relevant generation, then one attempt |
APNs TooManyProviderTokenUpdates |
Slow down renewals; do not switch to the secondary credential |
| APNs 5xx on a device send | Deferred, minimum apns.server_error_retry_delay (default 900 s, Apple’s 15 minutes); Expired if that passes the deadline |
APNs TooManyRequests (per device) |
Deferred, minimum apns.device_rate_limit_delay (default 60 s) |
FCM QUOTA_EXCEEDED or UNAVAILABLE |
Deferred, minimum fcm.throttled_min_delay (default 60 s), Retry-After if present; jitter never goes below the minimum |
| Generic HTTP/2 error, timeout after send, unparseable response | OutcomeUnknown with evidence kept (see outcomes.md) |
Types
Retry\RetryPlanner(the only planner),RetryBudget,RetryPlan,RetryPolicy,Backoff,RetryAfter(parser of theRetry-Afterheader: delta-seconds or an HTTP date, clamped to one day),UnknownOutcomeRetryGuard\RateLimiter,ThrottleGate,AuthBreaker(authentication),DestinationQualityBreaker,Guards(facade over them),GuardStore(InMemoryGuardStore; the Laravel layer backs it with the shared cache),AccountKey,TrafficClass. The quota signal is the provider-wide throttle (ThrottleGate, fed byDecision::$throttleProvider); there is no separate “quota breaker” class.Outcome\Decision,SendStatus::DeferredandExpired,ReasonCode::RetryBudgetExhausted,CircuitOpen,Expired,CredentialAction- Laravel:
Queue\QueueTimeouts/TimeoutChain(the chain above), config blocksretry,unknown_outcome,apns,fcm,guards
Worked cases
A. Expired provider token.
- Situation: APNs answers
ExpiredProviderTokenwhile several workers send. - The package: one worker refreshes the token of the relevant generation; the others reuse the new token (compare-and-swap, see
classification.md). Then one more attempt. - The caller sees: the normal result of that attempt. A rejected token never produces a second refresh loop.
B. FCM quota exceeded.
- Situation: FCM answers
QUOTA_EXCEEDEDwithRetry-After: 30. - The package:
Deferred, with the wait at least the configured minimum (fcm.throttled_min_delay, 60 s). ARetry-Aftershorter than the minimum does not lower it; a longer one raises it (Backoffnever schedules before the provider minimum). - The caller sees:
SendStatus::DeferredwithnextAttemptAt. The provider is not called before that time.
C. Backlog past its expiry.
- Situation: a worker picks up a deferred delivery after the logical expiry.
- The package:
Expired, decided without a request. - The caller sees:
SendStatus::Expired. The mock server counts zero requests for that destination.
D. Two aliases on one Firebase project.
- Situation: two connections point at the same project; the quota is per project.
- The package: both aliases share one quota counter.
- The caller sees: the quota error from one alias also slows the other (plan §13 lists this as a case to test).
Plan vs code
- Plan §4.5 says a deferred attempt beyond the deadline becomes
Expired.SendStatus::Expiredhas the same meaning in code. - The numbers the plan left open are now set in
RetryPolicyandconfig/mobile-push.php(see rule 10). Breaker defaults: authentication opens after 5 consecutive credential failures with a 60 s cooldown; destination quality needs a sample of 200 results and opens at 50 % bad destinations with a 300 s cooldown before a controlled probe. Rate ceilings:guards.rate.fcm500000 messages per minute per project,guards.rate.apnsunlimited, 20 % reserved for transactional traffic. Retry-Aftershorter than the provider minimum: the minimum wins (Backoffnever schedules earlier thanDecision::$minDelaySeconds).TooManyProviderTokenUpdatesisReasonCode::AuthTokenThrottledwithCredentialAction::SlowTokenRefresh: renewals slow down and the secondary credential is never used for it.
Open points
- Whether the provider-wide throttle for APNs should honour more than the documented signals: only the ones in the catalogs are used. Not verified against real traffic.
- The destination-quality thresholds are engineering defaults, not provider values; tune them against real traffic.
- Rate ceilings and the inline threshold were exercised against fake and mock transports only, never against Apple or Google.
This page is generated from docs/contracts/retry.md. To change it, edit that file in the repository, not this site.