Story NOTIF-010: Cross-cutting rate-limit (Redis) + idempotent retry policy¶

Module: notifications Slice: v0.1.0 NOTIF — Redis rate-limit + idempotent retry aligned to provider webhook (PRD V0 Envelope v0.1.0 "Rate-limit transverse (Redis) + politique de retry idempotente alignée sur le webhook provider"; V1 PRD NOTIF-US-06) Side: [BACKEND] Version target: V0 Sub-version: v0.1.0 Priority: 14 (v0.1.0) Depends on: NOTIF-001b (SMS adapter + status webhook + delivery_attempt lifecycle), NOTIF-004 (crash-recovery job — retry interacts with recovery) Can develop concurrently with: NOTIF-008, NOTIF-009 (may share the Redis store those stories' counters/flags use) Merge order: Any order within v0.1.0; if a shared Redis store is introduced here, NOTIF-008 / NOTIF-009 consume it Estimated complexity: L PRD User Stories: NOTIF-US-06 (AC-06.4 polling fallback, AC-06.5 producer-event idempotency) + PRD V0 Envelope v0.1.0 rate-limit Wireframe: N/A — backend only

⚠️ [ARCH-PENDING] — statut PRÉLIMINAIRE¶

Architecture is v0.0.0-only and explicitly defers the Redis rate-limit and the webhook-loss reconciliation/retry (architecture §0 "Not in v0.0.0"; §5 row 4 "webhook-loss 5-min reconciliation (V1)"). v0.0.0 deliberately ships no retries (state machine §2.3 "no retry in v0.0.0"). This story introduces both and is drafted from V1 PRD NOTIF-US-06 + the PRD V0 Envelope v0.1.0 row. Reconcile with the ARCHITECT v0.1.0 pass before dev-start — especially the Redis topology, the retry/backoff schedule, and how retry interacts with the v0.0.0 crash-recovery job (NOTIF-004).

Objective¶

v0.1.0 introduces (1) a cross-cutting send rate-limit backed by Redis so the platform smooths bursts to providers and shares one limiter store with the cap (NOTIF-008) and kill-switch (NOTIF-009), and (2) an idempotent retry / reconciliation policy aligned to provider webhooks so lost or delayed webhooks do not leave notifications stuck. This is the first sub-version where NOTIF retries; v0.0.0 had none.

Backend Scope¶

Entities¶

Redis: a shared rate-limit store (token-bucket / sliding-window per (tenant_id) and a global ceiling). No new SQL table for the limiter.
Reuses delivery_attempt.status_timeline (JSONB, v0.0.0) and delivery_attempt.attempt_index (v0.0.0 declared but always 1 until now) for retry tracking.
Reuses delivery_attempt.status value path; introduces the terminal UNKNOWN_TIMEOUT (treated as FAILED) per AC-06.4.

Migrations¶

None required for the limiter (Redis). Possibly a small index to support the reconciliation scan over delivery_attempt in SENT without terminal status > 5 min (confirm in ARCHITECT pass).

Service Layer¶

RateLimiter (Redis-backed): consulted before provider submit. On limit hit, the send is deferred (re-queued with backoff), not failed. Per-tenant and global ceilings.
WebhookReconciliationJob (every 5 min): for delivery_attempt rows in SENT (provider accepted) without a terminal status after 5 min, query the provider status API. Continue every 5 min up to 24 h. After 24 h with no terminal status → UNKNOWN_TIMEOUT, treated as FAILED for the parent notification (AC-06.4).
Idempotent retry: retries reuse the existing delivery_attempt (increment attempt_index), never create a duplicate provider submit for the same logical attempt; de-dup aligned to the provider webhook key (provider_id, provider_message_id, status) (v0.0.0 NOTIF-001b).
Producer-event idempotency (AC-06.5): a parent notification publishes notification.delivered / notification.failed exactly once per terminal transition, even under repeated terminal webhooks.

API Endpoints¶

None new. Reconciliation is internal; retries reuse the v0.0.0 status-webhook endpoint contract.

Validation Rules¶

Rate-limit deferral must not violate suppression (NOTIF-001c) or cap/kill-switch (NOTIF-008/009) — guards compose; a deferred send re-checks all gates when it runs.
Reconciliation polling stops at the first terminal status or at 24 h (UNKNOWN_TIMEOUT).

Multi-Tenant Considerations¶

Rate-limit buckets keyed by tenant_id (per-tenant fairness) under a global platform ceiling. Redis keys namespaced by tenant_id. Both V0 DB profiles must work; the Redis limiter is shared infra (not per-tenant DB).

Audit & Logging¶

Retry attempts append to status_timeline (raw provider response ≤ 8 KB, retained 10 years — AC-06.3, already in v0.0.0).
UNKNOWN_TIMEOUT resolution emits the existing NOTIF_FAILURE for the parent notification. No new event_type unless the ARCHITECT pass adds a rate-limit audit — confirm against the AUDIT catalog (critical rule #3) before adding.

Frontend Scope¶

N/A — backend only.

Acceptance Criteria¶

AC-010.1: Rate-limit defers, does not fail
Given a burst of sends exceeding the configured rate for a tenant
When the limiter is hit
Then excess sends are deferred (re-queued with backoff), not failed
And on retry they re-check suppression, cap, and kill-switch gates before submit

AC-010.2: Webhook-loss reconciliation
Given a delivery_attempt in SENT without a terminal status after 5 minutes
When the reconciliation tick fires
Then NOTIF queries the provider status API, repeating every 5 min up to 24 h
And after 24 h with no terminal status the attempt becomes UNKNOWN_TIMEOUT, treated as FAILED for the parent notification

AC-010.3: Idempotent retry and producer events
Given repeated terminal-status webhooks for the same provider_message_id
When they are processed
Then no duplicate provider submit and no duplicate delivery_attempt are created
And notification.delivered / notification.failed is published exactly once per terminal transition per notification

Compliance Rules¶

No direct compliance obligation. Proof-of-delivery reliability (V1 PRD §16.5) is improved by reconciliation. ARTCI B5/B10 production BLOQUANT unchanged.
Any new audit event_type must be in the AUDIT §B.1 catalog (critical rule #3).

Standards & Conventions¶

docs/standards/critical-rules.md (#3 audit; tenant isolation in Redis keys).
docs/standards/tech-stack.md (Redis topology — confirm version/usage).
docs/standards/java-spring-guidelines.md (scheduled jobs, async).
docs/standards/offline-sync.md (idempotency / retry semantics, if relevant).

Testing Requirements¶

Unit Tests¶

Limiter defers above threshold; deferred send re-checks all gates.
Reconciliation transitions to UNKNOWN_TIMEOUT at 24 h.
Producer events fire exactly once per terminal transition.

Integration Tests¶

Burst load → smooth submit under the ceiling, no failures.
Simulated lost webhook → reconciliation resolves status via polling.
Repeated terminal webhooks → single producer event, no duplicate attempt.
Retry composes correctly with suppression / cap / kill-switch.

What QA Will Validate¶

Bursts are smoothed without dropping sends; stuck SENT attempts self-heal via polling or end as UNKNOWN_TIMEOUT; no duplicate sends or duplicate downstream events under webhook replay.

Out of Scope¶

Tenant daily cap (NOTIF-008) and kill-switch (NOTIF-009) — this story may share their Redis store but does not implement them.
Manual operator retry (V1, NOTIF-US-05).
WhatsApp-first / SMS-fallback cascade (replaced in v0.0.0 by SMS-first at T0 → WA at T+5d; NOTIF-003).
Per-agency rate-limit (cap/limit are per-tenant + global; per-agency is V2 if ever).

Definition of Done¶

[ ] Redis rate-limiter defers (not fails) above per-tenant + global ceilings
[ ] Deferred sends re-check suppression / cap / kill-switch on retry
[ ] WebhookReconciliationJob polls SENT-without-terminal every 5 min up to 24 h → UNKNOWN_TIMEOUT
[ ] Retries are idempotent (reuse attempt, no duplicate submit); producer events fire exactly once per terminal transition
[ ] All acceptance criteria pass manually
[ ] Unit + integration tests written and passing (burst, lost-webhook, replay)
[ ] Tenant isolation verified (Redis keys namespaced by tenant_id; both V0 DB profiles tested)
[ ] Any new audit event_type present in AUDIT catalog
[ ] No silent failures on network errors
[ ] [ARCH-PENDING] Redis topology + retry schedule + interaction with NOTIF-004 reconciled with the ARCHITECT v0.1.0 pass before merge to a release branch