Aller au contenu

Story NOTIF-004: Crash-recovery startup job (SmsSubmitRecovery)

Module: notifications Slice: NOTIF-004 Side: [BACKEND] Version target: V0.0.0 Sub-version: v0.0.0 (architecture-backed) Priority: 7 Depends on: NOTIF-001b (uses the same SmsProvider adapter + delivery_attempt.attempt_id idempotency contract) Can develop concurrently with: NOTIF-002, NOTIF-003 Merge order: Any order after NOTIF-001b Estimated complexity: S PRD User Stories: N/A — Technical Story (closes architecture §5 row "Crash between accept and SMS submit") Wireframe: N/A — backend only

Objective

On JVM startup, scan for notification rows stuck in DISPATCHING_SMS older than 60 seconds with no delivery_attempt.provider_message_id set, and resubmit them via the same SmsProvider adapter. The adapter's client-ref idempotency (delivery_attempt.attempt_id) guarantees a resubmit returns the same provider_message_id, so this is exactly-once even if the original submit succeeded server-side but the JVM died before the local UPDATE landed.

Backend Scope

Entities

No new tables. Uses existing notification + delivery_attempt from NOTIF-001a/b.

Migrations

None new.

Service Layer

  • SmsSubmitRecovery — Spring ApplicationRunner (runs once on ApplicationReadyEvent):
  • SELECT n.*, a.* FROM notification n JOIN delivery_attempt a ON a.notification_id = n.id WHERE n.status='DISPATCHING_SMS' AND a.channel='SMS' AND a.attempt_index=1 AND a.provider_message_id IS NULL AND a.submitted_at < now() - INTERVAL '60 seconds' FOR UPDATE SKIP LOCKED LIMIT ${notif.recovery.batch.size:200}.
  • For each row, log INFO recovery_resubmit with {notification_id, attempt_id, correlation_id, age_seconds}.
  • Call SmsProvider.send(...) via the same adapter NOTIF-001b uses, passing client_ref = attempt_id. The adapter's idempotency contract (architecture §5 row 2) guarantees the provider returns the same provider_message_id whether or not the original submit reached it.
  • On 2xx → UPDATE notification SET status='SMS_SENT', sms_sent_at=now(), wa_scheduled_for = sms_sent_at + INTERVAL '5 days' (CAS from DISPATCHING_SMS); UPDATE delivery_attempt SET status='SENT', provider_message_id=? (CAS from SUBMITTING; idempotent on second start). Emit NOTIF_SENT outbox event ONLY if terminal_event_published_at IS NULL (exactly-once guarantee from NOTIF-001a's column). Transition to AWAITING_CASCADE.
  • On hard-error / timeout → notification.status='FAILED', last_failure_class='RECOVERY_HARD_ERROR', emit NOTIF_FAILURE (guarded by terminal_event_published_at).
  • Runs in batches (default 200) until the result set is empty; then exits. Total recovery work is bounded by the number of stuck rows at startup (typically 0 on a clean shutdown).
  • Property gate: notif.recovery.enabled (default true). Setting false skips recovery (useful for forensic startups).

API Endpoints

None.

Validation Rules

None new.

Multi-Tenant Considerations

  • SELECT FOR UPDATE SKIP LOCKED is global but each row carries its tenant_id into MDC for the resubmit. Critical-rule #1 enforced per-row.
  • Both V1 DB profiles supported (NOTIF tables on platformDataSource).
  • correlation_id from the stuck row is restored to MDC before the resubmit (NOTIF-ADR-07).

Audit & Logging

  • INFO log line on every recovery attempt: {notification_id, attempt_id, correlation_id, age_seconds, outcome}.
  • NOTIF_SENT / NOTIF_FAILURE emission is guarded by notification.terminal_event_published_at (set on first emission per NOTIF-001a/b) — recovery resubmit will NOT double-emit if the original @Async task already published before the JVM died.
  • Recovery batch metrics: stuck-row count at start, recovered count, failed count. Surfaced via the standard observability stack (architecture §9 / NFR observability — V1 dashboards; v0.0.0 emits to metrics endpoint without UI).

Frontend Scope

N/A — backend only.

Acceptance Criteria

AC-004.1: Simulated crash mid-submit → recovery succeeds exactly-once
Given a notification was enqueued via POST /dispatch
And the @Async SmsDispatcher was forcibly stopped (asyncExecutor.shutdownNow()) BEFORE its provider call landed
And the notification.status is therefore stuck at 'DISPATCHING_SMS' with delivery_attempt.provider_message_id=NULL and submitted_at > 60s ago
When SmsSubmitRecovery runs on the next startup
Then exactly one resubmit reaches the SMS provider (WireMock asserts request count == 1, even if the first attempt had actually made it to the provider — adapter idempotency via client_ref=attempt_id)
And notification.status transitions to 'AWAITING_CASCADE' with sms_sent_at and wa_scheduled_for set
And `NOTIF_SENT` is in the outbox EXACTLY ONCE for this notification (guarded by terminal_event_published_at)

AC-004.2: Hard-error on recovery → FAILED with RECOVERY_HARD_ERROR class
Given the same crash scenario as AC-004.1
When SmsSubmitRecovery runs and the SMS provider returns a hard error (4xx) on resubmit
Then notification.status='FAILED', last_failure_class='RECOVERY_HARD_ERROR'
And delivery_attempt.status='HARD_ERROR'
And `NOTIF_FAILURE` is emitted exactly once (guarded by terminal_event_published_at)
And `notification.failed` Spring event fires exactly once

Compliance Rules

  • Critical-rule #3 (audit completeness) — NOTIF_SENT / NOTIF_FAILURE MUST be emitted exactly once per terminal transition; the terminal_event_published_at guard is non-negotiable. Resubmit alone is not enough; the audit trail must reflect the eventual outcome without duplication.
  • Critical-rule #1 — every recovery query and resubmit scoped by tenant_id (carried from the stuck row).
  • This story does not lift or touch any INCERTAIN legal status.

Standards & Conventions

  • docs/standards/critical-rules.md
  • docs/standards/java-spring-guidelines.md (ApplicationRunner, ApplicationReadyEvent, MDC restoration)
  • docs/standards/database-guidelines.md (FOR UPDATE SKIP LOCKED, CAS UPDATE)
  • docs/standards/multi-datasource-patterns.md
  • docs/standards/multi-tenant-model.md
  • docs/standards/tech-stack.md

Testing Requirements

Unit Tests

  • SmsSubmitRecoveryQueryTest — query returns only rows older than 60s AND in DISPATCHING_SMS AND provider_message_id IS NULL.
  • SmsSubmitRecoveryIdempotencyTest — second invocation of recovery.run() on the same JVM produces zero re-resubmit (rows already transitioned to AWAITING_CASCADE).

Integration Tests

  • CrashRecoveryE2ETest (per architecture §13.3 sample structure) — dispatch, forcibly shut down the @Async executor, run SmsSubmitRecovery, assert WireMock count == 1, assert state and outbox.
  • RecoveryNoDoubleAuditTest — pre-seed terminal_event_published_at (simulating the case where the original @Async DID publish before crashing), run recovery, assert outbox count for NOTIF_SENT stays at 1.
  • RecoveryMultiTenantTest — three stuck rows across three tenants, run recovery, assert each row's audit event carries its correct tenant_id + correlation_id.
  • RecoveryDisabledTest — notif.recovery.enabled=false → recovery runner exits immediately, stuck rows remain stuck.

What QA Will Validate

  • Run dispatch, kill JVM hard, restart, observe recovery picking up the stuck row.
  • Provoke a doubled scenario (recovery + original @Async both completing) → exactly-one audit event.
  • Empty-startup case: no stuck rows → recovery exits in < 100ms, no side effects.

Out of Scope

  • Recovery for DISPATCHING_WA (WhatsApp cascade) — V1 (the cascade firer's FOR UPDATE SKIP LOCKED already handles its own restart semantics per architecture §5 row 3).
  • Retry policy for transient errors — v0.1.0 (architecture §5 row 5).
  • Operator UI surfacing of recovery activity — V1.
  • Manual operator-triggered recovery — V1.
  • Webhook-loss reconciliation (separate concern handled at V1 NOTIF-US-06).

Definition of Done

  • [ ] SmsSubmitRecovery ApplicationRunner implemented and registered
  • [ ] Property gate notif.recovery.enabled (default true) wired
  • [ ] Adapter resubmit uses client_ref = attempt_id (verified against WireMock idempotency assertions)
  • [ ] Both ACs pass
  • [ ] terminal_event_published_at correctly prevents double audit emission across recovery + original async paths
  • [ ] Tenant isolation verified (multi-tenant recovery test)
  • [ ] Both V1 DB profiles tested
  • [ ] correlation_id restored to MDC on the recovery thread
  • [ ] No silent failures: every recovered row produces either NOTIF_SENT or NOTIF_FAILURE exactly once
  • [ ] Recovery exits cleanly when no stuck rows exist (empty-case test)