Story ING-005: Startup recovery + ARTCI feature-flag prod gate + structured logging¶

Module: ingestion Slice: Tier 1 sub-slice ING-001-d (architecture §11) Side: [BACKEND] Version target: V0.0.0 Priority: 5 Depends on: ING-004 (ingestion_import table + status FSM exist; audit emission for ING_PARSE_FAILED wired) Can develop concurrently with: ING-006 (integration-test pack — read-only on this slice's outputs) Merge order: After ING-004; before ING-006 Estimated complexity: S PRD User Stories: Derived from V0 PRD §6.4 (ARTCI feature-flag prod gate) and architecture §5.1 stage 4 (crash recovery) Wireframe: N/A — backend only

Objective¶

Make the v0.0.0 cron production-safe: the IngestionStartupRecovery job clears stuck PICKED_UP rows orphaned by a crash mid-parse (architecture §5.1 stage 4 + §6.4), and the feature.artci.phone-auth.granted feature flag short-circuits the cron on prod profile while ARTCI authorisation is pending (architecture §0 + ADR-08). Structured logging makes the gated-skip visible to ops without burying the warning.

Backend Scope¶

Entities¶

None new. Reads + UPDATEs existing ingestion_import rows.

Migrations¶

None.

Service Layer¶

IngestionStartupRecovery (ApplicationRunner, runs ONCE at boot before @Scheduled activates):
For every active tenant: SELECT id, file_minio_inbox_key, filename FROM ingestion_import WHERE tenant_id=? AND status='PICKED_UP' AND picked_up_at < now() - INTERVAL '${INGESTION_STARTUP_RECOVERY_STALE_THRESHOLD_SECONDS:300} seconds'.
For each match: UPDATE status='PARSE_FAILED', parse_error_detail = '{"reason":"recovered_from_crash"}'::jsonb, finished_at = now(). Emit ING_PARSE_FAILED with the same reason. If the inbox file is still present, MinIO COPY+DELETE inbox→/<tid>/rejected/<ts>__<filename> with sidecar JSON {"reason":"ingestion crashed during parse, please re-drop the file"}.
Runs synchronously at boot, sequential per tenant, bounded by a 30 s total budget — if it takes longer the application still starts (recovery resumes next boot).
WARN-level log per recovered row: [ING-RECOVERY] tenant=<tid> import=<id> filename=<f> reason=recovered_from_crash.
ArtciProdGate (@Component): wraps IngestionService.tick(). Reads Environment.acceptsProfiles(Profiles.of("prod")) AND feature.artci.phone-auth.granted. Logic:
On non-prod profiles (dev, test, staging, local) → always proceed regardless of flag.
On prod AND flag=false → log WARN [LEGAL-REVIEW] ingestion cron skipped — ARTCI phone-auth pending. files in inbox NOT processed., return without calling MinIO LIST. No tenant iteration, no DB read.
On prod AND flag=true → proceed normally.
On startup (regardless of profile) when flag is false, log a single banner WARN [LEGAL-REVIEW] feature.artci.phone-auth.granted=false → ingestion cron disabled in prod profile.
Structured logging hook: every cron iteration boundary emits a single-line MDC-enriched INFO with {tenantId, objectsFound, durationMs, status} (logs are aggregated in CloudWatch / Loki per docs/standards/java-spring-guidelines.md § observability).

API Endpoints¶

None new.

Validation Rules¶

INGESTION_STARTUP_RECOVERY_STALE_THRESHOLD_SECONDS minimum 60 s (safeguard against marking healthy in-flight rows as failed). Default 300 (architecture §14.2).
Feature flag value must be a parseable boolean (true / false); any other value treated as false with a WARN log (fail-safe to closed).

Multi-Tenant Considerations¶

Startup recovery iterates TenantConfigCache.listActiveTenants() same as the cron. tenant_id scoping enforced on every SELECT and UPDATE. Audit emissions use actor_type=SYSTEM, actor_id=ingestion-cron (same as steady-state). Both DB profiles supported — recovery queries route through the same datasource resolver.

Audit & Logging¶

ING_PARSE_FAILED emitted once per recovered row with parse_error_detail.reason="recovered_from_crash" (catalog already extended by OQ-X10 in ING-004's prerequisite).
WARN logs on every gated skip + every startup-recovery hit.
No new event types added.

Frontend Scope¶

N/A — backend only.

UI States (ALL REQUIRED)¶

State	Behavior	French Copy
Loading	N/A — backend only	—
Empty	N/A — backend only	—
Error	The sidecar JSON written by recovery is operator-facing	`{"reason":"ingestion crashed during parse, please re-drop the file"}` (English; matches existing sidecar shape from ING-004)
Offline / low network	N/A — backend only	—
Success	N/A — backend only	—

Responsive Behavior¶

N/A.

Interactions¶

N/A.

Acceptance Criteria¶

Architecture §11 ING-001 binding ACs 4–5:

AC-001: Prod profile + flag=false ⇒ cron tick is a no-op
Given the application running with `spring.profiles.active=prod`
  And `feature.artci.phone-auth.granted=false`
  And a file in `/<tenantA>/inbox/clean-5-rows.csv`
When `IngestionService.tick()` fires
Then no MinIO LIST is performed
  And no `ingestion_import` row is created
  And the file remains in `/<tenantA>/inbox/`
  And the WARN log line `[LEGAL-REVIEW] ingestion cron skipped — ARTCI phone-auth pending. files in inbox NOT processed.` is emitted.

AC-002: Non-prod profiles ignore the feature flag
Given the application running with `spring.profiles.active=dev` (or `test`, `staging`, `local`)
  And `feature.artci.phone-auth.granted=false`
When `IngestionService.tick()` fires
Then the cron proceeds normally (MinIO LIST + downstream FSM)
  And no `[LEGAL-REVIEW] cron skipped` log line is emitted.

AC-003: Stuck PICKED_UP row >5 min old is recovered at boot
Given an `ingestion_import` row inserted with `status='PICKED_UP'` and `picked_up_at = now() - 10 minutes` for tenant A
  And the corresponding file `stale.csv` still present in `/<tenantA>/inbox/`
When the application boots and `IngestionStartupRecovery` runs once
Then the row is updated to `status='PARSE_FAILED'` with `parse_error_detail.reason="recovered_from_crash"` and `finished_at` set
  And the file is moved to `/<tenantA>/rejected/<ts>__stale.csv` with sidecar JSON `{"reason":"ingestion crashed during parse, please re-drop the file"}`
  And `ING_PARSE_FAILED` is present in `audit_log` for that `import_id`.

Compliance Rules¶

ARTCI loi 2013-450 Art. 14 (consentement préalable au démarchage) — [LEGAL-REVIEW] PRÉLIMINAIRE per V0 PRD §6.4. The flag default-false posture is the production-time mitigation. WARN log at startup ensures auditability.
Architecture ADR-08: feature flag default false on prod profile; ignored elsewhere.
Critical rule #3: every recovered row emits ING_PARSE_FAILED — catalog admits the event type.

Standards & Conventions¶

docs/standards/compliance-discipline.md — [LEGAL-REVIEW] warning posture.
docs/standards/critical-rules.md (#3 audit catalog).
docs/standards/java-spring-guidelines.md § ApplicationRunner, § MDC logging.
docs/standards/multi-tenant-model.md § cron context.

Testing Requirements¶

Unit Tests¶

ArtciProdGateTest — matrix {prod|dev|staging|test|local} × {flag=true|false|garbage|absent}; asserts skip vs proceed and exact log line.
IngestionStartupRecoveryTest (mocked DB + MinIO) — threshold honored; younger PICKED_UP rows untouched.

Integration Tests (TestContainers)¶

IngestionStartupRecoveryIT — AC-003 end-to-end with real Postgres + MinIO.
IngestionProdFlagClosedIT — AC-001 setup; assert zero MinIO requests via spy on IngestionInboxClient.
IngestionDevFlagIgnoredIT — AC-002 assertion that the cron proceeds.

What QA Will Validate¶

Boot with SPRING_PROFILES_ACTIVE=prod FEATURE_ARTCI_PHONE_AUTH_GRANTED=false; drop a file; tail logs; confirm the [LEGAL-REVIEW] cron skipped line appears and mc ls .../inbox/ still shows the file.
Insert a stale PICKED_UP row via psql; restart; confirm audit_log entry and mc ls .../rejected/.

Out of Scope¶

The cross-cutting integration test pack (tenant isolation, large file, magic byte, full crash-recovery matrix) — ING-006.
ShedLock / multi-instance cron coordination — V1 (single-instance deploy in v0.0.0; ADR-03).
Operator notification badge for rejected files — V0.0.1.
Email / SMS alerting on recovery — V2.
Auto-retry of recovered rows — never (operator must re-drop; architecture §6.4 explicit).
Feature-flag UI in the admin console — V1+ (env-var only in v0.0.0).

Definition of Done¶

[ ] All backend endpoints implemented and returning correct responses (none added)
[ ] All frontend pages render correctly at 360 / 768 / 1024 px (N/A — backend only)
[ ] All 5 UI states implemented (N/A — backend only)
[ ] French micro-copy matches design spec exactly (sidecar shape reused from ING-004; recovery reason string set)
[ ] All acceptance criteria pass manually
[ ] Unit + integration tests written and passing
[ ] Tenant isolation verified (recovery iterates per tenant; SHARED + BYO smoke green)
[ ] [CUSTOMER stories] payload + Slow-3G + resumability (N/A — backend only)
[ ] Audit log entries present for every required event (ING_PARSE_FAILED per recovered row)
[ ] No silent failures on network errors (recovery's MinIO move failures logged WARN; row stays PARSE_FAILED — re-running recovery is idempotent)