Aller au contenu

Story ING-002: Scheduler + MinIO inbox client + tenant iteration skeleton

Module: ingestion Slice: Tier 1 sub-slice ING-001-a (architecture §11) Side: [BACKEND] Version target: V0.0.0 Priority: 2 Depends on: ING-001 (test infra + empty migrations), TENANT-001 (TenantConfigCache bean + default-agency provisioning), AUDIT-001 (AuditEventBuilder + outbox) and AUDIT catalog extension OQ-X10 admitting ING_UPLOAD_RECEIVED Can develop concurrently with: ING-003 (pure parser/validator code has no overlap with this slice) Merge order: Before ING-004 Estimated complexity: S PRD User Stories: Derived from ING-US-01 (upload pickup pathway; cron substitute in v0.0.0) Wireframe: N/A — backend only

Objective

Wire the nightly @Scheduled cron that iterates active tenants, lists /<tenantId>/inbox/ in the platform MinIO bucket, and emits ING_UPLOAD_RECEIVED audit when a file is picked up. No parsing, no validation, no commit yet — just the orchestration skeleton plus the MinIO wrapper used by all subsequent slices. After this story, the operator dropping a file in MinIO causes an audit row and a processed/ move at the next tick.

Backend Scope

Entities

None new. This slice does not yet write ingestion_import rows — that arrives with ING-004. It only emits audit events with the MinIO key as subject_id-equivalent metadata.

Migrations

None. (V010 body still empty; filled in ING-004.)

Service Layer

  • IngestionScheduler (@Component): single @Scheduled(cron = "${INGESTION_CRON_EXPRESSION:0 0 2 * * *}", zone = "${INGESTION_CRON_ZONE:Africa/Abidjan}") method tick(). Also exposed via /internal/ingestion/tick POST endpoint only on local profile for the smoke verification command in architecture §14.3.
  • IngestionService.tick(): iterates TenantConfigCache.listActiveTenants(), opens TenantContext.scope(tenant.id, tenant.countryCode), calls processInbox(tenant) per tenant sequentially (architecture §4.1).
  • IngestionInboxClient (MinIO wrapper): list(tenantId) → Stream<InboxObject>, get(tenantId, key) → InputStream, moveToProcessed(...), moveToRejected(...), moveToDuplicate(...), putErrorsSidecar(...). Keys built only from the iteration-bound tenantId (architecture §2.5, §7.6 threat A4).
  • IngestionAuditEvents.uploadReceived(...): builds ING_UPLOAD_RECEIVED AuditEventBuilder payload with { filename, file_size_bytes, file_hash_sha256, minio_inbox_key } — no row content.
  • In this slice processInbox flow = list → for each object: read size + stream-hash SHA-256 → emit ING_UPLOAD_RECEIVED → call moveToProcessed (placeholder behavior; ING-004 replaces with full parse/commit/reject branching).

API Endpoints

Method Path Request Response Auth
POST /internal/ingestion/tick (local profile ONLY) — 204 No Content None (gated by profile filter)

Validation Rules

  • Magic-byte verification (CSV vs XLSX) deferred to ING-003 parsers.
  • File-size hard cap (INGESTION_MAX_FILE_BYTES=5_242_880) enforced at hash-streaming step: if cumulative bytes exceed limit, abort the iteration for that file, emit ING_UPLOAD_RECEIVED with file_size_bytes set to the limit + a truncated:true flag, move file to /<tid>/rejected/ with sidecar reason FICHIER_TROP_VOLUMINEUX. (Note: in ING-004 this becomes a proper PARSE_FAILED row; in ING-002 there is no DB row yet.)

Multi-Tenant Considerations

Cron carries no JWT. Per-iteration TenantContext.scope(tenant.id, tenant.countryCode) is the only tenant-source-of-truth. actor_type=SYSTEM, actor_id="ingestion-cron" on all audit emissions (architecture §4.1, §7.4). MinIO key derivation MUST use the iteration variable only — never a value from file content (architecture §7.6 A4). Both DB profiles work because the only DB writes here are audit-outbox rows on the platform DB (architecture §4.2, D12).

Audit & Logging

  • ING_UPLOAD_RECEIVED emitted once per object picked up. actor_type=SYSTEM, actor_id=ingestion-cron, subject_type=INGESTION_IMPORT, subject_id=<UUIDv7 minted for the import> (the same UUID is reused as the eventual ingestion_import.id once ING-004 lands; persist it in a thread-local for the iteration). Details: { filename, file_size_bytes, file_hash_sha256, minio_inbox_key }. PII rules: filename is operator-controlled; do NOT hash; file content is never echoed.
  • Structured INFO log per iteration boundary (tenantId, objectsFound, durationMs).

Frontend Scope

N/A — backend only.

UI States (ALL REQUIRED)

State Behavior French Copy
Loading N/A — backend only —
Empty N/A — backend only —
Error N/A — backend only —
Offline / low network N/A — backend only —
Success N/A — backend only —

Responsive Behavior

N/A.

Interactions

N/A.

Acceptance Criteria

AC-001: Cron iterates active tenants and emits ING_UPLOAD_RECEIVED per picked file
Given two active tenants A and B in `papillon_platform.tenant`
  And a 1-KB text file dropped in `<tenantA>/inbox/foo.csv`
When `IngestionService.tick()` runs
Then exactly one `ING_UPLOAD_RECEIVED` audit row exists for tenant A with subject_type=INGESTION_IMPORT
  And zero audit rows for tenant B
  And the audit details contain `filename=foo.csv`, the streamed SHA-256, and the inbox key prefixed by tenant A's UUID
  And the file is moved to `<tenantA>/processed/<yyyy-MM-dd>__foo.csv`.

AC-002: Local profile tick endpoint exists; non-local profiles do not expose it
Given the application running with `spring.profiles.active=local`
When `POST /internal/ingestion/tick` is called
Then it returns 204 and IngestionService.tick() is invoked
  And the same request on `spring.profiles.active=staging` returns 404 (endpoint not registered).

AC-003: File over 5 MB is rejected at the hash-streaming step without ever being fully buffered
Given a 6-MB binary file in `<tenantA>/inbox/big.csv`
When IngestionService.tick() runs
Then the file is moved to `<tenantA>/rejected/<ts>__big.csv` with sidecar JSON `{"reason":"FICHIER_TROP_VOLUMINEUX"}`
  And heap usage during the run stays under 256 MB (asserted by the test profile JFR snapshot).

Compliance Rules

  • Architecture §7.4: actor_type=SYSTEM, actor_id="ingestion-cron" is mandatory on every audit emission of this slice.
  • Critical rule #1 (tenant isolation, docs/standards/critical-rules.md): MinIO key construction must derive tenantId from the iteration variable only.
  • Architecture §7.6 A4: code review must reject any prefix derived from file content.

Standards & Conventions

  • docs/standards/critical-rules.md (rules #1 tenant isolation, #3 audit catalog).
  • docs/standards/multi-tenant-model.md § cron context.
  • docs/standards/java-spring-guidelines.md § @Scheduled + @TransactionalEventListener.
  • docs/standards/api-guidelines.md § internal endpoints (profile-bound).

Testing Requirements

Unit Tests

  • IngestionInboxClientTest: key construction uses iteration-bound tenantId only; refuses keys not starting with <tenantId>/.
  • IngestionAuditEventsTest: ING_UPLOAD_RECEIVED payload shape matches spec.

Integration Tests

  • IngestionSchedulerIT (TestContainers): two-tenant fixture, drop in A only → AC-001 assertions; verify audit_log row count by tenant.
  • IngestionLocalTickEndpointIT: profile-bound endpoint visibility per AC-002.
  • IngestionLargeFileIT: AC-003 with a generated 6-MB random file.

What QA Will Validate

  • Run smoke verification command from architecture §14.3 steps 1–4 (drop, tick, verify audit row, verify processed/ key). Step 5 (DB assertions on ingestion_import) is N/A until ING-004.

Out of Scope

  • CSV/XLSX parsing and row validation (ING-003).
  • ingestion_import / ingestion_import_row table writes, SHA-256 dedup against existing imports, PARSE_FAILED / DUPLICATE / COMMITTED status transitions (ING-004).
  • Startup recovery for stuck PICKED_UP rows (ING-005).
  • Feature flag feature.artci.phone-auth.granted gating on prod profile (ING-005).
  • Operator UI for upload or history (V0.0.3+).
  • code_agence resolution (V0.0.2+).

Definition of Done

  • [ ] All backend endpoints implemented and returning correct responses
  • [ ] All frontend pages render correctly at 360 / 768 / 1024 px (N/A — backend only)
  • [ ] All 5 UI states implemented (N/A — backend only)
  • [ ] French micro-copy matches design spec exactly (N/A — backend only)
  • [ ] All acceptance criteria pass manually
  • [ ] Unit + integration tests written and passing
  • [ ] Tenant isolation verified (tenant_id on every MinIO key + audit row; both V1 DB profiles tested — SHARED active, BYO smoke)
  • [ ] [CUSTOMER stories] payload + Slow-3G + resumability (N/A — backend only)
  • [ ] Audit log entries present for every required event (ING_UPLOAD_RECEIVED per pickup; verified in IT)
  • [ ] No silent failures on network errors (MinIO transient errors logged at WARN and the iteration moves on; file stays in inbox for next tick — architecture §5.1 stage 1)