Meta Data Customer Onboarding

Execution Strategy & Rollout Plan

Multi-Tenant Platform Migration

Executive Summary

This document outlines the strategy to migrate our multi-tenant platform from MongoDB to PostgreSQL. The migration is structured in two phases: a controlled rollout for the first five tenants, followed by a bucketed delivery model for the remaining customer base.

The approach is designed to minimize risk, preserve every customer's production configuration exactly as it exists today, and validate the new platform end-to-end before any customer is moved live. Mocking and isolated environments ensure we never disturb external partners or production systems during validation.

Migration at a Glance

1. Configuration & Operational Flow Extraction

Before any data moves, we need a complete inventory of how each tenant operates today. This is the foundation everything else depends on.

What we extract

• Tenant-level configurations — feature flags, environment-specific settings, branding, locale, and routing rules
• Workflows — business process definitions, state machines, approval chains, and trigger conditions
• Business rules — validation logic, eligibility criteria, calculation rules, and decision tables
• Integration settings — endpoints, credentials (referenced, not extracted), webhook subscriptions, and partner identifiers
• User and role mappings — RBAC definitions, team structures, and permission grants

Execution approach

We build a dedicated extraction utility that connects to each production tenant's MongoDB instance in read-only mode and produces a structured artifact per tenant — a single source of truth for that tenant's operational state. Each artifact is versioned, signed, and stored in a dedicated configuration repository. Sensitive values are tokenized; credentials are never copied, only referenced.

Deliverable

A per-tenant configuration manifest (one per tenant, five for Phase 1) that downstream teams use as the authoritative input for setting up the PostgreSQL environment. Any configuration drift between this manifest and what is actually configured in INTQA becomes an immediate flagged item.

2. Schema Mapping: MongoDB to PostgreSQL

The data models are fundamentally different. MongoDB's document model accommodates nested structures and flexible schemas; PostgreSQL is relational with strict typing. The mapping is not mechanical — it requires deliberate decisions about normalization, indexing, and how to handle existing flexibility.

Mapping principles

• Normalize where data is queried relationally — break embedded documents into related tables when joins are common
• Preserve as JSONB where data is genuinely schema-flexible or rarely queried by inner fields — PostgreSQL JSONB handles this well
• Establish primary key strategy — typically migrate Mongo ObjectIds to UUID columns, with the original ID preserved for traceability
• Define index parity — every query pattern that performs in MongoDB must have a corresponding index in PostgreSQL
• Handle nulls, defaults, and enums explicitly — MongoDB's permissiveness must become PostgreSQL's strictness without breaking existing records

Deliverable

A schema mapping specification, maintained as code, that defines for every MongoDB collection: the target PostgreSQL table(s), field-by-field type translation, normalization decisions, constraint rules, and index definitions. This document is reviewed and signed off by engineering before any transformation code is written.

3. Prod-to-Lower-Environment Sync Framework

Lower environments must reflect production configuration to make validation meaningful. Without this, teams test against a version of reality that does not match what customers actually use.

What gets synced

• Configurations, workflows, rules, and other operational settings — the same artifacts produced by the extraction step
• Reference data — lookup tables, code lists, and master data
• Selectively masked transactional data — only where required for realistic scenario coverage, with PII redacted

Execution approach

A sync framework is built as a one-way pipeline from production to lower environments. It runs on demand and on a schedule, with three modes:

• Full sync — used when standing up a fresh environment or after major configuration changes
• Delta sync — applies only changes since the last sync, used routinely to keep environments current
• Selective sync — operator chooses specific tenants or specific artifact types to refresh

All sync operations are audited, reversible (via snapshot before sync), and gated by approval for non-routine modes. PII handling rules are enforced at the framework level, not left to individual scripts.

Deliverable

A sync service with a simple operator interface, scheduled jobs for routine delta sync, and a full audit log of every sync operation. Lower environments converge to production within a defined SLA.

4. Data Validation Framework

Migrated data must be provably equivalent to source data. This is non-negotiable — financial, regulatory, and operational integrity all depend on it.

Validation layers

Execution approach

Validation runs as part of the migration pipeline, not as a separate afterthought. A migration is not considered complete for a tenant until all five validation layers pass. Discrepancies are logged with full context, classified by severity, and routed to the responsible team.

Deliverable

A validation report per tenant, produced automatically at the end of each migration run. The report is the formal artifact that signs off data correctness before a tenant proceeds to use-case validation.

5. End-to-End Use Case Validation Framework

Data correctness is necessary but not sufficient. The system must behave correctly under real-world operational flows. This framework validates that customer journeys work end-to-end on PostgreSQL.

What gets validated

• Critical customer journeys — every workflow that a tenant uses in production
• Cross-system flows — operations that touch multiple services or external partners
• Edge cases and known production scenarios — including historical incidents and complex configurations
• Performance baselines — response times under representative load

Execution approach

Each tenant's team builds a use-case catalogue from their production operational flows. These become automated end-to-end test suites that run in INTQA against the migrated environment. The same suites run pre-migration against MongoDB and post-migration against PostgreSQL — pass/fail parity is the gate.

Test suites are tenant-aware: each suite runs against its specific tenant configuration, ensuring that customer-specific behaviour is preserved, not just the platform default.

Deliverable

A use-case catalogue and automated test suite per tenant, with a pass/fail dashboard that leadership can review. A tenant does not move to production cutover until their suite is green and matches MongoDB behaviour.

6. Mocking Framework for External Integrations

During migration validation, we cannot call production third-party systems — doing so would create real-world side effects (payments, notifications, partner submissions). At the same time, we must validate that our integration code works correctly against the new database.

Approach

• A dedicated mocking service replicates the behaviour of every external integration our platform depends on
• Mocks are configured per scenario — happy path, error responses, timeout, partial success, etc.
• All outbound calls from INTQA are routed to the mocking layer; no production partner is ever called from a non-production environment
• Mock behaviour is captured from real production interactions (anonymized) so it reflects actual partner contracts, not idealized assumptions

Deliverable

A mocking service with a catalogue of scenarios for every integration. Teams can switch scenarios per test run to validate both happy-path and failure-mode behavior without external impact.

7. Integration Payload Validation with Mocking

Beyond mocking responses, we validate that our system sends the correct payloads and handles responses correctly — both sides of the wire.

What gets validated

• Request payloads — schema, required fields, data types, business field correctness
• Response handling — successful parsing, correct mapping to internal state, error handling
• Contract conformance — payloads match the published contract with each partner
• Behavior under partner failure modes — timeouts, 5xx errors, malformed responses

Execution approach

The mocking framework records every request/response pair during validation runs. These are diffed against a baseline captured from MongoDB production behavior. Any deviation in payload structure or values is flagged. This catches subtle issues where the migration changes a field's representation in a way that breaks downstream contracts.

Deliverable

A payload validation report per integration per tenant, with diffs against the production baseline. Zero unexplained deviations is the bar for cutover.

8. Phase 1 — Clean-State INTQA for First 5 Tenants

The first five tenants are migrated through dedicated, isolated INTQA environments — one per tenant — set up from a clean slate. Each tenant's team owns its environment and is responsible for configuring it to mirror production exactly as-is.

Setup model

Why clean-state matters

Starting from a clean state — rather than copying an existing test environment — eliminates inherited drift. Every configuration in INTQA exists because it was deliberately put there based on the production manifest. This makes drift detection meaningful and gives us a true representation of what each customer experiences.

Phase 1 success criteria

• All five tenant environments configured and signed off by their respective teams
• Data validation framework reports green for all five tenants
• End-to-end use case suites pass at parity with MongoDB
• Integration payload validation shows zero unexplained deviations
• Tenant team sign-off — the team that owns the customer relationship confirms readiness

9. Phase 2 — Customer Bucketing & Bucketed Go-Live

After the first five tenants are live, we shift from per-tenant treatment to bucketed delivery. Migrating the remaining customers one at a time would be too slow; treating them all the same would be too risky. Bucketing balances both.

How buckets are defined

Customers are grouped into buckets based on the configurations, workflows, rules, and other settings extracted in Section 1. Customers with similar operational profiles share a bucket, because they share migration risk and validation needs.

Bucketed delivery

• Each bucket gets a tailored migration playbook derived from Phase 1 learnings but adapted to the bucket's profile
• Within a bucket, customers are migrated in parallel where capacity allows, sequentially where risk requires it
• Validation thresholds are bucket-specific — high-complexity buckets get more rigorous gates
• Each bucket is fully delivered to live before the next bucket begins, unless explicitly parallelized with leadership approval

Bucket delivery cadence

After each bucket goes live, we hold a structured retrospective. Findings feed into the next bucket's playbook. This means the migration gets faster and lower-risk with each bucket, not just more numerous.

Deliverable

A bucket assignment for every remaining customer, a tailored playbook per bucket, and a sequenced delivery plan with milestones leadership can track. Each bucket completion is a clean checkpoint with a customer count moved, a risk summary, and lessons applied to the next.

Summary of Deliverables

HLD — Prod-to-Lower Environment Sync Framework

High-Level Design · Workstream 3 of 9 · MongoDB → Aurora PostgreSQL Migration Companion document: LLD · Source: metadata-documents/MongoDB_to_PostgreSQL_v2_WS3_Sync_Framework.docx

1. ⭐ Executive Summary

In one sentence: A two-hop, refNum-scoped, one-way pipeline that keeps lower environments looking like production — Hop 1 continuously refreshes Mongo Lower from Mongo Prod (automatic), and Hop 2 transforms Mongo Lower into Postgres Lower on operator demand by invoking the existing migration ingest pipeline (manual, gated).

Owner: DevOps / Platform Team | Phase: Phase 1 (first 5 customers) and Phase 2 (all remaining) | Workflow gates supported: Steps 1, 7, 8 of the nine-step workflow

What we get when it's done

• Hop 1 (continuous, automatic): Lower-env Mongo (INTQA, Staging, Dev) converges to production Mongo on a scheduled cadence, per refNum.
• Hop 2 (deliberate, gated): Operator-triggered, Full-only re-ingest from Mongo Lower into Postgres Lower (tenant_{refNum} schemas) using the same migration ingest pipeline that runs at cutover — strongest possible dogfooding.
• Pre-flight freshness check blocks Hop 2 against a stale Mongo Lower. Override requires senior approval and is audited.
• Snapshot before write on both hops; one-operation rollback.
• Masking is non-bypassable and driven by MDS field classifications — PII never reaches a lower environment.
• Auto-validation runs the Data Validation Framework immediately after Hop 2 completes.
• End-to-end auditability: every chain run is queryable by a single correlation ID linking Hop 1 → Hop 2 → ingest → validation.
• Refresh discipline post-cutover: once a customer cuts over, Hop 1 transitions to Aurora Prod → Aurora Lower; Hop 2 becomes unnecessary because source and target are both Aurora.

2. Scope

3. ⭐ System Context Diagram (C4 L1)

flowchart LR
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef actor fill:#fff3e0,stroke:#ef6c00,color:#e65100
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    Sched([Delta Scheduler]):::actor
    Op([Operator]):::actor
    Snr([Senior Approver]):::actor
    Sec([Security / Compliance]):::actor

    subgraph ProdZone[" Production zone (read-only egress) "]
        MProd[(MongoDB Prod<br/>read replica)]:::prod
        AProd[(Aurora Prod<br/>post-cutover only)]:::prod
        MDS[MDS — field classifications]:::external
    end

    Sync["Prod-to-Lower<br/>Sync Framework<br/>(Hop 1 + Hop 2)"]:::internal
    Preflight[Hop 2 Pre-flight<br/>Freshness Check]:::gate

    subgraph LowerZone[" Lower environments (per refNum) "]
        MLower[(Mongo Lower<br/>INTQA · Staging · Dev)]:::lower
        PLower[(Postgres Lower<br/>tenant_refNum)]:::lower
    end

    MigPipe[Migration Ingest Pipeline<br/>JOLT → DAAL]:::external
    Val[Data Validation Framework]:::external
    Audit[(Audit Sink)]:::internal

    Sched -- "Hop 1 Delta (auto)" --> Sync
    Op -- "Hop 1 Full / Selective (with approval)" --> Sync
    Op -- "Hop 2 trigger (Full, with approval)" --> Sync
    Snr -- "preflight override (if stale)" --> Sync

    Sync -- "reads (replica only)" --> MProd
    Sync -- "reads (replica only, post-cutover)" --> AProd
    Sync -- "reads classifications" --> MDS
    Sync -- "Hop 1: writes one-way" --> MLower

    Sync --> Preflight
    Preflight -- "pass" --> MigPipe
    Preflight -. "fail → 409" .-> Op
    MLower -- "Hop 2 source" --> MigPipe
    MigPipe -- "writes" --> PLower
    Sync -- "Hop 2 auto-trigger" --> Val
    Val -. "reads" .-> PLower

    Sync -- "records every operation" --> Audit
    Sec -- "reviews audit + approvals + overrides" --> Audit

    style ProdZone stroke-dasharray: 5 5,stroke:#00695c
    style LowerZone stroke-dasharray: 5 5,stroke:#1565c0

The Sync Framework owns both hops. Hop 1 writes directly to Mongo Lower. Hop 2 invokes the migration ingest pipeline — the same pipeline that runs at cutover — to populate Postgres Lower from Mongo Lower. Production is never a writable target from any lower environment.

4. High-Level Architecture (C4 L2 — Container)

flowchart TB
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef actor fill:#fff3e0,stroke:#ef6c00,color:#e65100
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    Op([Operator UI / CLI]):::actor
    Sched([Delta Scheduler]):::actor

    subgraph Sync[" Sync Framework "]
        direction TB
        API[Sync API / Orchestrator]:::internal
        Approval[Approval Workflow<br/>Hop 1 Full/Selective + Hop 2]:::internal
        Tagger[Correlation ID Tagger]:::internal
        Preflight[Hop 2 Pre-flight<br/>Freshness Validator]:::gate

        subgraph Hop1[" Hop 1 Pipeline "]
            direction TB
            Router[Mode Router<br/>Full · Delta · Selective]:::internal
            Reader[Production Reader<br/>replica-only, throttled]:::internal
            Mask[Masking Engine<br/>MDS-driven]:::internal
            Snap1["Snapshot Manager<br/>(Mongo Lower)"]:::internal
            Writer[Target Writer<br/>refNum-scoped]:::internal
        end

        subgraph Hop2[" Hop 2 Surface "]
            direction TB
            Snap2["Snapshot Manager<br/>(Postgres Lower)"]:::internal
            Adapter[Migration Pipeline<br/>Adapter]:::internal
            ValHook[Validation Webhook]:::internal
        end

        CDC[(CDC Resume<br/>Token Store)]:::store
        SnapStore[(Snapshot Store)]:::store
        AuditLog[(Audit Log)]:::store
    end

    ProdDB[(Production<br/>read replica)]:::prod
    MDS[MDS]:::internal
    MLower[(Mongo Lower<br/>tenant_refNum)]:::lower
    PLower[(Postgres Lower<br/>tenant_refNum)]:::lower
    MigPipe[Migration Ingest<br/>Pipeline]:::external
    Val[Data Validation<br/>Framework]:::external

    Op --> API
    Sched --> API
    API --> Tagger
    Tagger --> Approval
    Approval --> Router
    Approval --> Preflight

    Router --> Reader --> Mask --> Snap1 --> Writer --> MLower
    Reader --> ProdDB
    Mask --> MDS
    Reader -. resume tokens .-> CDC
    Snap1 -. snapshots .-> SnapStore

    Preflight -- "pass" --> Snap2 --> Adapter
    Snap2 -. snapshots .-> SnapStore
    Adapter -- "invoke (source=Mongo Lower, target=tenant_refNum, full)" --> MigPipe
    MLower --> MigPipe --> PLower
    Adapter --> ValHook --> Val

    Writer --> AuditLog
    Adapter --> AuditLog
    Preflight --> AuditLog

Hop 1 and Hop 2 share one control plane (API, Tagger, Approval, Snapshot Manager, Audit Log) but have distinct execution pipelines. Hop 2 does not contain its own ingest logic — it invokes the existing migration ingest pipeline.

5. Key Components

6. Core Principles

• One-way only. Data flows production → lower. Never the reverse. Enforced at the infrastructure layer, not by policy.
• refNum-scoped. Every operation in either hop touches one customer's data at a time. Cross-customer contamination is impossible by design.
• Hop 1 is continuous; Hop 2 is deliberate. Scheduled Delta keeps Mongo Lower fresh without operator attention. Hop 2 is always explicit, approved, gated, and snapshotted.
• Reuse the migration pipeline for Hop 2. The pipeline that runs at cutover is the pipeline that refreshes Postgres Lower — no parallel implementation, strongest dogfooding signal.
• Reversible. Every sync (both hops) takes a snapshot first. The snapshot is the restore point.
• PII handling at the framework level. Masking rules come from MDS classifications, not from individual scripts. Hop 2 inherits the already-masked data from Mongo Lower; it never re-touches production.
• Validation-aware. Every Hop 2 emits a correlation ID and auto-triggers the Data Validation Framework. The chain is auditable end-to-end by that ID.
• Audited. Every sync — both hops, including pre-flight overrides — is logged with actor, refNum, scope, snapshot id, and outcome.

7. Interactions With Other Workstreams

8. Non-Functional Requirements

9. Sync Operations

The framework exposes four operations, all governed by the same control plane.

10. ⭐ Chained Sync — Lifecycle & Promises

The two hops cooperate as one strategy. This section captures how the strategy changes through the migration lifecycle and the user-facing guarantees it offers.

10.1 How the chain changes over the migration lifecycle

10.2 What the chain promises

• refNum-scoped end to end. A chain run only affects one customer's data in Mongo Lower and tenant_{refNum} in Postgres Lower.
• No PII propagation. Hop 1 masks at the framework level; Hop 2 inherits the masked data. Masking is never re-applied or relaxed downstream.
• No destructive surprises. Hop 2 cannot truncate Postgres Lower without first snapshotting it and recording operator approval.
• No stale ingests. Pre-flight check blocks Hop 2 against a stale Mongo Lower unless senior approval is recorded.
• Auto-validated. Every Hop 2 success triggers the Data Validation Framework; validation outcomes are pinned to the chain.
• Auditable end-to-end. One correlation ID joins Hop 1 → Hop 2 → ingest → validation. Available via GET /chains/{correlation_id}.

Operating-model details and component-level mechanics are in the LLD §3.6, §4, §5.4–5.5.

11. ⭐ Risks & Mitigations

12. ⭐ Success Criteria

• Sync policy document signed off by DevOps, Security, and Compliance.
• Sync service built, deployed, and accessible to authorized operators in INTQA, Staging, Development.
• Masking driven by MDS field classifications — verified no PII reaches lower environments.
• Hop 1: scheduled Delta sync running successfully on cadence; CDC resume tokens persist between runs.
• Hop 1: snapshot capture and restore proven on a real environment.
• Hop 2: Migration Pipeline Adapter invokes the cutover pipeline against source=mongo-lower-{env} for every Phase 1 customer.
• Hop 2: pre-flight freshness check blocks stale-source runs; override path enforces senior approval and pages security.
• Hop 2: snapshot capture and restore of Postgres Lower proven on a real environment.
• Auto-trigger of Data Validation Framework after Hop 2 produces a validation report linked by correlation ID.
• Audit log captures every sync operation (both hops) with full traceability (who, what, when, refNum, scope, snapshot id, correlation id).
• Operators trained; runbooks published for Hop 1 Full / Selective / Restore and Hop 2 Trigger / Override / Restore.
• Lower environments measurably converge to production state within the defined freshness expectation.
• GET /chains/{correlation_id} returns the end-to-end view for every Phase 1 customer's chain runs.

13. Glossary Reference

See _shared/glossary.md. Key terms used in this document: refNum, MDS, CDC, Sync mode, Snapshot, Resume token, Hop 1, Hop 2, Correlation ID, Pre-flight freshness check.

HLD — Data Validation Framework

High-Level Design · Workstream 4 of 9 · MongoDB → Aurora PostgreSQL Migration Companion document: LLD · Source: metadata-documents/MongoDB_to_PostgreSQL_v2_WS4_Data_Validation.docx

1. ⭐ Executive Summary

In one sentence: Run the playbook's reconciliation rules — row count parity, FK integrity, required field coverage, sampled deep-diff — automatically, on every entity, for every customer, as a gate the migration cannot pass without.

Owner: QA / Data Validation Team | Phase: Phase 1 (proves the framework) and Phase 2 (runs unchanged) | Workflow gates supported: Steps 3, 7, 8, 9 of the nine-step workflow

What we get when it's done

• Every customer ingestion produces a signed, per-entity validation report.
• The four playbook reconciliation rules run automatically — no entity is hand-checked.
• Per-entity business assertions catch domain-specific errors that structural checks miss.
• Discrepancies are classified by severity; Critical issues block workflow advancement.
• Dead Letter Queue inventory is a first-class output — a clean DLQ is a precondition for sign-off.
• The framework is verified against seeded errors before any customer relies on it.
• Auto-triggered after Sync Framework Hop 2 — every chained sync run automatically produces a validation report, tagged with the chain's correlation_id for end-to-end traceability.

2. Scope

3. ⭐ System Context Diagram (C4 L1)

flowchart LR
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef actor fill:#fff3e0,stroke:#ef6c00,color:#e65100
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    QA([QA Team]):::actor
    DomainOwner([Domain Owners]):::actor
    Leadership([Leadership]):::actor

    MongoSrc[(MongoDB Source)]:::prod
    Ingest[Ingest Pipeline<br/>JOLT → DAAL]:::external
    Aurora[(Aurora Target<br/>tenant_refNum)]:::lower

    VF["Data Validation<br/>Framework"]:::internal

    MDS[MDS — entity decls<br/>+ assertions]:::external
    History[(metadata.<br/>ingest_run_history)]:::internal
    DLQ[(Dead Letter Queue)]:::internal
    Reports[(Per-customer<br/>per-entity reports)]:::internal
    Dashboard[Leadership Dashboard]:::internal

    MongoSrc --> Ingest --> Aurora
    Ingest -.->|run record| History
    Ingest -.->|failed records + context| DLQ

    VF -- "reads counts/keys/samples" --> MongoSrc
    VF -- "reads counts/keys/samples" --> Aurora
    VF -- "reads rules + assertions" --> MDS
    VF -- "reads run records + DLQ" --> History
    VF -- "reads DLQ inventory" --> DLQ
    VF -- "writes results" --> History
    VF -- "writes report" --> Reports

    Reports --> Dashboard --> Leadership
    Reports --> QA
    DomainOwner --> MDS

The Data Validation Framework attaches to the existing ingest pipeline. It is not a separate, opt-in step — every ingest produces a validation result alongside the run record.

4. High-Level Architecture (C4 L2 — Container)

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    Ingest[Ingest Pipeline]:::external
    MDS[MDS]:::external

    subgraph VF[" Data Validation Framework "]
        direction TB
        Engine[Rule Engine<br/>4 defaults + assertion executor]:::internal
        DLQI[DLQ Inspector]:::internal
        Classifier[Severity Classifier<br/>+ Router]:::internal
        Reporter[Reporting Service]:::internal
        Gate[Gate Decision API]:::gate

        Rules[(Rule + Assertion<br/>Library)]:::store
        Results[(Validation Results<br/>in metadata.ingest_run_history)]:::store
        ReportStore[(Report Store)]:::store
    end

    Dashboard[Leadership Dashboard]:::internal

    Ingest -- "ingest run event" --> Engine
    MDS -- "entity decls + assertions" --> Rules
    Rules --> Engine
    Engine --> Classifier
    DLQI -- "DLQ inventory" --> Engine
    Classifier --> Reporter
    Reporter --> Results
    Reporter --> ReportStore
    Classifier --> Gate
    Gate -- "block / allow" --> Ingest
    ReportStore --> Dashboard

The Gate Decision API is the integration point that prevents workflow advancement when Critical findings exist. Override is possible only with explicit, recorded approval.

5. Key Components

6. Core Principles

• Automated, not manual. Every check runs in code. Human review interprets reports; it does not perform validation.
• Built into the migration pipeline. Validation runs as part of every ingest, not as a separate step that can be skipped.
• Per-customer, per-entity reporting. Aggregate results hide entity-specific problems. The per-entity report is the artifact that signs off.
• Severity-classified discrepancies. Not every mismatch is a blocker; classification keeps focus on what matters.
• Reproducible. Re-running validation on the same dataset produces the same result. Auditable, trustworthy.
• The framework itself is verified. Seeded-error tests prove the engine catches what it should before any real customer relies on it.

7. Interactions With Other Workstreams

8. Non-Functional Requirements

9. Where Validation Fits in the Nine-Step Workflow

10. ⭐ Risks & Mitigations

11. ⭐ Success Criteria

• Default reconciliation rules configured per playbook defaults (0% row count tolerance, 0 orphans, ≥99.9% coverage, 0 diffs in sample).
• Entity-specific assertions authored for every Phase 1 entity.
• Validation framework integrated into the ingest pipeline; results land in metadata.ingest_run_history.
• Severity classification and routing rules configured and verified.
• Framework verified against seeded errors — every seeded error caught and correctly classified.
• Per-customer, per-entity validation reports produced automatically and used as workflow gates.
• DLQ records resolved for every Phase 1 entity before workflow advancement.
• All four default rules pass plus entity-specific assertions for each Phase 1 customer entity.

12. Glossary Reference

See _shared/glossary.md. Key terms used in this document: MDS, DAAL, JOLT, _lookup, auto-injected FK, refNum, DLQ, Reconciliation rule, Assertion, Severity class.

HLD — End-to-End Validation Framework

High-Level Design · Workstreams 5 + 6 + 7 of 9 · MongoDB → Aurora PostgreSQL Migration Companion document: LLD · Sources: MongoDB_to_PostgreSQL_v2_WS5_Use_Case_Validation.docx, WS6_Mocking_Framework.docx, WS7_Payload_Validation.docx

1. ⭐ Executive Summary

In one sentence: Prove that everything a customer does today still works after migration — workflows, screens, integrations, and the bytes we send to partners — by running automated, customer-specific suites against the migrated system, with mocks for every external partner and byte-level diffs against production baselines.

Owner: QA / Data Validation Team (Use Case + Payload) and Platform Eng (Mocking) | Phase: Phase 1 (proves the framework) and Phase 2 (runs unchanged) | Workflow gates supported: Steps 7, 8, 9 of the nine-step workflow

What we get when it's done

• Every Phase 1 customer has a signed, automated use-case suite that proves their business workflows produce the same outcomes on Aurora as on the existing system.
• Every external partner has a faithful mock; no INTQA call can ever reach a real partner.
• Every outbound integration payload is compared byte-by-byte against a production baseline; blocking differences cannot pass through to cutover.
• Performance regressions are caught against measured baselines, not assumed thresholds.
• A leadership-visible pass/fail dashboard shows readiness per customer.
• Daily smoke tests run throughout the 2-week hypercare window; sign-off requires zero P1/P2 for 7 consecutive days.

2. Why three workstreams in one framework

The three source workstreams answer three different questions, but they only deliver value together. Data Validation (WS4) confirms the data is correct at rest; this framework confirms the system behaves correctly when used.

We model them as one framework with three internal sub-systems, sharing one orchestrator, one dashboard, and one evidence package.

3. Scope

4. ⭐ System Context Diagram (C4 L1)

flowchart LR
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef actor fill:#fff3e0,stroke:#ef6c00,color:#e65100
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    CustomerTeam([Customer / Tenant Team]):::actor
    QA([QA Team]):::actor
    IntOwner([Integration Owner]):::actor
    Leadership([Leadership]):::actor

    subgraph ProdZone[" Production "]
        ProdSys[Existing MongoDB Platform]:::prod
        ProdPartners[Real External Partners]:::external
    end

    subgraph INTQAZone[" INTQA (migrated) "]
        Migrated[Aurora-backed Platform]:::lower
    end

    E2E["End-to-End<br/>Validation Framework<br/>(Use Cases · Mocks · Payload Diff)"]:::internal

    Baseline[(Parity + Perf<br/>Baselines)]:::internal
    Diff[(Payload Diff<br/>Reports)]:::internal
    Dashboard[Pass/Fail Dashboard]:::internal

    CustomerTeam -- "authors use cases" --> E2E
    IntOwner -- "owns mocks + classification" --> E2E
    QA -- "operates suites" --> E2E

    ProdSys -- "captured baselines" --> Baseline
    ProdSys -. "real traffic for capture only" .-> ProdPartners

    E2E -- "runs scenarios" --> Migrated
    Migrated -. "outbound integration calls" .-> E2E
    E2E -- "reads/writes" --> Baseline
    E2E -- "produces" --> Diff

    Diff --> Dashboard --> Leadership

    style ProdZone stroke-dasharray: 5 5,stroke:#00695c
    style INTQAZone stroke-dasharray: 5 5,stroke:#1565c0

No outbound traffic from INTQA ever reaches a real partner. Capture happens in production for read-only baselining; replay happens in INTQA against the mock layer inside this framework.

5. High-Level Architecture (C4 L2 — Container)

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40

    Migrated["Aurora-backed Platform<br/>(INTQA)"]:::lower
    ProdCap["Production traffic<br/>capture (read-only)"]:::prod

    subgraph E2E[" End-to-End Validation Framework "]
        direction TB
        Orchestrator[Use Case Orchestrator]:::internal
        Catalog[(Use Case Catalog<br/>per refNum)]:::store
        ParityBase[(Parity Baseline<br/>Store)]:::store
        PerfBase[(Performance Baseline<br/>Store)]:::store

        Mock[Mock Service<br/>+ Scenario Manager]:::internal
        ScenStore[(Mock Scenario Catalog)]:::store

        Capture[Outbound Payload<br/>Capture]:::internal
        Diff[Diff Engine<br/>+ Classification Rules]:::internal
        BaselineLib[(Production Payload<br/>Baseline Library)]:::store
        ClassRules[(Classification<br/>Rule Book)]:::store

        Reporter[Reporting + Dashboard API]:::internal
        Dashboard[Pass/Fail Dashboard]:::internal

        Catalog --> Orchestrator
        ParityBase --> Orchestrator
        PerfBase --> Orchestrator

        Orchestrator -- "run scenarios" --> Migrated
        Migrated -. "outbound call" .-> Capture
        Capture --> Mock
        Mock --> Migrated
        ScenStore --> Mock

        Capture --> Diff
        BaselineLib --> Diff
        ClassRules --> Diff

        Orchestrator --> Reporter
        Diff --> Reporter
        Reporter --> Dashboard
    end

    ProdCap --> BaselineLib

Outbound calls from the migrated platform are intercepted by the Capture component, routed to the Mock for a response, and simultaneously diffed against the production baseline. One pipeline produces both functional and payload evidence.

6. Key Components

7. Core Principles

• Customer-aware (per refNum). Each customer's test suite runs against their specific configuration; the same workflow can produce different correct results for different customers.
• Automated and repeatable. Every scenario runs in code. Re-running produces identical setup and predictable results.
• Pass/fail parity, not "looks fine". Each scenario runs on both the existing and migrated systems; outcomes must match. Parity is the gate.
• Owned by the tenant teams. The team that owns the customer relationship owns the catalog and is accountable for completeness.
• Visible to leadership. Test results are not buried in engineering tools; a clear dashboard surfaces status to anyone who needs it.
• Zero external impact during validation. No INTQA call reaches a real partner — network-level controls, not honor system.
• Byte-level comparison, judgment-level classification. The diff engine looks at every byte but does not panic at every difference; classification rules separate signal from noise.
• Production behavior is the baseline. Not documentation; not spec — what production actually does.

8. Interactions With Other Workstreams

9. Non-Functional Requirements

10. ⭐ Risks & Mitigations

11. ⭐ Success Criteria

• Use-case catalog complete and signed off by each Phase 1 customer team.
• Business-process smoke tests defined per entity (playbook §4.4).
• Parity baseline captured from the existing system for every scenario in scope.
• Automated test suites built, customer-aware, integrated into the validation pipeline.
• Performance baselines defined with agreed variance thresholds.
• Complete integration catalog signed off by all application teams.
• Mocking service deployed; mock implementation complete for every integration; scenario coverage verified.
• INTQA network routing verified — no path from INTQA to real partner systems exists.
• Production payload baseline captured per integration per customer.
• Classification rule book signed off by integration owners.
• Diff engine verified against known-different payload pairs.
• Zero unresolved Blocking payload differences for any customer before cutover.
• Pass/fail dashboard live and used by leadership and engineering.
• Hypercare completed for each customer with zero P1/P2 for 7 consecutive days.
• Customer sign-off recorded for each Phase 1 customer.

12. Glossary Reference

See _shared/glossary.md. Key terms used in this document: refNum, DAAL, MDS, Use case, Parity baseline, Performance baseline, Scenario, Capture, Diff classification, Hypercare.

LLD — Prod-to-Lower Environment Sync Framework

Low-Level Design · Workstream 3 of 9 · MongoDB → Aurora PostgreSQL Migration Companion document: HLD · Source: metadata-documents/MongoDB_to_PostgreSQL_v2_WS3_Sync_Framework.docx

1. ⭐ Executive Summary

In one sentence: A control-plane service that owns two hops — Hop 1 reads production replicas, applies MDS-driven masking, and writes per-refNum to Mongo Lower in one of three modes (Full / Delta / Selective); Hop 2 is an operator-triggered, snapshotted, pre-flight-gated invocation of the existing migration ingest pipeline (source=mongo-lower-{env}) that re-populates tenant_{refNum} in Postgres Lower and auto-triggers the Data Validation Framework.

If you only read one section, read this.

• What we build:
  • Hop 1: Sync API + Mode Router + Production Reader (replica-only, throttled) + Masking Engine (MDS-driven) + Snapshot Manager + Target Writer.
  • Hop 2 surface: Pre-flight Freshness Validator + Snapshot Manager (Postgres Lower) + Migration Pipeline Adapter + Validation Webhook.
  • Shared control plane: Sync API/Orchestrator + Correlation ID Tagger + Approval Workflow + Audit Log.
• Key design decisions:
  • Hop 2 reuses the cutover migration pipeline — no parallel ingestion code path.
  • Hop 2 is manual, Full-only, snapshotted, and pre-flight-gated against Mongo Lower freshness.
  • Pre-flight failure returns HTTP 409; override requires senior-approver and is page-worthy for security.
  • Correlation ID is the join key across Hop 1 → Hop 2 → ingest → validation. Persisted on SYNC_RUN.correlation_id (Sync-owned), INGEST_RUN.correlation_id (migration pipeline / DV-owned, physical table metadata.ingest_run_history), and VALIDATION_RUN.correlation_id (DV-owned).
  • Masking is enforced inside the framework on Hop 1; Hop 2 inherits the already-masked data and never touches production.
  • Snapshot before every write is non-negotiable on both hops.
• Compliance posture: No PII in lower environments. Per-customer refNum scoping makes cross-tenant contamination structurally impossible.
• Open items at v0.2: per-env default for pre-flight staleness window; replica-lag thresholds per environment; snapshot retention policy per environment; SIEM mirror destination.

2. Reference Architecture (recap from HLD)

flowchart LR
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    Prod[(Production<br/>read replica)]:::prod --> Hop1[Sync Framework<br/>Hop 1]:::internal --> MLower[(Mongo Lower)]:::lower
    MDS[MDS]:::internal --> Hop1
    Op([Operator]) --> Hop2[Sync Framework<br/>Hop 2 Surface]:::internal
    Hop2 --> Pre[Pre-flight Check]:::gate --> MigPipe[Migration Ingest<br/>Pipeline]:::external
    MLower --> MigPipe --> PLower[(Postgres Lower)]:::lower
    Hop2 -- "auto-trigger" --> Val[Data Validation]:::external
    Hop1 --> Audit[(Audit Log)]:::internal
    Hop2 --> Audit

Full HLD diagrams in the HLD §3 and §4. The LLD elaborates each container.

3. Component Designs (C4 L3)

3.1 Sync API / Orchestrator (shared control plane)

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92

    REST[REST API<br/>POST /syncs · GET /chains/...]:::internal --> Validator[Request Validator]:::internal
    Validator --> Authz[AuthZ<br/>scope = refNum + mode + chain_hop]:::internal
    Authz --> Tagger[Correlation ID Tagger<br/>Hop 2 only]:::internal
    Tagger --> Approval[Approval Gate<br/>Hop 1 Full/Selective + every Hop 2]:::internal
    Approval --> Enqueue[Run Enqueuer]:::internal
    Enqueue --> Queue[(Run Queue)]:::store
    Queue --> Dispatcher[Dispatcher<br/>per-refNum, per-target mutex]:::internal
    Dispatcher --> RouteHop{chain_hop?}
    RouteHop -- "1" --> Router1[Mode Router]:::internal
    RouteHop -- "2" --> Preflight[Pre-flight Validator]:::internal

3.2 Mode Router (Hop 1)

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121

    In[Hop 1 run from Queue]:::internal --> Decide{Mode?}
    Decide -- Full --> Full[Full-Sync Handler]:::internal
    Decide -- Delta --> Delta[Delta-Sync Handler]:::internal
    Decide -- Selective --> Sel[Selective-Sync Handler]:::internal

    Full --> Plan[Scope Planner<br/>categories + entities]:::internal
    Delta --> Plan
    Sel --> Plan
    Plan --> Reader[Production Reader]:::internal

3.3 Production Reader (Hop 1)

flowchart LR
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121

    Plan[Read Plan]:::internal --> Reader[Reader Pool]:::internal
    Reader --> Throttle[Replica Lag Monitor<br/>+ Throttle]:::internal
    Throttle --> Replica[(Prod Read Replica)]:::prod
    Replica --> Throttle --> Buffer[Bounded Buffer]:::internal
    Buffer --> Next[→ Masking]:::internal

• Reads only from a designated read replica; no direct primary access. Enforced by connection string + IAM.
• Replica-lag monitor compares replica heartbeat to primary; if lag exceeds the configured threshold, the reader pauses and resumes when the replica catches up.
• A bounded buffer downstream prevents memory blowup if Masking or Writer is slower than Reader.

3.4 Masking Engine (Hop 1)

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    In[Record from Reader]:::internal --> Lookup[MDS Classification Lookup]:::internal
    MDS[MDS]:::external --> Lookup
    Lookup --> Apply[Field-by-Field Masker]:::internal
    Apply --> Check[Policy Check<br/>fail-closed if classification missing]:::gate
    Check --> Out[→ Snapshot Manager]:::internal

• Classifications cached locally with a short TTL; cache miss forces an MDS roundtrip.
• Fail-closed: if a field's classification cannot be resolved, the record is rejected and the run fails. No "default to public".
• Masking strategies per classification: PII → deterministic pseudonymization (so referential integrity holds across rows), sensitive → null or fixed token, public → pass-through.
• Strategy is centrally defined per classification; not per call site.
• Hop 2 does not re-mask. Mongo Lower is already masked; Hop 2 just transforms the masked data into the Postgres shape.

3.5 Snapshot Manager + Target Writer (Hop 1)

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1

    In[Masked record stream]:::internal --> Snap[Snapshot Manager]:::internal
    Snap --> SnapStore[(Snapshot Store<br/>per refNum, retained N days)]:::store
    Snap --> TxBatch[Transactional Batcher]:::internal
    TxBatch --> Writer[Target Writer<br/>tenant_refNum scope]:::internal
    Writer --> Lower[(Mongo Lower<br/>tenant_refNum)]:::lower
    Writer --> Verify[Post-Write Verifier<br/>row counts + spot diff]:::internal
    Verify --> Audit[(Audit Log)]:::store

• Snapshot is taken before any write to the target schema. Snapshot ID is recorded against the run.
• Target Writer scopes every statement to tenant_{refNum} — never writes outside the requested schema; enforced by a session-level role.
• Post-Write Verifier runs lightweight reconciliation (counts, spot-check) and records the result in the audit entry. Heavier validation is delegated to the Data Validation Framework.

3.6 Hop 2 Surface

The Hop 2 surface is what makes the chain real. It is small — four components — because Hop 2 reuses the migration ingest pipeline rather than re-implementing it.

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    In[Hop 2 run with correlation_id<br/>from Tagger]:::internal --> Preflight[Pre-flight Freshness<br/>Validator]:::gate
    Preflight -- "fail (409)" --> Override[Override Path<br/>senior-approver]:::gate
    Preflight -- "pass" --> Apv[Approval Gate]:::internal
    Override --> Apv
    Apv --> Snap["Snapshot Manager<br/>(Postgres Lower)"]:::internal
    Snap --> SnapStore[(Snapshot Store)]:::store
    Snap --> Adapter[Migration Pipeline Adapter]:::internal
    Adapter -- "source=mongo-lower-{env}<br/>target=tenant_refNum (Postgres)<br/>mode=Full<br/>correlation_id=..." --> MigPipe[Migration Ingest Pipeline<br/>JOLT → DAAL]:::external
    MLower[(Mongo Lower)]:::lower --> MigPipe
    MigPipe --> PLower[(Postgres Lower)]:::lower
    MigPipe -- "ingest_run_id" --> Adapter
    Adapter --> Hook[Validation Webhook]:::internal
    Hook -- "trigger with correlation_id" --> Val[Data Validation<br/>Framework]:::external
    Adapter --> Audit[(Audit Log)]:::store
    Preflight --> Audit

Pre-flight check semantics (table form):

Correlation ID model (table form):

Concurrency model:

• Per (refNum, target_env, chain_hop): at most one non-terminal run. Existing Hop 1 per-refNum mutex extended to include (target_env, chain_hop).
• Across refNums: parallel Hop 2 runs allowed up to worker-pool size. Writes isolated to each customer's tenant_{refNum} schema, so concurrency is safe.
• Hop 1 ↔ Hop 2 interlock: Hop 2 will not dispatch while a Hop 1 is Running for the same (refNum, target_env). Pre-flight surfaces this as a clear error.

4. Data Model

The diagram below shows entities owned by the Sync Framework, plus the external entities it joins against to assemble the chain view. The external entities are defined authoritatively in the Data Validation LLD (§4) and the migration pipeline; Sync only depends on the columns shown here.

erDiagram
    SYNC_RUN ||--o{ SYNC_RUN_ITEM : contains
    SYNC_RUN ||--|| SNAPSHOT : "captures before run"
    SYNC_RUN ||--o{ AUDIT_ENTRY : "emits"
    SYNC_RUN }o--|| CDC_RESUME_TOKEN : "Delta mode reads"
    SYNC_RUN }o--|| APPROVAL : "Full/Selective + Hop 2 require"
    SYNC_RUN }o--o| SYNC_RUN : "Hop 2 links to Hop 1 (linked_hop1_run_id)"
    SYNC_RUN }o--o| INGEST_RUN : "Hop 2 links to ingest (linked_ingest_run_id)"
    INGEST_RUN ||--o{ VALIDATION_RUN : "validated by (DV-owned)"
    VALIDATION_RUN ||--|| REPORT : "produces (DV-owned)"
    MDS_CLASSIFICATION_CACHE ||--o{ SYNC_RUN_ITEM : "masking applied"

    SYNC_RUN {
        uuid id PK
        text refNum
        text target_env
        smallint chain_hop "1 or 2"
        text mode "Full|Delta|Selective"
        text status "Queued|AwaitingPreflight|AwaitingApproval|PreflightFailed|Running|Succeeded|SucceededWithBlockingFindings|Failed|RolledBack|Rejected"
        text correlation_id "Hop 2 only"
        uuid linked_hop1_run_id FK "Hop 2 only"
        uuid linked_ingest_run_id FK "Hop 2 only"
        uuid linked_validation_run_id FK "Hop 2 only; populated by webhook"
        text preflight_result "Hop 2 only: pass|fail|overridden"
        text preflight_override_by "Hop 2 only"
        text preflight_override_rationale "Hop 2 only; redacted at retention"
        text requested_by
        text approved_by
        uuid snapshot_id FK
        uuid resume_token_id FK "Hop 1 Delta only"
        timestamp started_at
        timestamp completed_at
        jsonb scope "categories + filters"
    }
    SYNC_RUN_ITEM {
        uuid id PK
        uuid run_id FK
        text category "operational|lookup|transactional"
        text artifact_type
        int records_in
        int records_written
        int records_skipped
        jsonb errors
    }
    SNAPSHOT {
        uuid id PK
        text refNum
        text target_env
        text target_tech "mongo|postgres"
        timestamp taken_at
        text storage_uri
        text retention_class
    }
    CDC_RESUME_TOKEN {
        uuid id PK
        text refNum
        text source_collection
        bytea token
        timestamp updated_at
    }
    APPROVAL {
        uuid id PK
        uuid run_id FK
        text approver
        text justification
        timestamp approved_at
    }
    AUDIT_ENTRY {
        uuid id PK
        uuid run_id FK
        text correlation_id "Hop 2 only"
        timestamp at
        text actor
        text action
        jsonb details
    }
    INGEST_RUN {
        uuid id PK "external: defined in DV LLD §4"
        text refNum
        text correlation_id "stamped by Hop 2"
        timestamp started_at
        timestamp completed_at
    }
    VALIDATION_RUN {
        uuid id PK "external: defined in DV LLD §4"
        uuid ingest_run_id FK
        text correlation_id
        text trigger_source "ingest|hop2|manual"
        text status "Queued|Running|Succeeded|Failed"
    }
    REPORT {
        uuid id PK "external: defined in DV LLD §4"
        uuid validation_run_id FK
        text refNum
        text entity
        text content_hash
    }
    MDS_CLASSIFICATION_CACHE {
        text field_path PK
        text classification "PII|sensitive|public"
        timestamp fetched_at
    }

Notes on chain columns added in v0.2:

• linked_validation_run_id is populated by the Validation Webhook callback (see §5.4). Until the callback arrives, the column is NULL and chain status is Succeeded (Hop 2 ingest complete, validation pending) or Running (validation in flight).
• preflight_override_rationale is captured as a structured column so auditors can query overrides without scanning AUDIT_ENTRY.details. The same rationale is also written to AUDIT_ENTRY for the override event.

Chain end-to-end view (CHAIN_RUN_VIEW):

The view joins one Hop 2 row to its Hop 1 predecessor, the migration pipeline's INGEST_RUN, and the Data Validation Framework's VALIDATION_RUN + REPORT — all by the chain correlation_id. Table and column names below match the authoritative definitions in DV LLD §4.

-- One row per Hop 2 run, with Hop 1, ingest, and validation joined for the chain view.
SELECT
  h2.correlation_id,
  h2.refNum,
  h2.target_env,
  h2.id                       AS hop2_run_id,
  h2.status                   AS hop2_status,
  h2.started_at               AS hop2_started_at,
  h2.completed_at             AS hop2_completed_at,
  h2.preflight_result,
  h2.preflight_override_by,
  h1.id                       AS hop1_run_id,
  h1.completed_at             AS hop1_completed_at,
  ir.id                       AS ingest_run_id,
  ir.started_at               AS ingest_started_at,
  ir.completed_at             AS ingest_completed_at,
  vr.id                       AS validation_run_id,
  vr.status                   AS validation_status,
  vr.trigger_source           AS validation_trigger_source,
  rep.id                      AS validation_report_id,
  rep.content_hash            AS validation_report_hash
FROM SYNC_RUN h2
LEFT JOIN SYNC_RUN       h1  ON h1.id  = h2.linked_hop1_run_id
LEFT JOIN INGEST_RUN     ir  ON ir.id  = h2.linked_ingest_run_id
LEFT JOIN VALIDATION_RUN vr  ON vr.correlation_id = h2.correlation_id
LEFT JOIN REPORT         rep ON rep.validation_run_id = vr.id
WHERE h2.chain_hop = 2;

DLQ counts are deliberately not in this view — they live on DLQ_RECORD (owned by DV) and are queried separately via DV's GET /dlq/inventory. Joining DLQ counts here would force a per-row aggregation; the chain view is for orientation, not deep DLQ analysis.

Key indexes:

Retention:

5. Key Workflows

5.1 Hop 1 — Scheduled Delta Sync (happy path)

sequenceDiagram
    autonumber
    participant Sched as Delta Scheduler
    participant API as Sync API
    participant Disp as Dispatcher
    participant Reader as Prod Reader
    participant Mask as Masking
    participant Snap as Snapshot
    participant Writer as Target Writer
    participant Aud as Audit Log

    Sched->>API: POST /syncs {chain_hop:1, mode:Delta, refNum}
    API->>API: validate, authz (no approval needed)
    API-->>Sched: 202 run_id
    API->>Disp: enqueue run
    Disp->>Reader: read since CDC resume token
    Reader->>Reader: throttle on replica lag
    Reader->>Mask: stream records
    Mask->>Mask: classify + apply policy
    Mask->>Snap: emit
    Snap->>Snap: capture pre-image (one-time for run)
    Snap->>Writer: write to tenant_refNum (Mongo Lower)
    Writer-->>Aud: per-item result
    Writer-->>Disp: run complete
    Disp->>Aud: persist resume token
    Disp-->>API: status = Succeeded

5.2 Hop 1 — Operator-Initiated Full Sync with Approval, Snapshot, and Rollback (error path)

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as Sync API
    participant Apv as Approver
    participant Snap as Snapshot Mgr
    participant Run as Run Executor
    participant Aud as Audit Log

    Op->>API: POST /syncs {chain_hop:1, mode:Full, refNum} + justification
    API->>API: validate, authz
    API-->>Op: 202 run_id (status: Awaiting Approval)
    API->>Apv: notify
    Apv->>API: POST /syncs/{id}/approve
    API->>Run: dispatch
    Run->>Snap: take snapshot of Mongo Lower tenant_refNum
    Snap-->>Run: snapshot_id stored
    Run->>Run: read + mask + write (errors mid-run)
    Run-->>Aud: failures recorded
    Run-->>API: status = Failed
    API-->>Op: alert with run_id
    Op->>API: POST /syncs/{id}/restore
    API->>Snap: restore from snapshot_id
    Snap-->>API: target restored
    API-->>Aud: restore event
    API-->>Op: status = Rolled Back

5.3 Sync Run State Lifecycle (both hops)

stateDiagram-v2
    [*] --> Queued
    Queued --> AwaitingPreflight: Hop 2
    Queued --> AwaitingApproval: Hop 1 Full/Selective
    Queued --> Running: Hop 1 Delta
    AwaitingPreflight --> AwaitingApproval: preflight pass
    AwaitingPreflight --> PreflightFailed: stale source
    PreflightFailed --> AwaitingApproval: senior override
    PreflightFailed --> [*]: operator cancels
    AwaitingApproval --> Running: approved
    AwaitingApproval --> Rejected: declined
    Running --> Succeeded
    Running --> Failed
    Succeeded --> SucceededWithBlockingFindings: validation callback fail_blocking (Hop 2)
    Failed --> RolledBack: operator restore
    Succeeded --> [*]
    SucceededWithBlockingFindings --> [*]
    RolledBack --> [*]
    Rejected --> [*]

5.4 Hop 2 — Happy path

Validation is invoked asynchronously. The Validation Webhook component on the Sync side calls DV's POST /validation/run with a callback_url; DV responds 202 with a validation_run_id and runs validation in the background. When validation finishes (success or failure), DV calls back into Sync at POST /chains/{correlation_id}/validation-complete, which causes Sync to update linked_validation_run_id and (if validation is blocking-Critical) flip status from Succeeded to SucceededWithBlockingFindings.

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as Sync API
    participant Pre as Pre-flight Validator
    participant Apv as Approver
    participant Snap as Snapshot Mgr
    participant Adp as Migration Pipeline Adapter
    participant Pipe as Migration Ingest Pipeline
    participant PLow as Postgres Lower
    participant Hook as Validation Webhook
    participant Val as Data Validation Framework
    participant Aud as Audit Log

    Op->>API: POST /syncs {chain_hop:2, mode:Full, refNum, target_env}
    API->>API: validate + authz + tag correlation_id
    API->>Pre: check freshness for (refNum, target_env)
    Pre-->>API: pass (last_hop1_completed_at within window)
    API-->>Op: 202 run_id, correlation_id (status: AwaitingApproval)
    API->>Apv: notify
    Apv->>API: POST /syncs/{id}/approve
    API->>Snap: snapshot Postgres Lower tenant_refNum
    Snap-->>API: snapshot_id
    API->>Adp: invoke ingest (source=mongo-lower-{env}, target=tenant_refNum, full, correlation_id)
    Adp->>Pipe: run with correlation_id propagated
    Pipe->>PLow: truncate + JOLT → DAAL writes
    Pipe-->>Adp: ingest_run_id, durations
    Adp-->>API: ingest complete, linked_ingest_run_id
    API->>API: persist linked_ingest_run_id, set status to Succeeded
    API-->>Aud: ingest-complete event
    API-->>Op: 200 status Succeeded, validation pending

    rect rgb(225,245,254)
    Note over Hook,Val: Asynchronous validation phase
    API->>Hook: enqueue validation trigger
    Hook->>Val: POST /validation/run with refNum, ingest_run_id, correlation_id, trigger_source hop2, callback_url
    Val-->>Hook: 202 validation_run_id
    Hook->>API: persist linked_validation_run_id
    Val->>Val: run rules and assertions (async)
    Val->>API: POST /chains/correlation_id/validation-complete with outcome pass or fail_blocking or fail_nonblocking
    API->>API: if fail_blocking then status SucceededWithBlockingFindings
    API-->>Aud: validation-complete event with outcome
    end

5.5 Hop 2 — Pre-flight failure with senior override

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as Sync API
    participant Pre as Pre-flight Validator
    participant Snr as Senior Approver
    participant Aud as Audit Log

    Op->>API: POST /syncs {chain_hop:2, refNum, target_env}
    API->>Pre: check freshness
    Pre-->>API: fail (last_hop1_completed_at older than window)
    API-->>Op: 409 with last_hop1_run_id, last_hop1_at, threshold
    Note over Op: Option A: run Hop 1 first, then retry
    Note over Op: Option B: request override (justified)
    Op->>Snr: request override with rationale
    Snr->>API: POST /chains/{id}/override-preflight {rationale}
    API-->>Aud: override event recorded (security page)
    Note over API: Sequence merges into 5.4 from "API-->>Op: 202..."

6. APIs & Contracts

Hop 1 request shape (excerpt) — POST /syncs:

{
  "chain_hop": 1,
  "mode": "Full | Delta | Selective",
  "refNum": "ABC123",
  "target_env": "INTQA | Staging | Dev",
  "scope": {
    "categories": ["operational", "lookup"],
    "artifact_types": ["entity_decl", "workflow", "ruleset"]
  },
  "justification": "fresh INTQA stand-up for customer ABC123"
}

Hop 2 request shape (excerpt) — POST /syncs:

{
  "chain_hop": 2,
  "mode": "Full",
  "refNum": "ABC123",
  "source_env": "mongo-lower-INTQA",
  "target_env": "postgres-lower-INTQA",
  "auto_validate": true,
  "justification": "INTQA baseline refresh for customer ABC123 Step 8 rehearsal"
}

Response includes run_id, correlation_id (Hop 2 only), current status, the approval URL (when applicable), and a chain_view_url pointing at GET /chains/{correlation_id}.

7. Configuration & Feature Flags

8. Security & Compliance

• AuthN: Service accounts (for scheduler) and operator identities (federated SSO). Tokens short-lived.
• AuthZ: Roles — sync-operator, sync-approver, senior-approver (for Hop 2 pre-flight overrides), auditor, dv-service (service account used by the Data Validation Framework to call POST /chains/{correlation_id}/validation-complete). Approver must be a different identity than the requester (enforced for both Hop 1 and Hop 2).
• Network: Sync Service runs in a dedicated subnet. Egress to lower envs allowed; egress to production primary is denied — only read-replica endpoints are reachable. No ingress from lower envs. The Migration Pipeline Adapter calls the migration ingest pipeline over an internal service mesh; the pipeline itself runs in its own VPC.
• Encryption: All data in transit TLS 1.3+. Snapshot store encrypted at rest with environment-scoped KMS keys.
• Masking (Hop 1): MDS classifications drive masking; missing classification is a hard error. PII uses deterministic pseudonymization keyed per-environment so referential integrity holds but identities cannot be reversed.
• Masking (Hop 2): Hop 2 inherits already-masked data from Mongo Lower. It does not re-mask, but it also cannot un-mask — the migration pipeline never reaches production.
• Audit: Every run (both hops) records actor, mode, refNum, approver, scope, snapshot id, resume token id (Hop 1), correlation id (Hop 2), pre-flight outcome (Hop 2), start/end time, and per-item outcomes. Audit entries are append-only.
• Compliance evidence: The audit log and chain view are referenced by Security & Compliance in the customer evidence package. Reports are exportable on demand and queryable by correlation ID.

9. Observability

Alerts:

10. Deployment

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    subgraph ProdVPC[" Production VPC "]
        ProdReplica[(MongoDB / Aurora<br/>Read Replica)]:::prod
    end

    subgraph SyncVPC[" Sync Service VPC "]
        SyncSvc[Sync Service<br/>API + Workers]:::internal
        QueueDb[(Run Queue)]:::internal
        SnapDb[(Snapshot Store<br/>S3 / equivalent)]:::internal
        AuditDb[(Audit Log)]:::internal
    end

    subgraph MigVPC[" Migration Pipeline VPC "]
        MigPipe[Migration Ingest Pipeline<br/>JOLT → DAAL]:::external
    end

    subgraph ValVPC[" Validation VPC "]
        Val[Data Validation Framework]:::external
    end

    subgraph LowerVPCs[" Lower-env VPCs (per env, per refNum) "]
        MLower[(Mongo Lower<br/>tenant_refNum)]:::lower
        PLower[(Postgres Lower<br/>tenant_refNum)]:::lower
    end

    SyncSvc -- "TLS, replica-only<br/>(Hop 1 reads)" --> ProdReplica
    SyncSvc -- "Hop 1 writes" --> MLower
    SyncSvc --> QueueDb
    SyncSvc --> SnapDb
    SyncSvc --> AuditDb

    SyncSvc -- "Hop 2: invoke<br/>(source=Mongo Lower)" --> MigPipe
    MLower --> MigPipe
    MigPipe --> PLower
    SyncSvc -- "Hop 2: trigger" --> Val
    Val -. reads .-> PLower

    style ProdVPC stroke-dasharray: 5 5,stroke:#00695c
    style LowerVPCs stroke-dasharray: 5 5,stroke:#1565c0
    style MigVPC stroke-dasharray: 5 5,stroke:#ad1457
    style ValVPC stroke-dasharray: 5 5,stroke:#ad1457

11. Operational Runbook Hooks

12. Failure Modes & Recovery

13. Open Questions / Decisions Log

14. Glossary Reference

See _shared/glossary.md.

LLD — Data Validation Framework

Low-Level Design · Workstream 4 of 9 · MongoDB → Aurora PostgreSQL Migration Companion document: HLD · Source: metadata-documents/MongoDB_to_PostgreSQL_v2_WS4_Data_Validation.docx

1. ⭐ Executive Summary

In one sentence: A pipeline-integrated engine that runs the four playbook reconciliation rules and per-entity business assertions on every ingest, classifies findings by severity, blocks workflow advancement on Critical findings, and produces a signed per-customer-per-entity report.

If you only read one section, read this.

• What we build: A Rule Engine with four checkers (row count, FK integrity, required-field coverage, sampled deep-diff), an Assertion Executor, a DLQ Inspector, a Severity Classifier, a Reporting Service, and a Gate Decision API used by the ingest pipeline and workflow tooling.
• Key design decisions: Validation is not an optional step — it's wired into the ingest pipeline so results land in metadata.ingest_run_history alongside the run record. Reproducibility is non-negotiable: rule version and data version are recorded on every finding. Validation runs accept an optional correlation_id (set by Sync Framework Hop 2) so the resulting report participates in the chain view.
• Trust model: The framework itself is verified against seeded errors (Step 5) before any customer relies on it. False-negative rate is measured.
• Phasing: Built once, applies to every customer. Phase 2 inherits without modification.
• Triggers: ingest pipeline at cutover; Sync Framework Hop 2 (auto-trigger); operator manual re-run.
• Open items at v0.1: sampled deep-diff sample size formula per entity volume; assertion-authoring linting rules; DLQ disposition SLA per severity.

2. Reference Architecture (recap from HLD)

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    Ingest[Ingest Pipeline]:::external --> Engine[Rule Engine]:::internal --> Class[Severity Classifier]:::internal --> Gate[Gate API]:::gate
    Class --> Reporter[Reporting Service]:::internal

Full diagrams in the HLD. The LLD elaborates each container.

3. Component Designs (C4 L3)

3.1 Rule Engine

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92

    In[Ingest Run Event<br/>refNum + entity + run_id]:::internal --> Plan[Execution Planner]:::internal
    Plan --> Row[Row Count Checker]:::internal
    Plan --> FK[FK Integrity Walker]:::internal
    Plan --> Cov[Required-Field Coverage Analyzer]:::internal
    Plan --> Diff[Sampled Deep-Diff Comparator]:::internal
    Plan --> Asrt[Assertion Executor]:::internal

    Row --> Findings[(Findings)]:::store
    FK --> Findings
    Cov --> Findings
    Diff --> Findings
    Asrt --> Findings

    Rules[(Rule + Assertion Library)]:::store --> Plan

3.2 DLQ Inspector

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92

    DLQ[(Dead Letter Queue)]:::store --> Inspector[DLQ Inspector]:::internal
    Inspector --> Group[Reason Grouper]:::internal
    Group --> Aging[Aging Tracker]:::internal
    Aging --> Findings[(Findings)]:::store

• Groups DLQ records by reason (required_field_missing, lookup_miss, type_conversion_failure, fk_integrity_failure, assertion_failure).
• Aging tracker flags records older than the disposition SLA per severity.
• Output joins into the same Findings store as the rule engine — one report covers both rule failures and DLQ inventory.

3.3 Severity Classifier + Router

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    Findings[(Findings)]:::internal --> Classifier[Classifier]:::internal
    Classifier --> Routes{Severity?}
    Routes -- Critical --> Block[Block Gate]:::gate
    Routes -- High --> Review[Active Review Queue]:::internal
    Routes -- Medium --> Logged[Logged]:::internal
    Routes -- Informational --> Logged
    Block --> GateAPI[Gate Decision API]:::gate

Severity matrix (from the source playbook):

3.4 Reporting Service

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92

    Findings[(Findings)]:::internal --> Composer[Report Composer]:::internal
    Composer --> Sign[Signer<br/>rule_version + data_version + hash]:::internal
    Sign --> ReportStore[(Report Store)]:::store
    Sign --> History[(metadata.ingest_run_history)]:::store
    ReportStore --> Dash[Leadership Dashboard]:::internal

• One report per customer per entity per ingest run.
• Reports are cryptographically signed with rule + data version metadata; re-running the same inputs produces an identical report hash.
• The Leadership Dashboard reads from the Report Store; engineering can drill into specific findings; security can verify signatures.

3.5 Gate Decision API

The pipeline calls the Gate API at Steps 3, 7, 8, and 9. Workflow tooling can disable advancement entirely until pass.

4. Data Model

erDiagram
    INGEST_RUN ||--o{ VALIDATION_RUN : "produces"
    VALIDATION_RUN ||--o{ FINDING : "contains"
    VALIDATION_RUN ||--|| REPORT : "produces"
    FINDING }o--|| RULE_DEFINITION : "based on"
    FINDING }o--o| ASSERTION : "based on"
    FINDING }o--|| SEVERITY_RULE : "classified by"
    DLQ_RECORD ||--o{ FINDING : "inventoried into"
    REPORT ||--o| GATE_DECISION : "drives"

    INGEST_RUN {
        uuid id PK
        text refNum
        text entity
        text step "3|7|8|9"
        timestamp started_at
        timestamp ended_at
    }
    VALIDATION_RUN {
        uuid id PK
        uuid ingest_run_id FK
        text correlation_id "stamped by Sync Hop 2; null otherwise"
        text rule_library_version
        text entity_decl_version
        text trigger_source "ingest|hop2|manual"
        text status "Queued|Running|Succeeded|Failed"
    }
    RULE_DEFINITION {
        text id PK "e.g. row_count_parity"
        text kind "default|entity_specific"
        text version
        jsonb spec
    }
    ASSERTION {
        text id PK
        text entity
        text version
        text expression
        text severity_override
    }
    SEVERITY_RULE {
        text finding_kind PK
        text default_severity
        text override_policy
    }
    FINDING {
        uuid id PK
        uuid validation_run_id FK
        text rule_id FK
        text assertion_id FK
        text severity "Critical|High|Medium|Informational"
        int observed_value
        int expected_value
        jsonb evidence
    }
    DLQ_RECORD {
        uuid id PK
        text refNum
        text entity
        text reason
        timestamp arrived_at
        jsonb source_document
        jsonb jolt_output
    }
    REPORT {
        uuid id PK
        uuid validation_run_id FK
        text refNum
        text entity
        text rule_library_version
        text content_hash
        text signature
        timestamp created_at
        text storage_uri
    }
    GATE_DECISION {
        uuid id PK
        uuid report_id FK
        text outcome "pass|fail|overridden"
        text approver
        timestamp decided_at
    }

Key indexes:

Retention:

5. Key Workflows

5.1 Per-Ingest Validation (happy path → gate pass)

sequenceDiagram
    autonumber
    participant Ingest as Ingest Pipeline
    participant Engine as Rule Engine
    participant DLQI as DLQ Inspector
    participant Class as Classifier
    participant Rep as Reporting
    participant Gate as Gate API
    participant Wf as Workflow Tool

    Ingest->>Engine: ingest_run_completed(refNum, entity, run_id)
    Engine->>Engine: plan = rules ∪ assertions for entity
    par
        Engine->>Engine: row count
        Engine->>Engine: FK integrity
        Engine->>Engine: required-field coverage
        Engine->>Engine: sampled deep-diff
        Engine->>Engine: assertions
    end
    Engine->>DLQI: inventory DLQ for (refNum, entity)
    DLQI-->>Engine: DLQ findings
    Engine->>Class: findings
    Class->>Class: classify each finding
    Class->>Rep: classified findings
    Rep->>Rep: compose + sign report
    Rep-->>Gate: report ready
    Wf->>Gate: GET /gate/{refNum}/{entity}/{step}
    Gate-->>Wf: pass

5.2 Step 8 Bulk Dry-Run with DLQ Resolution (error path)

sequenceDiagram
    autonumber
    participant QA as QA Team
    participant Ingest as Bulk Ingest
    participant Engine as Rule Engine
    participant DLQI as DLQ Inspector
    participant DomOwner as Domain Owner
    participant Mapping as Mapping Team
    participant Gate as Gate API

    QA->>Ingest: kick off bulk dry-run
    Ingest->>Engine: run validation per entity
    Engine-->>QA: findings (Critical: 3 DLQ assertion_failure)
    DLQI-->>QA: DLQ inventory grouped by reason
    QA->>DomOwner: triage assertion_failure records
    DomOwner->>Mapping: correct JOLT spec or assertion
    Mapping-->>Ingest: re-register spec
    QA->>Ingest: re-run for affected entity
    Engine-->>QA: findings (Critical: 0)
    Engine-->>Gate: report ready, pass
    QA->>Gate: GET /gate/{refNum}/{entity}/8 → pass
    Note over QA,Gate: Step 8 sign-off can proceed

5.3 Validation Run State Lifecycle

stateDiagram-v2
    [*] --> Queued
    Queued --> Running
    Running --> SucceededClean: 0 Critical
    Running --> SucceededWithFindings: High/Medium only
    Running --> FailedBlocking: ≥1 Critical
    FailedBlocking --> Overridden: senior approval
    SucceededClean --> [*]
    SucceededWithFindings --> [*]
    Overridden --> [*]

6. APIs & Contracts

POST /validation/run request shape (excerpt):

{
  "refNum": "ABC123",
  "ingest_run_id": "uuid",
  "trigger_source": "hop2",
  "correlation_id": "chain-ABC123-20260518-7f3a",
  "callback_url": "https://sync-service.internal/chains/chain-ABC123-20260518-7f3a/validation-complete"
}

POST /validation/run returns immediately with 202 { "validation_run_id": "uuid" }. The framework runs validation in the background; on completion (success or failure), it POSTs to callback_url with:

{
  "validation_run_id": "uuid",
  "report_id": "uuid",
  "outcome": "pass | fail_blocking | fail_nonblocking",
  "summary": {
    "critical": 0,
    "high": 0,
    "medium": 0,
    "informational": 0
  }
}

The callback is retried with exponential backoff up to chain.callback.max_retries (default 5); the receiver (Sync Framework) is expected to be idempotent on (correlation_id, validation_run_id).

When correlation_id is present, the resulting VALIDATION_RUN row carries it through; the Sync Framework's GET /chains/{correlation_id} view joins this report into the end-to-end chain. trigger_source is one of ingest, hop2, manual. callback_url is only honored when trigger_source = hop2; for ingest and manual, callers poll GET /validation/runs/{id} instead.

Findings response (excerpt):

{
  "validation_run_id": "uuid",
  "refNum": "ABC123",
  "entity": "claim",
  "rule_library_version": "2026.05.18-3",
  "correlation_id": "chain-ABC123-20260518-7f3a",
  "findings": [
    {
      "rule_id": "row_count_parity",
      "severity": "Critical",
      "observed": 99987,
      "expected": 100000,
      "evidence": { "source_query": "...", "target_query": "..." }
    }
  ]
}

7. Configuration & Feature Flags

8. Security & Compliance

• AuthN/AuthZ: Roles — pipeline (service account for ingest), sync-service (service account used by Sync Framework Hop 2 to call POST /validation/run and GET /validation/runs?correlation_id=…), qa-eng, viewer, senior-approver, auditor. The framework also makes outbound calls back into the Sync Framework on validation completion; that outbound call uses a dv-service identity issued to this framework.
• Signed reports: Each report carries rule version, entity declaration version, content hash, and signature. Any change in inputs produces a different hash; tampering is detectable.
• Audit: Every gate decision, override, and report retrieval is logged. Override events are page-worthy for security awareness.
• Data sensitivity: Reports contain aggregate counts, not raw rows. Deep-diff evidence stores field-level diffs but redacts values classified as PII via MDS.
• Retention: 7 years for reports and findings (compliance baseline; configurable to satisfy specific regulators).

9. Observability

Alerts:

10. Deployment

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    subgraph IngestVPC[" Ingest VPC "]
        Ingest[Ingest Pipeline]:::external
        DLQ[(DLQ)]:::store
        History[(metadata.<br/>ingest_run_history)]:::store
    end

    subgraph ValVPC[" Validation Service VPC "]
        Eng[Rule Engine workers]:::internal
        Cls[Classifier]:::internal
        Rep[Reporting]:::internal
        Gate[Gate API]:::internal
        RuleStore[(Rule Library)]:::store
        ReportStore[(Report Store)]:::store
    end

    Ingest -->|run event| Eng
    Ingest --> DLQ --> Eng
    Eng --> Cls --> Rep --> ReportStore
    Rep --> History
    Cls --> Gate
    RuleStore --> Eng

    Dash[Leadership Dashboard]:::internal
    ReportStore --> Dash

11. Operational Runbook Hooks

12. Failure Modes & Recovery

13. Open Questions / Decisions Log

14. Glossary Reference

See _shared/glossary.md.

LLD — End-to-End Validation Framework

Low-Level Design · Workstreams 5 + 6 + 7 of 9 · MongoDB → Aurora PostgreSQL Migration Companion document: HLD · Sources: WS5_Use_Case_Validation.docx, WS6_Mocking_Framework.docx, WS7_Payload_Validation.docx

1. ⭐ Executive Summary

In one sentence: A unified framework with three cooperating sub-systems — Use Case Orchestrator (workflows + parity + performance), Mock Service (every external partner, scenario-switchable), and Payload Diff Engine (byte-level diff against production baselines) — wired together so a single test run produces functional, performance, and payload evidence.

If you only read one section, read this.

• What we build: Three sub-systems backed by one orchestrator, one dashboard, and one evidence pipeline. INTQA outbound traffic is intercepted, mocked, and diffed in the same flow.
• Key design decisions: Mock interception sits at the platform's outbound HTTP layer (single integration point per app, not per-call). Network-level controls also block direct egress as a safety net. Baselines are anonymized using MDS classifications. Diff classification is rule-driven; only Blocking differences gate workflow advancement.
• Operating model: Customer teams own use-case catalogs. Integration owners own classification rules and per-integration sign-off. Platform Eng owns the mock service and diff engine.
• Phasing: Built once. Phase 2 customers reuse infrastructure with their own catalogs and baselines; bucketing makes catalogs largely shared.
• Open items at v0.1: scenario versioning strategy across customers; performance baseline staleness threshold; hypercare clock automation policy.

2. Reference Architecture (recap from HLD)

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40

    Orch[Use Case Orchestrator]:::internal --> Plat[Migrated Platform - INTQA]:::lower
    Plat -. outbound .-> Mock[Mock Service]:::internal
    Plat -. outbound .-> Cap[Outbound Capture]:::internal
    Cap --> Diff[Diff Engine]:::internal
    Base[(Production Baselines)]:::prod --> Diff

Full diagrams in the HLD. The LLD elaborates each sub-system.

3. Component Designs (C4 L3)

3.1 Use Case Orchestrator (WS5)

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92

    Trigger[Trigger<br/>scheduler · operator · CI]:::internal --> Loader[Catalog Loader]:::internal
    Loader --> Picker[Scenario Picker<br/>per refNum]:::internal
    Picker --> Runner[Scenario Runner]:::internal
    Runner --> ParityCmp[Parity Comparator]:::internal
    Runner --> PerfGate[Performance Gate]:::internal
    ParityCmp --> Resulter[Result Aggregator]:::internal
    PerfGate --> Resulter
    Resulter --> ReporterAPI[Reporting API]:::internal

    Catalog[(Use Case Catalog)]:::store --> Loader
    ParityBase[(Parity Baseline)]:::store --> ParityCmp
    PerfBase[(Perf Baseline)]:::store --> PerfGate

3.2 Mock Service + Scenario Manager (WS6)

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    AppCall[App outbound call<br/>via Mock Adapter]:::external --> Router[Request Router<br/>per integration]:::internal
    Router --> Selector[Scenario Selector<br/>active scenario per test run]:::internal
    Selector --> Matcher[Request Matcher<br/>method · path · headers · body]:::internal
    Matcher --> Builder[Response Builder<br/>shape · headers · status · latency profile]:::internal
    Builder --> Out[Response to app]:::external
    Webhook[Webhook Trigger<br/>incoming events]:::internal --> Plat[App webhook endpoint]:::external

    ScenStore[(Scenario Catalog)]:::store --> Selector
    BehRef[(Behavior Reference<br/>captured from prod)]:::store --> Builder
    Log[(Mock Interaction Log)]:::store
    Router --> Log
    Builder --> Log

Scenarios per integration (standard catalog): happy_path, validation_error, auth_failure, timeout, rate_limit, malformed_response, partial_success, 5xx_server_error.

3.3 Outbound Payload Capture (WS7)

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    App[Migrated App]:::external --> Adapter["Mock Adapter<br/>(also captures)"]:::internal
    Adapter --> Capture[Capture Recorder]:::internal
    Capture --> Anon[Anonymizer<br/>MDS classifications]:::internal
    Anon --> Store[(Captured Payload Store)]:::internal
    Adapter --> MockSvc[Mock Service]:::internal

• The same adapter that routes calls to the Mock also captures the payload. One integration point, two outputs.
• Anonymization runs before persistence — captures never store raw PII regardless of source.

3.4 Diff Engine + Classification (WS7)

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef gate fill:#ffebee,stroke:#c62828,color:#b71c1c

    Cap[(Captured Payload)]:::store --> Normalize[Normalizer<br/>canonical form]:::internal
    Base[(Production Baseline)]:::store --> Normalize2[Normalizer]:::internal
    Normalize --> Comparator[Byte / Tree Comparator]:::internal
    Normalize2 --> Comparator
    Comparator --> Classifier[Classification Engine]:::internal
    Rules[(Classification Rule Book)]:::store --> Classifier
    Classifier --> Output{Diff class}
    Output -- Blocking --> Gate[Blocking Gate]:::gate
    Output -- Notable --> Review[Active Review]:::internal
    Output -- Tolerable --> Logged[Logged in aggregate]:::internal
    Output -- Expected --> Ignored[Ignored]:::internal
    Classifier --> Report[Diff Report Writer]:::internal

Classification matrix (from the source playbook):

3.5 Reporting + Dashboard

flowchart LR
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92

    OrchRes[Orchestrator Results]:::internal --> Agg[Aggregator]:::internal
    DiffRes[Diff Reports]:::internal --> Agg
    MockLog[Mock Interaction Logs]:::internal --> Agg
    Agg --> Pkg[Evidence Packager]:::internal
    Pkg --> Reports[(Per-customer Evidence)]:::store
    Pkg --> Dash[Pass/Fail Dashboard]:::internal

The dashboard shows per-customer readiness; drilldown gets to per-scenario and per-integration detail. The evidence package is what the customer team signs at Step 9.

4. Data Model

erDiagram
    USE_CASE ||--o{ SCENARIO_RUN : "executed as"
    USE_CASE }o--|| CUSTOMER_CATALOG : "belongs to"
    SCENARIO_RUN ||--|| PARITY_RESULT : "produces"
    SCENARIO_RUN ||--|| PERF_RESULT : "produces"
    SCENARIO_RUN ||--o{ CAPTURED_PAYLOAD : "may capture"
    CAPTURED_PAYLOAD ||--|| DIFF_RECORD : "compared into"
    DIFF_RECORD }o--|| PRODUCTION_BASELINE : "vs"
    DIFF_RECORD }o--|| CLASSIFICATION_RULE : "classified by"
    MOCK_SCENARIO ||--o{ MOCK_INTERACTION : "produced"
    INTEGRATION ||--o{ MOCK_SCENARIO : "has"
    INTEGRATION ||--o{ PRODUCTION_BASELINE : "has"
    CUSTOMER_CATALOG }o--|| CUSTOMER : "belongs to"
    PERF_BASELINE }o--|| USE_CASE : "for"

    USE_CASE {
        uuid id PK
        text refNum
        text name
        text category "journey|smoke|cross_system|edge|perf"
        text version
    }
    CUSTOMER_CATALOG {
        uuid id PK
        text refNum
        text version
        timestamp signed_off_at
        text signed_off_by
    }
    SCENARIO_RUN {
        uuid id PK
        uuid use_case_id FK
        text refNum
        text status "Pass|Fail|Error"
        text pinned_correlation_id "optional: Sync Framework Hop 2 chain run"
        timestamp started_at
        timestamp ended_at
        jsonb outputs
    }
    PARITY_RESULT {
        uuid id PK
        uuid scenario_run_id FK
        text outcome "Match|Mismatch"
        jsonb diff
    }
    PERF_RESULT {
        uuid id PK
        uuid scenario_run_id FK
        int observed_p95_ms
        int baseline_p95_ms
        text outcome "Within|Variance|Regressed"
    }
    PERF_BASELINE {
        uuid id PK
        uuid use_case_id FK
        int baseline_p50_ms
        int baseline_p95_ms
        int baseline_p99_ms
        int tolerance_pct
        timestamp measured_at
    }
    INTEGRATION {
        uuid id PK
        text name
        text owner_team
        text contract_version
    }
    MOCK_SCENARIO {
        uuid id PK
        uuid integration_id FK
        text name "happy_path|validation_error|..."
        jsonb response_spec
        int latency_profile_ms
    }
    MOCK_INTERACTION {
        uuid id PK
        uuid scenario_run_id FK
        uuid mock_scenario_id FK
        jsonb request
        jsonb response
        timestamp at
    }
    CAPTURED_PAYLOAD {
        uuid id PK
        uuid scenario_run_id FK
        uuid integration_id FK
        text refNum
        jsonb request_payload
        jsonb response_payload
        text hash
    }
    PRODUCTION_BASELINE {
        uuid id PK
        uuid integration_id FK
        text refNum
        jsonb request_payload
        jsonb response_payload
        text hash
        timestamp captured_at
    }
    DIFF_RECORD {
        uuid id PK
        uuid captured_payload_id FK
        uuid baseline_id FK
        text classification "Expected|Tolerable|Notable|Blocking"
        jsonb diff_tree
    }
    CLASSIFICATION_RULE {
        uuid id PK
        uuid integration_id FK
        text version
        jsonb rule_spec
        text signed_off_by
    }
    CUSTOMER {
        text refNum PK
        text name
    }

Key indexes:

Retention:

5. Key Workflows

5.1 INTQA Test Run with Mocked Integrations + Payload Diff (happy path)

sequenceDiagram
    autonumber
    participant Trig as Trigger (CI / Operator)
    participant Orch as Use Case Orchestrator
    participant App as Migrated App (INTQA)
    participant Ada as Mock Adapter (in app)
    participant Mock as Mock Service
    participant Cap as Capture
    participant Diff as Diff Engine
    participant Dash as Dashboard

    Trig->>Orch: run customer suite (refNum)
    Orch->>App: execute scenario via DAAL
    App->>Ada: outbound HTTP call
    par interception
        Ada->>Mock: route to active scenario
        Mock-->>Ada: response (shape, status, latency)
    and capture
        Ada->>Cap: record request + response (anonymized)
    end
    Ada-->>App: response
    App-->>Orch: scenario output + timings
    Cap->>Diff: compare to production baseline
    Diff-->>Orch: classification per integration
    Orch->>Orch: parity check + perf check
    Orch->>Dash: aggregated pass/fail

5.2 INTQA Test Run with Forced Failure Scenario (timeout)

sequenceDiagram
    autonumber
    participant Op as Operator
    participant Mock as Mock Service
    participant Orch as Orchestrator
    participant App as Migrated App
    participant Ada as Mock Adapter

    Op->>Mock: switch integration X to "timeout"
    Op->>Orch: re-run scenario Y
    Orch->>App: execute
    App->>Ada: outbound call
    Ada->>Mock: route
    Mock-->>Ada: (no response - simulated timeout)
    Note over Ada,App: App enters retry/backoff logic
    Ada-->>App: timeout error after configured latency
    App-->>Orch: scenario result (handled gracefully?)
    Orch->>Orch: parity check: did app handle timeout per baseline?

5.3 Hypercare Daily Smoke

sequenceDiagram
    autonumber
    participant Cron as Hypercare Scheduler
    participant Orch as Orchestrator
    participant Dash as Dashboard
    participant Triage as Triage Hook (PagerDuty/Slack)

    Cron->>Orch: kick smoke suite for refNum
    Orch->>Orch: run scenarios + parity + perf
    Orch->>Dash: post results
    alt new P1/P2
        Orch->>Triage: alert + reset 7-day clock
    else clean
        Orch->>Dash: increment clean streak
    end

5.4 Hypercare Clock Lifecycle

stateDiagram-v2
    [*] --> Day1: cutover
    Day1 --> Day2: clean
    Day2 --> Day3: clean
    Day3 --> Day4: clean
    Day4 --> Day5: clean
    Day5 --> Day6: clean
    Day6 --> Day7: clean
    Day7 --> SignOffEligible: 7 clean days
    Day2 --> Day1: P1/P2 raised (reset)
    Day3 --> Day1: P1/P2 raised (reset)
    Day4 --> Day1: P1/P2 raised
    Day5 --> Day1: P1/P2 raised
    Day6 --> Day1: P1/P2 raised
    Day7 --> Day1: P1/P2 raised
    SignOffEligible --> [*]: customer sign-off

6. APIs & Contracts

Orchestrator

Mock Service

Payload Diff

Gate

7. Configuration & Feature Flags

8. Security & Compliance

• Network isolation: INTQA outbound to real-partner endpoints is denied at the firewall. The Mock Service is the only allowed destination for integration calls. Unexpected egress triggers an alert.
• Anonymization: All production captures pass through the Anonymizer, which uses MDS classifications to redact PII before storage. The same anonymization runs at INTQA capture time as a defense-in-depth measure.
• Baseline access: Production baselines are read-only to the diff engine; refresh is authenticated and audited.
• Classification rule changes: All rule changes are versioned, reviewed, and signed off by the integration owner. The rule version is recorded on each diff record.
• Sign-off artifacts: Per-integration, per-customer signed reports are cryptographically signed; reproducible on demand.
• Mock interactions retention: 90 days, then archived. Hot retention sized for replay use cases.
• Override discipline: A blocking-gate override requires a senior-approver identity and a free-text rationale; security reviews overrides weekly.

9. Observability

Alerts:

10. Deployment

flowchart TB
    classDef internal fill:#f5f5f5,stroke:#616161,color:#212121
    classDef lower fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef store fill:#ede7f6,stroke:#5e35b1,color:#311b92
    classDef prod fill:#e0f2f1,stroke:#00695c,color:#004d40
    classDef external fill:#fce4ec,stroke:#ad1457,color:#880e4f

    subgraph INTQAEnv[" INTQA VPC (per customer / shared) "]
        App[Migrated App<br/>with Mock Adapter]:::lower
        Mock[Mock Service]:::internal
        Cap[Capture Workers]:::internal
        Diff[Diff Engine workers]:::internal
        OrchSvc[Orchestrator]:::internal
        ScenStore[(Scenario Catalog)]:::store
        Captured[(Captured Payload Store)]:::store
        DiffStore[(Diff Records)]:::store
    end

    subgraph ProdEnv[" Production VPC "]
        ProdApp[Existing Platform]:::prod
        CapWindow["Production Capture Window<br/>(read-only)"]:::prod
    end

    subgraph BaseEnv[" Baseline Storage "]
        BaseStore[(Anonymized Production Baselines)]:::store
    end

    subgraph Net[" Firewall "]
        FW[INTQA Egress Policy<br/>real partners DENIED]:::internal
    end

    OrchSvc --> App
    App --> Mock
    App --> Cap
    Cap --> Captured
    Cap --> Diff
    Diff --> DiffStore
    Diff --> BaseStore

    ProdApp --> CapWindow --> BaseStore

    App -. blocked .- FW
    FW -.->|alerts on attempts| Sec[Security]:::external

    Dash[Pass/Fail Dashboard]:::internal
    DiffStore --> Dash
    OrchSvc --> Dash

    style INTQAEnv stroke-dasharray: 5 5,stroke:#1565c0
    style ProdEnv stroke-dasharray: 5 5,stroke:#00695c

11. Operational Runbook Hooks

12. Failure Modes & Recovery

13. Open Questions / Decisions Log

14. Glossary Reference

See _shared/glossary.md.