Sprint 6.3 · Federation + 2nd Hi-End Server

1. 🎯 Sprint Summary

2. 🏗️ Federation Architecture

┌─────────────────────────┐                ┌─────────────────────────┐
│  KL DC · Region A       │ ◄──────────►   │  Cyberjaya · Region B   │
│                         │  10G dark      │                         │
│  Hi-End Cluster #1      │  fibre + VPN   │  Hi-End Cluster #2      │
│  4× L40S GPU            │                │  4× L40S GPU            │
│  Inference: vLLM        │                │  Inference: vLLM        │
│  Storage: NVMe primary  │                │  Storage: NVMe replica  │
│                         │                │                         │
│  MariaDB Galera node 1  │ ◄─async────►   │  MariaDB Galera node 2  │
│  pgvector primary       │  cross-region  │  pgvector replica       │
│  MinIO bucket replica   │  < 1s p95     │  MinIO bucket replica   │
└──────────┬──────────────┘                └──────────┬──────────────┘
           │                                          │
           ▼                                          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    HAProxy Region Router (DNS-aware)                 │
│  Tenant region tag → primary region · automatic failover on health   │
│  Health check every 10s · failover RTO target < 30s                 │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
                    Tenants · Patients · Doctors

PER-TENANT REGION PINNING
  · Tenant created with primary_region tag (kl_dc_1 OR cyberjaya_1)
  · Read/write goes to primary first
  · On primary unhealthy → automatic failover to secondary
  · After recovery → automated catch-up + re-pin to primary

REPLICATION GUARANTEES
  · Galera cluster · synchronous within region · async cross-region
  · pgvector · async streaming replica · < 1s lag
  · MinIO · cross-region bucket replication · 5min interval
  · Audit log · WORM with cross-region hash chain

3. 🚦 Pre-Sprint Gate Checklist

• Sprint 6.2 closeout · 30 tenants stable across 4 states
• Cyberjaya colocation contract signed · power audit passed
• 2nd Hi-End hardware delivered · 4× L40S GPUs · same spec as KL DC
• 10G dark fibre + VPN to KL DC provisioned · < 5ms RTT verified
• Galera cluster strategy agreed (synchronous in-region · async cross-region)
• HAProxy region router deployed in test environment
• Per-tenant region tag schema migration drafted
• DR runbook v2 with cross-region procedures
• Pen-test scope updated (multi-region surface)
• Compliance Lead reviewed cross-region data residency (still on Malaysian soil)

4. 🧩 Sprint Scope

• Cyberjaya Cluster Bring-Up: Rack · power · cooling · 4× L40S install · OS + driver · vLLM · Triton · Litellm proxy
• Cross-Region Networking: 10G dark fibre · VPN tunnel · IPsec · DNS region-aware
• Galera Cross-Region Cluster: Synchronous in-region · async streaming cross-region · conflict resolution · last-writer-wins with audit
• pgvector Async Replica: Streaming replication · < 1s lag p95 · monitoring
• MinIO Bucket Replication: Cross-region S3 replication · 5min interval · DICOM + audio shared
• Per-Tenant Region Pinning: Tenant migration with region tag · routing rules · admin override
• HAProxy Region Router: DNS-aware · health check 10s · failover RTO < 30s · automatic re-pin
• Audit Log Cross-Region Hash Chain: WORM extended cross-region · M9 audit verifies both sides
• DR Drill v2: Full restore from cold backup · cross-region · target RTO < 4h · documented
• Multi-Region Monitoring: Filament admin extension · per-region health · replication lag · latency
• Per-Tenant Migration: 5 East Malaysia tenants migrated to Cyberjaya as primary (closer to East ML)

5. 📅 Day-by-Day Plan (15 days)

6. 📦 Deliverables

7. 👥 Team Capacity

8. 🔔 Sprint Ceremonies

• Mon 15 Nov 9am — Sprint Planning (90 min)
• Daily 9am — Standup (15 min · DevOps deep-dive Tue/Thu)
• Fri 19 + Fri 26 Nov 4pm — Mid-sprint demos (60 min each)
• Wed 25 Nov 4pm — Failover drill rehearsal (90 min)
• Tue 30 Nov 4pm — DR drill review (60 min)
• Thu 2 Dec 2pm — Pen-test debrief (60 min)
• Mon 5 Dec 9am — Sprint Demo (90 min)
• Mon 5 Dec 11am — Sprint Retro (60 min)

9. 🩺 Sign-off Items

• Cross-region replication lag < 1s p95 verified
• Failover RTO < 30s on 5 different drill scenarios
• DR drill: full restore < 4h · cross-region · audit intact
• Per-tenant region pinning works · admin override functional
• WORM audit cross-region hash chain unbroken
• 5 East Malaysia tenants migrated · latency improvement measured · zero data loss
• Pen-test light: 0 critical · 0 high · ≤ 2 medium with mitigations
• Compliance pack updated · external consultant attestation
• Final demo (5 Dec) — written sign-off · Compliance Lead + Doc Zam

10. 🎬 Demo Agenda — 5 Dec 9am (90 min)

11. 🛡️ Contingency