System Design: Banking & High-Scale Systems
A Complete Guide for Senior Engineers
1. Design a UPI-like Transaction Processing System
The Challenge
UPI (Unified Payments Interface) processes 10+ billion transactions monthly in India. During peak hours (salary days, festivals, IPL matches), it handles 50,000+ transactions per second.
Let's design a system that can handle this scale while ensuring:
- Zero duplicate payments
- Sub-second response times
- 99.99% availability
- Strict consistency for money movement
High-Level Architecture
┌─────────────────────────────────────────────────────────────────────────────────┐ │ UPI TRANSACTION SYSTEM │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Payer │ │ PhonePe/ │ │ Payee │ │ │ │ (User) │───▶│ GPay │◀───│ (User) │ │ │ └──────────┘ └────┬─────┘ └──────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ NPCI (Switch) │ │ │ │ ┌───────────────────────────────────────────────────────────────┐ │ │ │ │ │ API Gateway │ │ │ │ │ │ • Rate Limiting (per VPA) │ │ │ │ │ │ • Request Validation │ │ │ │ │ │ • TLS Termination │ │ │ │ │ └───────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ │ ┌───────────────────────────────────────────────────────────────┐ │ │ │ │ │ Transaction Router │ │ │ │ │ │ • Route to correct bank │ │ │ │ │ │ • Load balancing │ │ │ │ │ │ • Circuit breaker per bank │ │ │ │ │ └───────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ │ ┌───────────────────────────────────────────────────────────────┐ │ │ │ │ │ Transaction Processor │ │ │ │ │ │ • Idempotency check │ │ │ │ │ │ • Transaction state machine │ │ │ │ │ │ • Timeout handling │ │ │ │ │ └───────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────────┼───────────────────┐ │ │ ▼ ▼ ▼ │ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │ │ HDFC │ │ SBI │ │ ICICI │ │ │ │ Bank │ │ Bank │ │ Bank │ │ │ │ │ │ │ │ │ │ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ │ │ │ Core │ │ │ │ Core │ │ │ │ Core │ │ │ │ │ │Banking │ │ │ │Banking │ │ │ │Banking │ │ │ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ │ └────────────┘ └────────────┘ └────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Transaction Flow (The Happy Path)
┌─────────────────────────────────────────────────────────────────────────────────┐ │ UPI TRANSACTION FLOW │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ User A (Payer) User B (Payee) │ │ VPA: usera@hdfc VPA: userb@sbi │ │ Bank: HDFC Bank: SBI │ │ │ │ ┌─────┐ 1. Pay ₹100 ┌─────┐ 2. Route ┌─────┐ │ │ │ App │───────────────▶│NPCI │────────────▶│ SBI │ │ │ │ │ │ │ │ │ │ │ └─────┘ └─────┘ └──┬──┘ │ │ │ │ │ │ │ │ │ 3. Validate │ │ │ │ │ VPA exists │ │ │ │ │◀──────────────────┘ │ │ │ │ │ │ │ │ 4. Debit Request │ │ │ ┌─────┐◀───────────┤ │ │ │ │HDFC │ │ │ │ │ └──┬──┘ │ │ │ │ │ │ │ │ │ │ 5. Check balance, debit │ │ │ │ 6. Return debit confirmation │ │ │ │────────────────▶ │ │ │ │ │ │ │ │ 7. Credit Request │ │ │ │─────────────────▶┌─────┐ │ │ │ │ │ SBI │ │ │ │ │ 8. Credit done │ │ │ │ │ │◀─────────────────└─────┘ │ │ │ │ │ │ │ 9. Success │ │ │ │◀─────────────────────┤ │ │ │ │ TOTAL TIME: < 2 seconds │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Database Schema
sql-- Transaction table (partitioned by date) CREATE TABLE transactions ( txn_id UUID PRIMARY KEY, payer_vpa VARCHAR(50) NOT NULL, payee_vpa VARCHAR(50) NOT NULL, payer_bank_code VARCHAR(10) NOT NULL, payee_bank_code VARCHAR(10) NOT NULL, amount DECIMAL(15,2) NOT NULL, currency VARCHAR(3) DEFAULT 'INR', status VARCHAR(20) NOT NULL, -- INITIATED, DEBITED, CREDITED, FAILED, REVERSED failure_reason VARCHAR(100), initiated_at TIMESTAMP NOT NULL, completed_at TIMESTAMP, idempotency_key VARCHAR(100) UNIQUE, -- Audit fields created_at TIMESTAMP DEFAULT NOW(), updated_at TIMESTAMP DEFAULT NOW() ) PARTITION BY RANGE (initiated_at); -- Create monthly partitions CREATE TABLE transactions_2024_01 PARTITION OF transactions FOR VALUES FROM ('2024-01-01') TO ('2024-02-01'); -- Indexes CREATE INDEX idx_txn_payer ON transactions(payer_vpa, initiated_at DESC); CREATE INDEX idx_txn_payee ON transactions(payee_vpa, initiated_at DESC); CREATE INDEX idx_txn_status ON transactions(status) WHERE status IN ('INITIATED', 'DEBITED');
Handling Scale: What Happens at Different Loads
Low Load (1,000 TPS)
┌─────────────────────────────────────────────────────────────────┐ │ LOW LOAD ARCHITECTURE │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ SETUP: │ │ • 3 API Gateway instances │ │ • 5 Transaction Processor instances │ │ • 1 PostgreSQL primary + 2 replicas │ │ • Single Redis cluster (3 nodes) │ │ │ │ BEHAVIOR: │ │ • Synchronous processing │ │ • Direct DB writes │ │ • Simple round-robin load balancing │ │ • p99 latency: 200ms │ │ │ └─────────────────────────────────────────────────────────────────┘
Medium Load (10,000 TPS)
┌─────────────────────────────────────────────────────────────────┐ │ MEDIUM LOAD ARCHITECTURE │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ CHANGES FROM LOW LOAD: │ │ │ │ 1. ADD KAFKA FOR ASYNC PROCESSING │ │ ┌─────────────────────────────────────────────────┐ │ │ │ Transaction Processor (sync) │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ ┌────────────┐ ┌────────────────────┐ │ │ │ │ │ Kafka │───▶│ Async Settlement │ │ │ │ │ │ (Buffer) │ │ Workers │ │ │ │ │ └────────────┘ └────────────────────┘ │ │ │ └─────────────────────────────────────────────────┘ │ │ │ │ 2. DATABASE SHARDING │ │ • Shard by payer_bank_code (distributes load) │ │ • Each bank's transactions on separate shard │ │ │ │ 3. CACHING │ │ • VPA validation results cached (5 min TTL) │ │ • Bank routing rules cached │ │ │ │ BEHAVIOR: │ │ • Debit is sync, credit is async │ │ • User gets "Success" after debit confirms │ │ • Credit settles in background │ │ • p99 latency: 500ms │ │ │ └─────────────────────────────────────────────────────────────────┘
High Load (50,000+ TPS) - IPL Final Scenario
┌─────────────────────────────────────────────────────────────────┐ │ HIGH LOAD ARCHITECTURE │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ SPIKE HANDLING: │ │ │ │ 1. PRE-PROVISIONING │ │ • Know IPL schedule in advance │ │ • Scale up 2 hours before match │ │ • 100 API Gateway instances │ │ • 200 Transaction Processor instances │ │ │ │ 2. REQUEST QUEUING │ │ ┌─────────────────────────────────────────────────┐ │ │ │ │ │ │ │ Incoming ──▶ Queue ──▶ Rate-limited ──▶ Process│ │ │ │ 50K TPS Buffer 30K TPS Workers │ │ │ │ │ │ │ │ Queue absorbs 20K TPS spike │ │ │ │ User sees "Processing..." for extra 2-3 sec │ │ │ │ │ │ │ └─────────────────────────────────────────────────┘ │ │ │ │ 3. CIRCUIT BREAKERS PER BANK │ │ ┌─────────────────────────────────────────────────┐ │ │ │ if hdfc_failure_rate > 50%: │ │ │ │ open_circuit(hdfc) │ │ │ │ return "Bank temporarily unavailable" │ │ │ │ # Prevents cascade failure │ │ │ └─────────────────────────────────────────────────┘ │ │ │ │ 4. GRACEFUL DEGRADATION │ │ • Disable non-critical features (balance check) │ │ • Increase timeouts for slower banks │ │ • Queue low-value transactions │ │ │ │ BEHAVIOR: │ │ • Some transactions queued (user waits 5-10 sec) │ │ • Critical path prioritized │ │ • p99 latency: 2-5 seconds (acceptable during spike) │ │ │ └─────────────────────────────────────────────────────────────────┘
Preventing Double Spending
This is the MOST critical requirement. Here's how we prevent it:
go// Idempotency implementation type TransactionService struct { db *sql.DB redis *redis.Client kafkaProducer *kafka.Producer } func (s *TransactionService) ProcessPayment(ctx context.Context, req *PaymentRequest) (*PaymentResponse, error) { // STEP 1: Generate idempotency key from request idempotencyKey := generateIdempotencyKey(req) // Key = SHA256(payer_vpa + payee_vpa + amount + client_txn_id) // STEP 2: Check if already processed (Redis first, then DB) if existing := s.checkIdempotency(ctx, idempotencyKey); existing != nil { return existing, nil // Return same response } // STEP 3: Acquire distributed lock on payer account lockKey := fmt.Sprintf("lock:account:%s", req.PayerVPA) lock, err := s.redis.SetNX(ctx, lockKey, req.TxnID, 30*time.Second).Result() if !lock { return nil, ErrAccountBusy // Another transaction in progress } defer s.redis.Del(ctx, lockKey) // STEP 4: Create transaction record with INITIATED status txn := &Transaction{ ID: uuid.New(), IdempotencyKey: idempotencyKey, Status: "INITIATED", // ... other fields } // Use database transaction for atomicity dbTx, _ := s.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable}) defer dbTx.Rollback() // STEP 5: Insert with conflict handling _, err = dbTx.ExecContext(ctx, ` INSERT INTO transactions (id, idempotency_key, status, ...) VALUES ($1, $2, $3, ...) ON CONFLICT (idempotency_key) DO NOTHING `, txn.ID, idempotencyKey, txn.Status) if err != nil { return nil, err } // Check if insert happened (returns 0 rows if duplicate) var insertedID string err = dbTx.QueryRowContext(ctx, ` SELECT id FROM transactions WHERE idempotency_key = $1 `, idempotencyKey).Scan(&insertedID) if insertedID != txn.ID.String() { // Duplicate request - return existing transaction return s.getExistingTransaction(ctx, idempotencyKey) } // STEP 6: Proceed with debit debitResult, err := s.debitAccount(ctx, req.PayerVPA, req.Amount) if err != nil { s.updateTransactionStatus(ctx, dbTx, txn.ID, "FAILED", err.Error()) dbTx.Commit() return nil, err } // STEP 7: Update to DEBITED and commit s.updateTransactionStatus(ctx, dbTx, txn.ID, "DEBITED", "") dbTx.Commit() // STEP 8: Queue credit (async) s.kafkaProducer.Produce(&kafka.Message{ Topic: "credit-requests", Key: []byte(req.PayeeVPA), Value: serializeCredit(txn), }) // STEP 9: Cache idempotency result s.redis.Set(ctx, "idem:"+idempotencyKey, txn.ID.String(), 24*time.Hour) return &PaymentResponse{ TxnID: txn.ID, Status: "SUCCESS", // From user's perspective }, nil }
Counter Questions & Answers
Q: "What if the credit fails after debit succeeds?"
Answer:
SCENARIO: User A debited ₹100, but SBI is down, credit fails. HANDLING: 1. IMMEDIATE (within 30 seconds): - Retry credit 3 times with exponential backoff - If SBI responds with "account not found", mark as FAILED 2. SHORT-TERM (30 sec - 5 min): - Move to dead letter queue - Dedicated worker retries every minute - Alert on-call if still failing 3. REVERSAL (after 5 min): - Initiate automatic reversal - Credit back to payer's account - Update status to REVERSED - Notify user: "Payment reversed due to technical issue" 4. MANUAL INTERVENTION (if reversal fails): - Create support ticket - Ops team manually reconciles - User notified within 24 hours STATE MACHINE: INITIATED → DEBITED → CREDITED (success) → CREDIT_FAILED → REVERSAL_INITIATED → REVERSED → MANUAL_REVIEW (if reversal fails)
Q: "How do you handle timeout from bank?"
Answer:
TIMEOUT SCENARIO: We sent debit request to HDFC, no response in 30 seconds. THE PROBLEM: - Did debit happen or not? - We don't know the state SOLUTION: Transaction Status Check API 1. IMMEDIATE: - Mark transaction as TIMEOUT_PENDING - Do NOT retry (could cause duplicate debit!) 2. VERIFICATION: - Call bank's "Transaction Status API" with our txn_id - Bank responds: SUCCESS / FAILED / NOT_FOUND 3. BASED ON RESPONSE: SUCCESS → Continue with credit FAILED → Mark as failed, notify user NOT_FOUND → Retry the debit (it never reached bank) 4. IF STATUS API ALSO TIMES OUT: - Wait for bank's end-of-day reconciliation file - Settle in T+1 batch - User sees "Pending" status CODE:
gofunc (s *TransactionService) handleTimeout(ctx context.Context, txn *Transaction) { // Set timeout status s.updateStatus(txn.ID, "TIMEOUT_PENDING") // Check with bank for attempt := 0; attempt < 5; attempt++ { status, err := s.bankClient.CheckTransactionStatus(ctx, txn.BankRefID) if err != nil { time.Sleep(time.Duration(attempt*2) * time.Second) continue } switch status { case "SUCCESS": s.updateStatus(txn.ID, "DEBITED") s.proceedWithCredit(ctx, txn) return case "FAILED": s.updateStatus(txn.ID, "FAILED") return case "NOT_FOUND": // Safe to retry s.retryDebit(ctx, txn) return } } // All status checks failed - manual intervention s.createSupportTicket(txn) s.updateStatus(txn.ID, "MANUAL_REVIEW") }
Q: "What if NPCI itself goes down?"
Answer:
NPCI FAILURE = NATIONAL PAYMENT OUTAGE MITIGATION LAYERS: 1. MULTI-REGION DEPLOYMENT: - NPCI runs in Mumbai, Chennai, Hyderabad - Active-active setup - DNS failover between regions 2. BANK-TO-BANK FALLBACK: - For same-bank transfers (HDFC to HDFC) - Route directly to bank, bypass NPCI - Covers 30% of transactions 3. QUEUING: - If NPCI is slow but not down - Queue requests, process when available - User sees "Processing may take longer" 4. GRACEFUL DEGRADATION: - Disable new payment initiation - Allow checking existing transaction status - Show "UPI temporarily unavailable" 5. COMMUNICATION: - Real-time status page - Push notification to users - Estimated recovery time REALITY: - NPCI has 99.99% uptime (< 1 hour downtime/year) - Most "UPI down" issues are individual bank issues
2. Design Swiggy's Order Placement System (1M Orders/Hour)
The Challenge
1 million orders per hour = ~278 orders per second sustained, with peaks of 1000+ orders/second during lunch/dinner.
Each order involves:
- User authentication
- Restaurant availability check
- Menu validation
- Price calculation
- Coupon application
- Payment processing
- Order confirmation
- Kitchen notification
- Delivery partner assignment
High-Level Architecture
┌─────────────────────────────────────────────────────────────────────────────────┐ │ SWIGGY ORDER SYSTEM ARCHITECTURE │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ │ │ │ Mobile │ │ │ │ App │ │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ CDN (Static Assets) │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ API Gateway │ │ │ │ • Rate limiting (100 req/min per user) │ │ │ │ • JWT validation │ │ │ │ • Request routing │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ├──────────────┬──────────────┬──────────────┬──────────────┐ │ │ ▼ ▼ ▼ ▼ ▼ │ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │ │ User │ │ Restaurant │ │ Order │ │ Payment │ │ Delivery │ │ │ │ Service │ │ Service │ │ Service │ │ Service │ │ Service │ │ │ └────────────┘ └────────────┘ └────────────┘ └────────────┘ └────────────┘ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ │ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │ │ User DB │ │Restaurant │ │ Order DB │ │ Payment DB │ │Delivery DB │ │ │ │ (Postgres) │ │ DB (Mongo) │ │ (Postgres) │ │ (Postgres) │ │ (Redis) │ │ │ └────────────┘ └────────────┘ └────────────┘ └────────────┘ └────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ Kafka (Event Bus) │ │ │ │ Topics: order-created, order-confirmed, payment-completed, │ │ │ │ delivery-assigned, order-delivered │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ Redis Cluster │ │ │ │ • Session cache │ │ │ │ • Restaurant availability cache │ │ │ │ • Cart storage │ │ │ │ • Rate limiting counters │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Order Placement Flow
┌─────────────────────────────────────────────────────────────────────────────────┐ │ ORDER PLACEMENT SAGA │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ USER CLICKS "PLACE ORDER" │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ STEP 1: VALIDATE ORDER │ │ │ │ • Check restaurant is open │ │ │ │ • Check items are available │ │ │ │ • Validate prices haven't changed │ │ │ │ • Apply coupons, calculate final amount │ │ │ │ │ │ │ │ TIME: 50ms | ROLLBACK: None needed │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ STEP 2: RESERVE INVENTORY │ │ │ │ • Mark items as "reserved" in restaurant inventory │ │ │ │ • Prevent overselling during concurrent orders │ │ │ │ • Reservation expires in 10 minutes if not confirmed │ │ │ │ │ │ │ │ TIME: 30ms | ROLLBACK: Release reservation │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ STEP 3: CREATE ORDER (PENDING) │ │ │ │ • Generate order ID │ │ │ │ • Store order details │ │ │ │ • Status: PENDING_PAYMENT │ │ │ │ │ │ │ │ TIME: 20ms | ROLLBACK: Cancel order │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ STEP 4: PROCESS PAYMENT │ │ │ │ • Call payment gateway (Razorpay/Juspay) │ │ │ │ • Handle 3DS authentication if needed │ │ │ │ • Wait for payment confirmation │ │ │ │ │ │ │ │ TIME: 2-30 sec | ROLLBACK: Refund payment │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ STEP 5: CONFIRM ORDER │ │ │ │ • Update order status: CONFIRMED │ │ │ │ • Convert reservation to actual sale │ │ │ │ • Notify restaurant (push notification + tablet) │ │ │ │ • Start delivery partner search │ │ │ │ │ │ │ │ TIME: 100ms | ROLLBACK: Cancel + Refund │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ STEP 6: ASSIGN DELIVERY PARTNER (Async) │ │ │ │ • Find nearby available partners │ │ │ │ • Send assignment request │ │ │ │ • If rejected, try next partner │ │ │ │ • Timeout: 5 minutes │ │ │ │ │ │ │ │ TIME: 1-5 min | ROLLBACK: Self-pickup or cancel │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ TOTAL ORDER CONFIRMATION TIME: 3-35 seconds │ │ USER SEES: "Order Confirmed!" after Step 5 │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Handling 1M Orders/Hour
The Math
1,000,000 orders / hour = 16,667 orders / minute = 278 orders / second (average) Peak (lunch 12-1 PM, dinner 8-9 PM): = 500-1000 orders / second Each order = ~10 service calls = 2,780 - 10,000 internal requests / second
Scaling Strategy
┌─────────────────────────────────────────────────────────────────────────────────┐ │ SCALING FOR 1M ORDERS/HOUR │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ LAYER 1: API GATEWAY │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • 50 instances behind ALB │ │ │ │ • Each handles 200 req/sec │ │ │ │ • Total: 10,000 req/sec capacity │ │ │ │ • Auto-scale trigger: CPU > 60% │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ LAYER 2: ORDER SERVICE (Critical Path) │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • 100 instances │ │ │ │ • Each handles 10 orders/sec │ │ │ │ • Total: 1,000 orders/sec capacity │ │ │ │ • Pre-scaled before peak hours │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ LAYER 3: DATABASE │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ ORDER DB: │ │ │ │ • Primary: Writes (orders, status updates) │ │ │ │ • 5 Read Replicas: Reads (order history, tracking) │ │ │ │ • Connection pool: 500 connections │ │ │ │ │ │ │ │ SHARDING (for 100M+ orders): │ │ │ │ • Shard by city_id (10 shards) │ │ │ │ • Each shard handles 100 orders/sec │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ LAYER 4: CACHE (Redis Cluster) │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Restaurant data: 100K restaurants cached │ │ │ │ • Menu prices: Updated every 5 min │ │ │ │ • User cart: Stored in Redis (not DB) │ │ │ │ • Availability: Real-time from restaurant │ │ │ │ │ │ │ │ Cache hit ratio target: 95% │ │ │ │ DB calls reduced: 10x │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ LAYER 5: ASYNC PROCESSING (Kafka) │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ NON-BLOCKING OPERATIONS (moved out of order flow): │ │ │ │ • Analytics event publishing │ │ │ │ • Email/SMS notifications │ │ │ │ • Loyalty points calculation │ │ │ │ • Fraud scoring │ │ │ │ • Search index updates │ │ │ │ │ │ │ │ RESULT: Order confirmation 3x faster │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
What Happens When Load Increases
┌─────────────────────────────────────────────────────────────────────────────────┐ │ LOAD INCREASE SCENARIO: IPL FINAL │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ NORMAL (6 PM): 300 orders/sec │ │ IPL FINAL (8 PM): 3,000 orders/sec (10x spike!) │ │ │ │ T-2 HOURS (6 PM): PRE-SCALING │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Scale Order Service: 100 → 500 instances │ │ │ │ • Scale Payment Service: 50 → 200 instances │ │ │ │ • Warm up Redis cache with popular restaurants │ │ │ │ • Pre-allocate database connections │ │ │ │ • Alert delivery partners in advance │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ T-0 (8 PM): SPIKE BEGINS │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ FIRST 5 MINUTES: │ │ │ │ • Auto-scaling kicks in (but takes 2-3 min) │ │ │ │ • Queue builds up in Kafka │ │ │ │ • Some users see "High demand, please wait" │ │ │ │ │ │ │ │ GRACEFUL DEGRADATION ACTIVATED: │ │ │ │ • Disable coupon validation (use cached) │ │ │ │ • Skip fraud check for trusted users │ │ │ │ • Reduce order history fetch │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ T+5 MINUTES: STABLE AT HIGH LOAD │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • All instances warmed up │ │ │ │ • Queue draining at 3,000/sec │ │ │ │ • p99 latency: 2 sec (acceptable) │ │ │ │ • Error rate: < 0.1% │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ T+2 HOURS (10 PM): LOAD DECREASES │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Traffic drops to 500 orders/sec │ │ │ │ • Scale-in begins (gradual, over 30 min) │ │ │ │ • Re-enable all features │ │ │ │ • Process deferred fraud checks │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Counter Questions & Answers
Q: "Restaurant goes offline mid-order. What happens?"
Answer:
SCENARIO: User places order, restaurant marks itself offline before confirming. DETECTION: - Restaurant tablet sends "going offline" event - OR restaurant doesn't accept order within 2 minutes HANDLING: 1. ORDER NOT YET CONFIRMED: ┌─────────────────────────────────────────────────┐ │ • Cancel order immediately │ │ • Refund payment (instant for wallet, 3-5 days │ │ for cards) │ │ • Show user: "Restaurant closed. Here are │ │ similar options nearby" │ │ • Offer 10% discount code for inconvenience │ └─────────────────────────────────────────────────┘ 2. ORDER ALREADY CONFIRMED, FOOD NOT STARTED: ┌─────────────────────────────────────────────────┐ │ • Cancel and full refund │ │ • Apologize with compensation │ │ • Log incident for restaurant quality score │ └─────────────────────────────────────────────────┘ 3. ORDER CONFIRMED, FOOD BEING PREPARED: ┌─────────────────────────────────────────────────┐ │ • Try to reach restaurant (call) │ │ • If reachable: Complete the order │ │ • If not: Cancel + full refund + extra credit │ └─────────────────────────────────────────────────┘ PREVENTION: - Restaurants can't go offline if they have orders in preparation - Force them to complete or cancel explicitly - Penalty for repeated offline-during-order incidents
Q: "How do you prevent overselling a popular item?"
Answer:
SCENARIO: Biryani shop has 50 biryanis. 100 people order simultaneously. NAIVE APPROACH (WRONG): check_inventory() → 50 available place_order() → OK Problem: 100 concurrent checks all see 50, all place orders! CORRECT APPROACH: Distributed Lock + Atomic Decrement SOLUTION 1: Redis Atomic Counter
gofunc (s *OrderService) ReserveItem(ctx context.Context, itemID string, quantity int) error { key := fmt.Sprintf("inventory:%s", itemID) // DECRBY is atomic newCount, err := s.redis.DecrBy(ctx, key, int64(quantity)).Result() if err != nil { return err } if newCount < 0 { // Oversold! Rollback s.redis.IncrBy(ctx, key, int64(quantity)) return ErrOutOfStock } return nil }
SOLUTION 2: Database WITH SELECT FOR UPDATE
sqlBEGIN; -- Lock the row SELECT quantity FROM inventory WHERE item_id = 'biryani' FOR UPDATE; -- Check and update atomically UPDATE inventory SET quantity = quantity - 1 WHERE item_id = 'biryani' AND quantity >= 1; -- If no rows updated, out of stock COMMIT;
SOLUTION 3: Pre-allocated Buckets (for extreme scale) Instead of 1 counter, split into 10: inventory:biryani:bucket:0 = 5 inventory:biryani:bucket:1 = 5 ... inventory:biryani:bucket:9 = 5 Each server uses a different bucket. No contention between servers!
3. Design a Real-Time Order Tracking System
The Challenge
Users want to see:
- Order status changes (confirmed, preparing, picked up, on the way)
- Live delivery partner location (updating every 5 seconds)
- ETA updates
For 1M active orders, that's potentially 1M WebSocket connections with 200K location updates per second.
Architecture
┌─────────────────────────────────────────────────────────────────────────────────┐ │ REAL-TIME TRACKING ARCHITECTURE │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ DELIVERY PARTNER APP USER APP │ │ ┌─────────────────┐ ┌─────────────────┐ │ │ │ GPS Updates │ │ WebSocket │ │ │ │ every 5 sec │ │ Connection │ │ │ └────────┬────────┘ └────────┬────────┘ │ │ │ │ │ │ ▼ │ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ LOCATION INGESTION LAYER │ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ Kafka │───▶│ Location │───▶│ Redis │ │ │ │ │ │ (buffer) │ │ Processor │ │ (latest) │ │ │ │ │ │ │ │ 100 inst │ │ │ │ │ │ │ └─────────────┘ └─────────────┘ └──────┬──────┘ │ │ │ │ │ │ │ │ └─────────────────────────────────────────────────┼────────────────────────┘ │ │ │ │ │ ┌─────────────────────────────────────────────────┼────────────────────────┐ │ │ │ WEBSOCKET LAYER │ │ │ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌──────▼──────┐ │ │ │ │ │ WebSocket │◀───│ Redis │◀───│ Pub/Sub │ │ │ │ │ │ Servers │ │ Pub/Sub │ │ Publisher │ │ │ │ │ │ (500) │ │ │ │ │ │ │ │ │ └──────┬──────┘ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ │ └──────────┼───────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────┐ │ │ │ USER DEVICE │ │ │ │ (Live Map) │ │ │ └─────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Location Update Flow
┌─────────────────────────────────────────────────────────────────────────────────┐ │ LOCATION UPDATE FLOW │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ 1. DELIVERY PARTNER sends GPS coordinates │ │ POST /location │ │ { │ │ "partner_id": "dp123", │ │ "lat": 12.9716, │ │ "lng": 77.5946, │ │ "timestamp": 1708300000, │ │ "accuracy": 10, │ │ "speed": 25, │ │ "bearing": 180 │ │ } │ │ │ │ 2. KAFKA buffers incoming locations │ │ Topic: partner-locations │ │ Partition key: partner_id (ensures ordering per partner) │ │ Retention: 1 hour │ │ │ │ 3. LOCATION PROCESSOR consumes and processes │ │ - Filter noisy readings (GPS jump detection) │ │ - Calculate ETA based on route │ │ - Store latest location in Redis │ │ - Publish to Redis Pub/Sub │ │ │ │ 4. REDIS stores latest location │ │ Key: location:dp123 │ │ Value: {lat, lng, eta, updated_at} │ │ TTL: 5 minutes │ │ │ │ 5. REDIS PUB/SUB broadcasts to WebSocket servers │ │ Channel: order:12345:location │ │ Subscribers: WebSocket servers with connected users │ │ │ │ 6. WEBSOCKET SERVER pushes to user │ │ ws.send({type: "location", lat: 12.9716, lng: 77.5946, eta: "5 min"}) │ │ │ │ LATENCY: Partner sends → User sees: < 500ms │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Scaling WebSocket Connections
go// WebSocket server with Redis Pub/Sub type TrackingServer struct { redis *redis.Client connections sync.Map // orderID -> []*websocket.Conn } func (s *TrackingServer) HandleConnection(w http.ResponseWriter, r *http.Request) { orderID := r.URL.Query().Get("order_id") // Upgrade to WebSocket conn, _ := upgrader.Upgrade(w, r, nil) defer conn.Close() // Register connection s.registerConnection(orderID, conn) defer s.unregisterConnection(orderID, conn) // Subscribe to order updates pubsub := s.redis.Subscribe(ctx, "order:"+orderID+":location") defer pubsub.Close() // Forward updates to WebSocket ch := pubsub.Channel() for msg := range ch { var location LocationUpdate json.Unmarshal([]byte(msg.Payload), &location) err := conn.WriteJSON(location) if err != nil { return // Connection closed } } } // Location processor publishes updates func (p *LocationProcessor) ProcessLocation(ctx context.Context, loc *PartnerLocation) { // Get active orders for this partner orders := p.getActiveOrders(loc.PartnerID) for _, orderID := range orders { // Calculate ETA eta := p.calculateETA(loc, orderID) // Store latest p.redis.Set(ctx, "location:"+loc.PartnerID, loc, 5*time.Minute) // Publish to subscribers update := LocationUpdate{ Lat: loc.Lat, Lng: loc.Lng, ETA: eta, } p.redis.Publish(ctx, "order:"+orderID+":location", update) } }
Handling 1M Concurrent WebSocket Connections
┌─────────────────────────────────────────────────────────────────────────────────┐ │ WEBSOCKET SCALING STRATEGY │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ CHALLENGE: 1M concurrent connections │ │ │ │ SOLUTION: Horizontal scaling with sticky sessions │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ 500 WebSocket Servers │ │ │ │ • Each handles 2,000 connections │ │ │ │ • 8 GB RAM per server │ │ │ │ • Connection = ~4KB memory │ │ │ │ │ │ │ │ 2,000 connections × 4KB = 8MB per server (negligible) │ │ │ │ Actual bottleneck: CPU for message processing │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ CONNECTION ROUTING: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Use consistent hashing on order_id │ │ │ │ • All connections for same order → same server │ │ │ │ • If server dies → reconnect to new server │ │ │ │ • Client handles reconnection with exponential backoff │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ MESSAGE BROADCAST (200K location updates/sec): │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Redis Pub/Sub fans out to subscribed servers │ │ │ │ • Each server only subscribes to relevant orders │ │ │ │ • Dynamic subscription: subscribe when user connects, │ │ │ │ unsubscribe when disconnects │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ FALLBACK FOR SCALE ISSUES: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Reduce update frequency: 5 sec → 10 sec │ │ │ │ • Batch updates: Send 3 locations in 1 message │ │ │ │ • Polling fallback: If WS fails, poll every 15 sec │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
4. Design a Bank Ledger System with Strict Consistency
The Challenge
A bank ledger must:
- Never lose money (durability)
- Never create money (consistency)
- Always balance (debit = credit)
- Support auditing (immutability)
- Handle 100K transactions/second
Double-Entry Bookkeeping
┌─────────────────────────────────────────────────────────────────────────────────┐ │ DOUBLE-ENTRY BOOKKEEPING │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ RULE: Every transaction has equal debits and credits │ │ │ │ EXAMPLE: User A transfers ₹100 to User B │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ LEDGER ENTRIES: │ │ │ │ │ │ │ │ Entry 1: DEBIT User A's account ₹100 │ │ │ │ Entry 2: CREDIT User B's account ₹100 │ │ │ │ │ │ │ │ Sum of debits = Sum of credits = ₹100 ✓ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ WHY DOUBLE-ENTRY: │ │ • Self-balancing: Can detect errors by checking totals │ │ • Audit trail: Know where money came from and went │ │ • Immutable: Never update, only append │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Ledger Schema
sql-- Accounts table CREATE TABLE accounts ( account_id UUID PRIMARY KEY, account_type VARCHAR(20) NOT NULL, -- ASSET, LIABILITY, EQUITY, REVENUE, EXPENSE account_name VARCHAR(100) NOT NULL, currency VARCHAR(3) DEFAULT 'INR', created_at TIMESTAMP DEFAULT NOW(), is_active BOOLEAN DEFAULT TRUE ); -- Ledger entries (append-only, immutable) CREATE TABLE ledger_entries ( entry_id BIGSERIAL PRIMARY KEY, transaction_id UUID NOT NULL, account_id UUID NOT NULL REFERENCES accounts(account_id), entry_type VARCHAR(10) NOT NULL, -- DEBIT or CREDIT amount DECIMAL(18,2) NOT NULL CHECK (amount > 0), balance_after DECIMAL(18,2) NOT NULL, -- Running balance currency VARCHAR(3) DEFAULT 'INR', description TEXT, created_at TIMESTAMP DEFAULT NOW(), -- Immutability: No updates allowed CONSTRAINT no_negative_balance CHECK (balance_after >= 0) ); -- Transactions table (groups ledger entries) CREATE TABLE transactions ( transaction_id UUID PRIMARY KEY, transaction_type VARCHAR(50) NOT NULL, status VARCHAR(20) NOT NULL, -- PENDING, COMPLETED, FAILED idempotency_key VARCHAR(100) UNIQUE, created_at TIMESTAMP DEFAULT NOW(), completed_at TIMESTAMP, -- Ensure balanced transaction CONSTRAINT balanced_transaction CHECK ( (SELECT SUM(CASE WHEN entry_type = 'DEBIT' THEN amount ELSE 0 END) - SUM(CASE WHEN entry_type = 'CREDIT' THEN amount ELSE 0 END) FROM ledger_entries WHERE ledger_entries.transaction_id = transactions.transaction_id) = 0 ) ); -- Indexes for performance CREATE INDEX idx_ledger_account ON ledger_entries(account_id, created_at DESC); CREATE INDEX idx_ledger_txn ON ledger_entries(transaction_id);
Transaction Processing with Strict Consistency
go// Ledger service with serializable transactions type LedgerService struct { db *sql.DB } func (s *LedgerService) Transfer(ctx context.Context, req *TransferRequest) (*TransferResult, error) { // SERIALIZABLE isolation ensures no concurrent modifications tx, err := s.db.BeginTx(ctx, &sql.TxOptions{ Isolation: sql.LevelSerializable, }) if err != nil { return nil, err } defer tx.Rollback() // Generate transaction ID txnID := uuid.New() // 1. Check idempotency var existingTxn string err = tx.QueryRowContext(ctx, ` SELECT transaction_id FROM transactions WHERE idempotency_key = $1 `, req.IdempotencyKey).Scan(&existingTxn) if err == nil { // Already processed return s.getExistingResult(ctx, existingTxn) } // 2. Get current balances with row lock var fromBalance, toBalance decimal.Decimal err = tx.QueryRowContext(ctx, ` SELECT COALESCE( (SELECT balance_after FROM ledger_entries WHERE account_id = $1 ORDER BY entry_id DESC LIMIT 1), 0 ) FOR UPDATE `, req.FromAccount).Scan(&fromBalance) if err != nil { return nil, err } // Check sufficient balance if fromBalance.LessThan(req.Amount) { return nil, ErrInsufficientBalance } err = tx.QueryRowContext(ctx, ` SELECT COALESCE( (SELECT balance_after FROM ledger_entries WHERE account_id = $1 ORDER BY entry_id DESC LIMIT 1), 0 ) FOR UPDATE `, req.ToAccount).Scan(&toBalance) if err != nil { return nil, err } // 3. Create transaction record _, err = tx.ExecContext(ctx, ` INSERT INTO transactions (transaction_id, transaction_type, status, idempotency_key) VALUES ($1, 'TRANSFER', 'COMPLETED', $2) `, txnID, req.IdempotencyKey) if err != nil { return nil, err } // 4. Create ledger entries (debit and credit) newFromBalance := fromBalance.Sub(req.Amount) newToBalance := toBalance.Add(req.Amount) // Debit entry _, err = tx.ExecContext(ctx, ` INSERT INTO ledger_entries (transaction_id, account_id, entry_type, amount, balance_after, description) VALUES ($1, $2, 'DEBIT', $3, $4, $5) `, txnID, req.FromAccount, req.Amount, newFromBalance, req.Description) if err != nil { return nil, err } // Credit entry _, err = tx.ExecContext(ctx, ` INSERT INTO ledger_entries (transaction_id, account_id, entry_type, amount, balance_after, description) VALUES ($1, $2, 'CREDIT', $3, $4, $5) `, txnID, req.ToAccount, req.Amount, newToBalance, req.Description) if err != nil { return nil, err } // 5. Commit atomically if err = tx.Commit(); err != nil { return nil, err } return &TransferResult{ TransactionID: txnID, Status: "COMPLETED", }, nil }
Scaling the Ledger
┌─────────────────────────────────────────────────────────────────────────────────┐ │ LEDGER SCALING STRATEGY │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ CHALLENGE: 100K transactions/second with SERIALIZABLE isolation │ │ │ │ SOLUTION: Shard by account │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ SHARDING STRATEGY │ │ │ │ │ │ │ │ Shard key: account_id │ │ │ │ Number of shards: 100 │ │ │ │ Distribution: hash(account_id) % 100 │ │ │ │ │ │ │ │ Shard 0: Accounts 0-999999 │ │ │ │ Shard 1: Accounts 1000000-1999999 │ │ │ │ ... │ │ │ │ │ │ │ │ Each shard handles: 1,000 TPS │ │ │ │ Total capacity: 100,000 TPS │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ CROSS-SHARD TRANSACTIONS: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Problem: Account A (Shard 1) → Account B (Shard 5) │ │ │ │ │ │ │ │ Solution: Two-Phase Commit (2PC) with Saga fallback │ │ │ │ │ │ │ │ Phase 1: PREPARE │ │ │ │ • Shard 1: Reserve ₹100 from A (PENDING_DEBIT) │ │ │ │ • Shard 5: Prepare to receive (PENDING_CREDIT) │ │ │ │ │ │ │ │ Phase 2: COMMIT │ │ │ │ • Coordinator: Both prepared? → COMMIT │ │ │ │ • Shard 1: Finalize debit │ │ │ │ • Shard 5: Finalize credit │ │ │ │ │ │ │ │ Failure handling: │ │ │ │ • If any PREPARE fails → ABORT all │ │ │ │ • If COMMIT fails → Retry with idempotency │ │ │ │ • If coordinator dies → Timeout + recovery │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
5. Design a Refund System with Idempotency Guarantees
The Challenge
Refunds are dangerous because:
- Users spam the refund button
- Network retries duplicate requests
- Partial failures can cause double refunds
- Must handle original payment method no longer valid
Idempotent Refund Architecture
┌─────────────────────────────────────────────────────────────────────────────────┐ │ IDEMPOTENT REFUND SYSTEM │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────┐ │ │ │ User │ │ │ │ Clicks │ │ │ │ "Refund" │ │ │ └────┬─────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ REFUND SERVICE │ │ │ │ │ │ │ │ STEP 1: IDEMPOTENCY CHECK │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ │ │ idempotency_key = hash(order_id + refund_type + amount) │ │ │ │ │ │ │ │ │ │ │ │ Check Redis: GET refund:{idempotency_key} │ │ │ │ │ │ If exists → Return cached response (no new refund) │ │ │ │ │ │ If not → Continue processing │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ STEP 2: VALIDATE REFUND │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ │ │ • Order exists and is delivered? │ │ │ │ │ │ • Refund amount ≤ order amount? │ │ │ │ │ │ • Within refund window (7 days)? │ │ │ │ │ │ • Not already fully refunded? │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ STEP 3: CREATE REFUND RECORD │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ │ │ INSERT INTO refunds (id, order_id, amount, status, idem_key) │ │ │ │ │ │ ON CONFLICT (idem_key) DO NOTHING │ │ │ │ │ │ │ │ │ │ │ │ If insert returns 0 rows → Duplicate, fetch existing │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ STEP 4: PROCESS REFUND │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ │ │ Call payment gateway with OUR refund_id │ │ │ │ │ │ Gateway is also idempotent on our refund_id │ │ │ │ │ │ Update status: PROCESSING → COMPLETED / FAILED │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ STEP 5: CACHE RESULT │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ │ │ SET refund:{idempotency_key} = response │ │ │ │ │ │ EXPIRE 24 hours │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Implementation
gotype RefundService struct { db *sql.DB redis *redis.Client paymentGateway PaymentGateway } func (s *RefundService) ProcessRefund(ctx context.Context, req *RefundRequest) (*RefundResponse, error) { // Generate idempotency key idempotencyKey := s.generateIdempotencyKey(req.OrderID, req.RefundType, req.Amount) // STEP 1: Check cache first (fast path) if cached := s.checkCache(ctx, idempotencyKey); cached != nil { return cached, nil } // STEP 2: Validate refund eligibility order, err := s.getOrder(ctx, req.OrderID) if err != nil { return nil, ErrOrderNotFound } if err := s.validateRefund(order, req); err != nil { return nil, err } // STEP 3: Create refund record with idempotency refund := &Refund{ ID: uuid.New(), OrderID: req.OrderID, Amount: req.Amount, Status: "INITIATED", IdempotencyKey: idempotencyKey, } // Use INSERT ... ON CONFLICT for atomicity result, err := s.db.ExecContext(ctx, ` INSERT INTO refunds (id, order_id, amount, status, idempotency_key) VALUES ($1, $2, $3, $4, $5) ON CONFLICT (idempotency_key) DO NOTHING `, refund.ID, refund.OrderID, refund.Amount, refund.Status, refund.IdempotencyKey) if err != nil { return nil, err } rowsAffected, _ := result.RowsAffected() if rowsAffected == 0 { // Duplicate request - fetch existing refund return s.getExistingRefund(ctx, idempotencyKey) } // STEP 4: Process with payment gateway gatewayResp, err := s.paymentGateway.Refund(ctx, &GatewayRefundRequest{ RefundID: refund.ID.String(), // Our ID for their idempotency OriginalPaymentID: order.PaymentID, Amount: req.Amount, }) // Update status based on response var status string var failureReason string if err != nil || !gatewayResp.Success { status = "FAILED" if err != nil { failureReason = err.Error() } else { failureReason = gatewayResp.FailureReason } } else { status = "COMPLETED" } s.db.ExecContext(ctx, ` UPDATE refunds SET status = $1, failure_reason = $2, completed_at = NOW() WHERE id = $3 `, status, failureReason, refund.ID) // STEP 5: Cache result response := &RefundResponse{ RefundID: refund.ID, Status: status, Amount: req.Amount, } s.cacheResult(ctx, idempotencyKey, response) return response, nil } func (s *RefundService) generateIdempotencyKey(orderID string, refundType string, amount decimal.Decimal) string { data := fmt.Sprintf("%s:%s:%s", orderID, refundType, amount.String()) hash := sha256.Sum256([]byte(data)) return hex.EncodeToString(hash[:]) }
Handling Edge Cases
┌─────────────────────────────────────────────────────────────────────────────────┐ │ REFUND EDGE CASES │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ CASE 1: Card expired / cancelled │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ Gateway returns: CARD_NOT_VALID │ │ │ │ │ │ │ │ Solution: │ │ │ │ 1. Try original payment method first │ │ │ │ 2. If fails, check for backup payment method │ │ │ │ 3. If no backup, credit to wallet │ │ │ │ 4. If no wallet, create wallet and credit │ │ │ │ 5. Notify user: "Refund credited to wallet" │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ CASE 2: Partial refund already done │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ Order: ₹500 │ │ │ │ Previous refund: ₹200 │ │ │ │ New refund request: ₹400 │ │ │ │ │ │ │ │ Solution: │ │ │ │ 1. Calculate remaining refundable: 500 - 200 = ₹300 │ │ │ │ 2. Return error: "Max refundable amount is ₹300" │ │ │ │ 3. OR auto-adjust: Refund ₹300 instead of ₹400 │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ CASE 3: Gateway timeout │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ We sent refund request, gateway didn't respond │ │ │ │ │ │ │ │ Solution: │ │ │ │ 1. Mark as PENDING_VERIFICATION │ │ │ │ 2. Background job checks gateway status every minute │ │ │ │ 3. Gateway API: GET /refunds/{our_refund_id}/status │ │ │ │ 4. Update our status based on gateway response │ │ │ │ 5. If gateway says "not found" → safe to retry │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ CASE 4: User requests refund during refund processing │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ Refund 1: PROCESSING (takes 30 sec) │ │ │ │ User clicks again → Refund 2 request │ │ │ │ │ │ │ │ Solution: │ │ │ │ Same idempotency key → Returns "Refund in progress" │ │ │ │ Different key (different amount) → Waits for first to complete │ │ │ │ Use distributed lock on order_id during refund │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
6. Design a Wallet System with Eventual Consistency
The Challenge
Unlike bank ledgers, wallets can tolerate eventual consistency for:
- Balance display (can be slightly stale)
- Transaction history (can lag by seconds)
But must be strongly consistent for:
- Actual balance deduction (no overdraft)
- Single transaction processing
Architecture
┌─────────────────────────────────────────────────────────────────────────────────┐ │ WALLET SYSTEM ARCHITECTURE │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ WRITE PATH (Strong Consistency) │ │ │ │ │ │ │ │ User Action ──▶ API Gateway ──▶ Wallet Service ──▶ Primary DB │ │ │ │ (Add/Deduct) │ │ │ │ │ │ │ │ • Serializable transaction │ │ │ │ • Balance check + update atomic │ │ │ │ • Event published to Kafka │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ READ PATH (Eventual Consistency) │ │ │ │ │ │ │ │ User Action ──▶ API Gateway ──▶ Cache (Redis) ──▶ Read Replica │ │ │ │ (View Balance) (if miss) │ │ │ │ │ │ │ │ • Cache TTL: 30 seconds │ │ │ │ • Replica lag: < 1 second │ │ │ │ • User sees balance updated within 1-30 seconds │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ ASYNC PROCESSING │ │ │ │ │ │ │ │ Kafka ──▶ Transaction History Service ──▶ History DB │ │ │ │ ──▶ Analytics Service ──▶ Data Warehouse │ │ │ │ ──▶ Notification Service ──▶ Push/Email │ │ │ │ │ │ │ │ • Transaction history updated async │ │ │ │ • Notifications sent async │ │ │ │ • Analytics processed in near-real-time │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
Eventual Consistency Implementation
gotype WalletService struct { primaryDB *sql.DB // For writes replicaDB *sql.DB // For reads redis *redis.Client kafkaProducer *kafka.Producer } // WRITE: Strong consistency func (s *WalletService) Deduct(ctx context.Context, userID string, amount decimal.Decimal, txnID string) error { tx, err := s.primaryDB.BeginTx(ctx, &sql.TxOptions{ Isolation: sql.LevelSerializable, }) if err != nil { return err } defer tx.Rollback() // Lock and check balance var balance decimal.Decimal err = tx.QueryRowContext(ctx, ` SELECT balance FROM wallets WHERE user_id = $1 FOR UPDATE `, userID).Scan(&balance) if err != nil { return err } if balance.LessThan(amount) { return ErrInsufficientBalance } // Deduct newBalance := balance.Sub(amount) _, err = tx.ExecContext(ctx, ` UPDATE wallets SET balance = $1, updated_at = NOW() WHERE user_id = $2 `, newBalance, userID) if err != nil { return err } // Record transaction _, err = tx.ExecContext(ctx, ` INSERT INTO wallet_transactions (id, user_id, type, amount, balance_after) VALUES ($1, $2, 'DEBIT', $3, $4) `, txnID, userID, amount, newBalance) if err != nil { return err } // Commit if err = tx.Commit(); err != nil { return err } // ASYNC: Invalidate cache (eventual) go s.redis.Del(context.Background(), "wallet:"+userID) // ASYNC: Publish event (eventual) go s.publishEvent(WalletDebitedEvent{ UserID: userID, Amount: amount, Balance: newBalance, Timestamp: time.Now(), }) return nil } // READ: Eventual consistency (cache + replica) func (s *WalletService) GetBalance(ctx context.Context, userID string) (decimal.Decimal, error) { // Try cache first cacheKey := "wallet:" + userID cached, err := s.redis.Get(ctx, cacheKey).Result() if err == nil { balance, _ := decimal.NewFromString(cached) return balance, nil } // Cache miss - read from replica var balance decimal.Decimal err = s.replicaDB.QueryRowContext(ctx, ` SELECT balance FROM wallets WHERE user_id = $1 `, userID).Scan(&balance) if err != nil { return decimal.Zero, err } // Cache for 30 seconds s.redis.Set(ctx, cacheKey, balance.String(), 30*time.Second) return balance, nil } // CRITICAL READ: When user is about to pay, get fresh balance func (s *WalletService) GetBalanceForPayment(ctx context.Context, userID string) (decimal.Decimal, error) { // Read from PRIMARY for critical operations var balance decimal.Decimal err := s.primaryDB.QueryRowContext(ctx, ` SELECT balance FROM wallets WHERE user_id = $1 `, userID).Scan(&balance) return balance, err }
Handling Consistency Trade-offs
┌─────────────────────────────────────────────────────────────────────────────────┐ │ CONSISTENCY TRADE-OFF DECISIONS │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ SCENARIO 1: User sees old balance, tries to pay │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ Actual balance: ₹50 │ │ │ │ Displayed (cached): ₹100 │ │ │ │ User tries to pay: ₹80 │ │ │ │ │ │ │ │ Handling: │ │ │ │ 1. Payment service reads FRESH balance from primary │ │ │ │ 2. ₹50 < ₹80 → Reject with "Insufficient balance" │ │ │ │ 3. Refresh displayed balance │ │ │ │ 4. User sees updated ₹50 │ │ │ │ │ │ │ │ UX: Slightly confusing but safe (no overdraft) │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ SCENARIO 2: User adds money, sees old balance │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ User adds ₹100 │ │ │ │ Primary updated: ₹150 │ │ │ │ Cache still shows: ₹50 │ │ │ │ │ │ │ │ Handling: │ │ │ │ 1. On successful add, invalidate cache immediately │ │ │ │ 2. Next read fetches from replica │ │ │ │ 3. If replica lagging, show "Balance updating..." │ │ │ │ 4. After 5 sec, force read from primary │ │ │ │ │ │ │ │ UX: Brief delay but eventually correct │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ │ SCENARIO 3: Concurrent deductions │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ Balance: ₹100 │ │ │ │ Request 1: Deduct ₹60 │ │ │ │ Request 2: Deduct ₹50 (arrives 10ms later) │ │ │ │ │ │ │ │ Handling (Serializable isolation): │ │ │ │ 1. Request 1 acquires row lock │ │ │ │ 2. Request 2 waits │ │ │ │ 3. Request 1 commits (balance = ₹40) │ │ │ │ 4. Request 2 checks ₹40 < ₹50 → Fails │ │ │ │ │ │ │ │ Result: Exactly one succeeds, no overdraft │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘
This completes Part 1 covering banking and high-scale system designs. The document continues with more system design topics in subsequent parts.