Part 41: Security in Distributed Systems - Protecting the Extended Surface

A monolithic application running on a single server has a relatively contained security surface. The network perimeter is clear: protect the entry points, and the interior is implicitly trusted. But distributed systems shatter this model. Services communicate over networks. Components run on many machines. Data flows between systems controlled by different teams or organizations.

Every network connection is a potential attack vector. Every service is a potential target. Every message might be intercepted, modified, or forged. The interior of the system—once implicitly trusted—becomes a collection of mutually suspicious components that must verify each other's identity and authority.

This expansion of the attack surface doesn't make security impossible, but it changes how we must approach it. Instead of building walls around a perimeter, we must build security into every interaction, every component, every layer.

The foundation of distributed system security is knowing who is making requests. In a world where anyone might claim to be anyone, establishing identity is the first challenge.

Service-to-service authentication verifies that the service making a request is who it claims to be. When Service A calls Service B, how does B know it's really A and not an attacker impersonating A?

Mutual TLS (mTLS) provides strong service identity. Each service has a certificate issued by a trusted certificate authority. When services connect, they exchange and verify certificates. Both sides know who they're talking to. The TLS connection also encrypts traffic, protecting against eavesdropping.

Certificate management is the operational challenge of mTLS. Certificates must be issued, distributed to services, and rotated before expiration. Certificate authorities must be secured—a compromised CA can issue certificates for any identity. Service meshes and cloud platforms often provide certificate management infrastructure.

Service accounts and API keys are simpler alternatives. Each service has credentials it presents with requests. The receiving service validates these credentials. This approach is easier to implement but provides weaker guarantees—keys can be stolen or leaked more easily than certificates.

User authentication establishes the identity of human users. OAuth and OpenID Connect have become standard protocols, enabling users to authenticate once and access multiple services. JSON Web Tokens (JWTs) carry authenticated identity information between services.

Authentication establishes identity; authorization determines what that identity is allowed to do. In distributed systems, authorization decisions happen throughout the system, not just at the edge.

Role-based access control (RBAC) assigns permissions to roles, and roles to identities. An "admin" role might have all permissions; a "viewer" role might only read data. This model is widely used and well understood.

Attribute-based access control (ABAC) makes decisions based on attributes of the request, resource, and environment. "Users can only access data from their own department during business hours" is an ABAC policy. ABAC is more flexible than RBAC but more complex to manage.

Policy enforcement must happen at every service. If Service B has sensitive operations, it must check authorization even if the request came from Service A. Service A might have been compromised, or the end user making the request through A might not be authorized.

Policy decision points (PDPs) and policy enforcement points (PEPs) separate concerns. The PDP—often a centralized policy service—makes authorization decisions. The PEP—typically within each service—enforces those decisions. This architecture allows centralized policy management while distributing enforcement.

Distributed systems have many secrets: database passwords, API keys, encryption keys, certificates, and more. Managing these secrets securely is critical.

Never store secrets in code or configuration files that are checked into version control. This is the most common secrets management failure. Once a secret is in a Git repository, it's in the repository's entire history, even if you delete it later.

Dedicated secrets management systems—like HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets—provide secure storage for secrets. Services retrieve secrets at runtime rather than having them embedded. Access to secrets is controlled and audited.

Secret rotation limits the impact of compromise. If a secret is rotated regularly—perhaps daily or weekly—a stolen secret is only useful briefly. Rotation requires systems that can update secrets without downtime, which complicates implementation.

Least privilege applies to secrets too. Services should only have access to the secrets they need. A service that processes payments doesn't need access to analytics API keys. Scoped access limits damage if a service is compromised.

Encryption protects data confidentiality. In distributed systems, encryption applies at multiple layers.

Encryption in transit protects data as it moves between components. TLS encrypts network connections. Without it, anyone who can observe network traffic—compromised routers, malicious insiders, or sophisticated attackers—can read everything.

Encryption at rest protects stored data. Databases, file systems, and backups should encrypt their contents. If storage media is stolen or improperly disposed of, encrypted data remains protected.

End-to-end encryption protects data even from intermediate services. If Service A encrypts data that only Service C can decrypt, Service B—which might handle the data in between—cannot read it. This limits trust requirements but complicates processing.

Key management is the hard part of encryption. Keys must be generated, stored securely, distributed to services that need them, rotated periodically, and eventually retired. Losing a key means losing access to encrypted data. Compromising a key means compromising everything encrypted with it.

Despite internal networks seeming protected, network security within distributed systems remains important. The "zero trust" model assumes that internal networks are not safe—any communication might be observed or manipulated by an attacker who has gained internal access.

Network segmentation limits lateral movement. If an attacker compromises one service, segmentation prevents them from easily reaching others. Services are grouped into segments, and traffic between segments is controlled by firewalls.

Service meshes provide network security features. They automatically apply mTLS to service-to-service traffic. They enforce policies about which services can communicate. They provide visibility into network traffic patterns.

Defense in depth applies multiple security measures at different layers. Network encryption, authentication, authorization, input validation, output encoding—each layer stops some attacks. An attacker must bypass all layers, not just one.

Every input from an untrusted source is a potential attack vector. Distributed systems have many such inputs: API requests, messages from queues, data from databases, responses from other services.

Injection attacks exploit inputs that are incorporated into commands or queries. SQL injection occurs when user input becomes part of a SQL query. Command injection occurs when input becomes part of a shell command. The solution is the same: never construct commands by concatenating untrusted input. Use parameterized queries, prepared statements, and safe APIs.

Input validation checks that inputs meet expected formats and constraints. An email address field should contain an email address. A quantity field should contain a positive integer. Validation catches malformed inputs early, before they can cause problems.

Output encoding prevents injected content from being interpreted as code. When displaying user-provided content in HTML, encode special characters so they're displayed literally rather than interpreted as HTML. This prevents cross-site scripting (XSS) attacks.

Schema validation ensures that messages between services conform to expected structures. If Service A expects a specific JSON structure, validate incoming messages against that schema. Malformed messages are rejected before processing.

Security requires visibility. You need to know what's happening in your system—who accessed what, what changes were made, what anomalies occurred.

Audit logs record security-relevant events. Authentication attempts (successful and failed), authorization decisions, configuration changes, access to sensitive data—all should be logged. The logs should include who, what, when, and from where.

Log integrity is crucial. If an attacker can modify or delete audit logs, they can hide their activities. Logs should be stored securely, ideally shipped to a separate system where they're protected from modification.

Log analysis detects anomalies and attacks. Unexpected patterns—many failed login attempts, access from unusual locations, unusual data access patterns—might indicate attacks in progress. Security information and event management (SIEM) systems collect and analyze logs for threats.

Retention policies balance utility against cost and privacy. Logs are useful for investigating incidents but consume storage and might contain sensitive information. Retention periods should match your incident investigation needs and compliance requirements.

Distributed systems often handle regulated data: personal information protected by GDPR or CCPA, health data protected by HIPAA, payment data protected by PCI DSS. Security measures must satisfy compliance requirements.

Data classification identifies what protection different data requires. Not all data is equally sensitive. Personal identifiable information (PII) requires specific protections. Public data might require none. Classification guides where data can be stored, who can access it, and what protections apply.

Data minimization limits what data is collected and retained. Collect only what's needed. Retain only as long as necessary. Delete when no longer required. Less data means less to protect and less exposure if a breach occurs.

Consent and access controls implement privacy requirements. Users might need to consent to data collection. They might have rights to access, correct, or delete their data. Systems must support these operations.

Geographic restrictions apply to some data. Some regulations require data to stay within certain jurisdictions. Distributed systems spanning multiple regions must ensure data doesn't flow where it shouldn't.

Security must be tested, not just designed. Testing reveals vulnerabilities that design reviews miss.

Penetration testing simulates attacks on your system. Security experts try to breach your defenses, using the same techniques real attackers would use. The findings reveal vulnerabilities to fix.

Static analysis examines code for security issues without executing it. Tools identify potential injection vulnerabilities, insecure configurations, and dangerous function calls. Static analysis catches common issues cheaply.

Dynamic analysis tests running systems. Fuzzers send malformed inputs to find crashes or unexpected behaviors. Vulnerability scanners check for known weaknesses in dependencies or configurations.

Dependency scanning identifies vulnerabilities in libraries you depend on. Most code uses open-source libraries, and libraries have vulnerabilities. Scanning tools alert you when vulnerable versions are in use.

Security in distributed systems isn't a checklist you complete; it's a mindset you adopt. Every design decision has security implications. Every change might introduce vulnerabilities.

Assume breach: design as if attackers will get in, and limit what they can do once inside. Defense in depth, least privilege, and segmentation all reflect this assumption.

Security is everyone's responsibility. Developers, operators, and security specialists all contribute. Security reviews should be part of design and deployment processes, not afterthoughts.

Stay informed. New vulnerabilities emerge constantly. Security practices evolve. Staying current on threats and defenses is essential.