TCP/IP: The Foundation of Internet Communication

What Problem Does TCP/IP Solve?

1. Routing across networks (the IP part): How do you get a message from your laptop in San Francisco to a server in Tokyo, passing through dozens of intermediate routers?
2. Reliable delivery (the TCP part): How do you ensure that data arrives intact, in order, and without corruption—even when the underlying network randomly loses packets, duplicates them, or delivers them out of sequence?

Diagram 1

• Packets get lost: Network congestion, buffer overflows, or hardware failures
• Packets arrive out of order: Different packets take different paths
• Packets get corrupted: Electrical interference or bit errors
• Packets get duplicated: Routing loops or retransmissions

Real-World Analogy

1. You get a tracking number (sequence number)
2. The recipient must sign for it (acknowledgment)
3. If it doesn't arrive, it gets resent automatically (retransmission)
4. Multiple packages arrive in the correct order, even if shipped separately (ordering)
5. The postal service won't flood you with packages faster than you can process them (flow control)

The Solution

How Does TCP/IP Solve the Problem?

• Best-effort delivery: Gets packets from source to destination
• Addressing: Uses 32-bit (IPv4) or 128-bit (IPv6) addresses
• Routing: Each router makes independent forwarding decisions
• Fragmentation: Breaks large packets to fit network MTU (Maximum Transmission Unit)
• No guarantees: Packets can be lost, duplicated, corrupted, or reordered

• Reliability: Ensures all data arrives correctly
• Ordering: Delivers data in the correct sequence
• Error detection: Checksums detect corruption
• Flow control: Prevents overwhelming the receiver
• Congestion control: Prevents overwhelming the network
• Connection-oriented: Establishes state before data transfer

Diagram 2

1. Sequence numbers: Every byte gets a unique number, allowing detection of loss and reordering
2. Acknowledgments (ACKs): Receiver tells sender what was received successfully
3. Sliding window: Allows multiple unacknowledged segments in flight (pipelining)
4. Adaptive retransmission: Dynamically adjusts timeouts based on network conditions
5. Congestion signals: Uses packet loss as feedback to slow down

Building the Mental Model

The Complete TCP Connection Lifecycle

Diagram 3

• SYN: Client declares initial sequence number (ISN). Why? Each connection needs unique sequence numbers to prevent old duplicate packets from corrupting new connections.
• SYN-ACK: Server acknowledges client's ISN and declares its own ISN. Why two numbers? TCP is full-duplex; data flows both directions simultaneously.
• ACK: Client acknowledges server's ISN. Why needed? Without it, server doesn't know if its SYN-ACK arrived.

• Why four steps instead of three? TCP is full-duplex. One side finishing doesn't mean the other is done. The FIN and ACK from each side can't be combined because there might be a delay between receiving a FIN and being ready to send one.

• Why wait 2*MSL (Maximum Segment Lifetime)? To ensure the final ACK isn't lost. If the server doesn't receive the final ACK, it retransmits its FIN. The client must be around to re-ACK it. Also prevents old duplicate packets from corrupting a new connection using the same port numbers.

TCP State Machine

Diagram 4

Flow Control Mechanism

Diagram 5

• Without it, TCP would be "stop-and-wait": send one packet, wait for ACK, repeat. This wastes bandwidth, especially on high-latency networks.
• With sliding window, multiple packets can be "in flight" simultaneously, utilizing available bandwidth efficiently.

• The receiver's application might read data slowly (e.g., writing to disk, processing, waiting for user input).
• If TCP kept accepting data, the receive buffer would overflow, forcing packet drops.
• By advertising available buffer space, the receiver controls the flow.

Congestion Control Visualization

Diagram 6

• Starts conservatively (1 MSS - Maximum Segment Size, typically 1460 bytes) because TCP doesn't know network capacity.
• Doubles every RTT (Round Trip Time) to quickly discover available bandwidth.
• "Slow" is relative—it's exponential growth!

• Once near network capacity (ssthresh = slow start threshold), growth becomes linear.
• Why linear? Exponential growth would quickly cause congestion again.
• Adds 1 MSS per RTT, gently probing for more capacity.

• Three duplicate ACKs indicate packet loss but network still delivering (not totally congested).
• Halves window but doesn't drop to 1 like a timeout would.
• Why? Some packets are still getting through; don't be too aggressive.

• Indicates serious congestion—no ACKs arriving.
• Resets to slow start (cwnd = 1) to avoid making congestion worse.

Packet Structure Deep Dive

Diagram 7

• Identifies the byte position of the first data byte in this segment.
• Why 32 bits? Allows 4.3 billion unique sequence numbers. With sequence number wraparound, TCP can handle connections at 10 Gbps for hours without ambiguity.
• Initial sequence number (ISN) is randomized for security (prevents old duplicate packets from being accepted).

• The next sequence number the receiver expects.
• If Ack=5001, it means "I've received everything up to byte 5000."
• Cumulative ACK: Acknowledges all data up to this point, even if received out of order.

• Advertises receive buffer space (0-65,535 bytes).
• Limits throughput: max = window_size / RTT.
• TCP Window Scaling (option) extends this to 1 GB.

• SYN: Synchronize sequence numbers (connection setup)
• ACK: Acknowledgment field is valid
• FIN: No more data from sender (connection teardown)
• RST: Reset connection (error condition)
• PSH: Push data to application immediately
• URG: Urgent data present (rarely used)

Deep Technical Dive

Architecture Breakdown

Diagram 8

1. Application → TCP Socket API: Application calls write() or send(), passing data.
2. TCP Socket API → Send Buffer: Data is copied into the socket's send buffer (kernel memory).
3. TCP Protocol Engine → Segmentation:
   • Breaks data into Maximum Segment Size (MSS) chunks, typically 1460 bytes (1500 MTU - 20 IP header - 20 TCP header).
   • Assigns sequence numbers to each byte.
   • Calculates checksum.
4. TCP → Retransmission Queue: Keeps copy of sent-but-unacknowledged segments.
5. TCP → IP Layer: Passes segments to IP with destination address.
6. IP → Routing Table: Determines next hop (next router or final destination).
7. IP → Data Link Layer: Encapsulates in Ethernet/WiFi frame with MAC addresses.
8. Receiver Side (Reverse Flow):
   • Frame → IP packet → TCP segment
   • TCP checks sequence numbers, reorders if needed
   • Places in receive buffer
   • Application calls read() to retrieve data

Internal Mechanics

TCP Segment Structure in Detail

0000   45 00 00 3c 1c 46 40 00  40 06 b1 e6 c0 a8 01 64   E..<.F@.@......d
0010   c0 a8 01 65 04 d2 00 50  00 00 00 01 00 00 00 00   ...e...P........
0020   a0 02 72 10 fe 30 00 00  02 04 05 b4 04 02 08 0a   ..r..0..........
0030   00 00 00 00 00 00 00 00  01 03 03 07               ............

IP Header (bytes 0-19):
  45        Version=4, Header Length=5*4=20 bytes
  00        Type of Service (TOS)
  00 3c     Total Length = 60 bytes
  1c 46     Identification
  40 00     Flags=DF (Don't Fragment), Fragment Offset=0
  40        TTL = 64 hops
  06        Protocol = 6 (TCP)
  b1 e6     Header Checksum
  c0 a8 01 64    Source IP = 192.168.1.100
  c0 a8 01 65    Dest IP = 192.168.1.101

TCP Header (bytes 20-39+):
  04 d2     Source Port = 1234
  00 50     Dest Port = 80 (HTTP)
  00 00 00 01    Sequence Number = 1
  00 00 00 00    Ack Number = 0 (not valid, ACK flag not set)
  a0 02     Data Offset=10*4=40 bytes, Flags=SYN
  72 10     Window Size = 29,200 bytes
  fe 30     Checksum
  00 00     Urgent Pointer = 0
  
TCP Options (bytes 40-59):
  02 04 05 b4    MSS = 1460 bytes
  04 02          SACK Permitted
  08 0a 00 00 00 00 00 00 00 00    Timestamps
  01 03 03 07    Window Scale = 7 (multiply window by 128)

• Sequence = 1: This is a SYN packet (initial connection). The ISN could be any value; 1 is just an example.
• MSS = 1460: Ethernet MTU is 1500 bytes. Subtract 20 (IP) + 20 (TCP) = 1460 bytes for data.
• Window Scale: Without scaling, max window is 65 KB. With scale factor 7, max window = 65536 * 2^7 = 8 MB. Essential for high-bandwidth, high-latency networks.
• SACK (Selective Acknowledgment): Allows receiver to acknowledge non-contiguous blocks, improving performance when multiple packets are lost.

Memory Layout and Buffers

Diagram 9

• Bandwidth-Delay Product (BDP): Optimal buffer size = Bandwidth × RTT
  • Example: 100 Mbps, 100ms RTT → BDP = 1.25 MB
  • Buffer should be ≥ BDP to fully utilize bandwidth
  • Default 256 KB limits throughput to ~20 Mbps on high-latency links
• Buffer Bloat: Excessively large buffers cause high latency
  • Routers with multi-second buffers lead to "bufferbloat"
  • TCP congestion control relies on packet loss signals
  • Large buffers delay these signals, causing latency spikes

Protocol Specifications

TCP Port Numbers

• Well-known ports (0-1023): Require root/admin privileges
  • 20/21: FTP
  • 22: SSH
  • 25: SMTP
  • 80: HTTP
  • 443: HTTPS
• Registered ports (1024-49151): Application-specific
  • 3306: MySQL
  • 5432: PostgreSQL
  • 6379: Redis
  • 27017: MongoDB
• Dynamic/ephemeral ports (49152-65535): Client-side connections
  • OS assigns from this range for outgoing connections

• Uniquely identifies a TCP connection
• Allows 65K simultaneous connections per remote host

TCP Options

• RTT measurement: More accurate than relying on ACKs alone
• PAWS (Protect Against Wrapped Sequences): With high-speed networks, sequence numbers can wrap around in seconds. Timestamps disambiguate old vs. new data.

Code Deep Dive

Example 1: TCP Server in Go

go
// tcp_server.go
package main

import (
    "bufio"
    "fmt"
    "net"
    "os"
    "time"
)

func main() {
    // Listen on all interfaces, port 8080
    // Protocol: "tcp", "tcp4", or "tcp6"
    listener, err := net.Listen("tcp", ":8080")
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to listen: %v\n", err)
        os.Exit(1)
    }
    defer listener.Close()
    
    fmt.Println("Server listening on :8080")
    
    for {
        // Accept blocks until a client connects
        // Under the hood: completes 3-way handshake
        conn, err := listener.Accept()
        if err != nil {
            fmt.Fprintf(os.Stderr, "Failed to accept: %v\n", err)
            continue // Keep accepting other connections
        }
        
        // Handle each connection concurrently
        // Why goroutine? One blocked connection shouldn't block others
        go handleConnection(conn)
    }
}

func handleConnection(conn net.Conn) {
    // Defer ensures cleanup even if panic occurs
    defer conn.Close()
    
    // Get client address for logging
    clientAddr := conn.RemoteAddr().String()
    fmt.Printf("Client connected: %s\n", clientAddr)
    
    // Set read timeout to prevent infinite blocking
    // Why? Protects against slow-loris attacks, hung clients
    conn.SetReadDeadline(time.Now().Add(30 * time.Second))
    
    // Buffered reader reduces system calls
    // Default bufio size: 4096 bytes
    reader := bufio.NewReader(conn)
    
    for {
        // ReadString reads until delimiter or EOF
        // Why '\n'? Simple text protocol convention
        message, err := reader.ReadString('\n')
        if err != nil {
            // EOF means client closed connection gracefully
            if err.Error() == "EOF" {
                fmt.Printf("Client disconnected: %s\n", clientAddr)
            } else {
                fmt.Printf("Read error from %s: %v\n", clientAddr, err)
            }
            return
        }
        
        fmt.Printf("Received from %s: %s", clientAddr, message)
        
        // Echo back to client
        // Write buffers data; doesn't guarantee immediate transmission
        _, err = conn.Write([]byte("Echo: " + message))
        if err != nil {
            fmt.Printf("Write error to %s: %v\n", clientAddr, err)
            return
        }
        
        // Reset read deadline after successful operation
        conn.SetReadDeadline(time.Now().Add(30 * time.Second))
    }
}

1. net.Listen("tcp", ":8080"):
   • Creates a socket:
     socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)
   • Binds to port: bind(sockfd, {0.0.0.0, 8080})
   • Marks as passive socket: listen(sockfd, backlog)
   • Backlog (typically 128) = max pending connections in SYN_RECEIVED state
2. listener.Accept():
   • Blocks in system call: accept(sockfd, ...)
   • Kernel completes 3-way handshake with client
   • Returns new socket for established connection
   • Original listening socket remains open for new connections
3. reader.ReadString('\n'):
   • System call: read(connfd, buffer, size)
   • Blocks until data arrives in receive buffer
   • TCP handles buffering, reordering, retransmission
   • Application sees reliable byte stream
4. conn.Write([]byte(...)):
   • System call: write(connfd, buffer, length)
   • Data copied to socket send buffer (kernel space)
   • TCP segments and transmits asynchronously
   • Write returns immediately; doesn't wait for ACK

Example 2: TCP Client in Python

python
# tcp_client.py
import socket
import sys
import time

def main():
    # Create TCP socket
    # AF_INET = IPv4, SOCK_STREAM = TCP
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    
    # Set socket options
    # SO_KEEPALIVE: Send TCP keepalive probes
    # Why? Detect broken connections (router failure, cable unplugged)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    
    # TCP_NODELAY: Disable Nagle's algorithm
    # Nagle's algorithm: buffer small packets to reduce overhead
    # Why disable? For interactive applications needing low latency
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
    
    # Set connection timeout
    # Default is blocking forever, which is dangerous
    sock.settimeout(10.0)
    
    try:
        # Connect initiates 3-way handshake
        # Blocks until handshake completes or timeout
        print("Connecting to localhost:8080...")
        sock.connect(('localhost', 8080))
        print("Connected!")
        
        # Disable timeout for data transfer
        # We want blocking reads/writes now
        sock.settimeout(None)
        
        # Send data
        messages = [
            "Hello, server!",
            "How are you?",
            "Goodbye!"
        ]
        
        for msg in messages:
            # Encode string to bytes
            # TCP transports bytes, not characters
            data = (msg + '\n').encode('utf-8')
            
            # Send all data
            # Why sendall? send() might return after sending partial data
            # sendall() loops until all data sent or error occurs
            sock.sendall(data)
            print(f"Sent: {msg}")
            
            # Receive response
            # recv() returns up to 4096 bytes
            # May return less if less data available
            response = sock.recv(4096)
            print(f"Received: {response.decode('utf-8').strip()}")
            
            time.sleep(1)  # Pause between messages
            
    except socket.timeout:
        print("Connection timeout", file=sys.stderr)
        sys.exit(1)
    except ConnectionRefusedError:
        print("Connection refused - is server running?", file=sys.stderr)
        sys.exit(1)
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)
    finally:
        # Always close socket
        # Initiates graceful shutdown (FIN handshake)
        print("Closing connection...")
        sock.close()

if __name__ == '__main__':
    main()

1. socket.socket(AF_INET, SOCK_STREAM)

   :
   • System call:
     socket(AF_INET, SOCK_STREAM, 0)
   • OS allocates socket data structures
   • Returns file descriptor (Unix) or handle (Windows)
2. sock.connect(('localhost', 8080))

   :
   • Resolves hostname to IP (if needed)
   • Initiates 3-way handshake:
     • Sends SYN
     • Waits for SYN-ACK
     • Sends ACK
   • Blocks until ESTABLISHED state or timeout
3. sock.sendall(data):
   • Loops calling send() until all bytes sent
   • Each send() copies data to kernel send buffer
   • Returns when all data buffered, NOT when ACKed
4. sock.recv(4096):
   • System call: recv(sockfd, buffer, 4096, 0)
   • Blocks until at least 1 byte available
   • May return less than 4096 bytes
   • TCP stream has no message boundaries!

Example 3: Examining TCP State with netstat

bash
#!/bin/bash
# tcp_monitor.sh
# Monitor TCP connections and state transitions

echo "Starting TCP connection monitor..."
echo "===================================="

# Function to display TCP connections in a formatted way
monitor_tcp() {
    while true; do
        clear
        echo "TCP Connection States ($(date))"
        echo "----------------------------------------"
        
        # On Linux: use ss (socket statistics) - faster than netstat
        # On macOS: use netstat
        if command -v ss &> /dev/null; then
            # -t: TCP only
            # -n: Numeric addresses (no DNS lookup)
            # -a: All sockets (listening and established)
            # -o: Show timer information
            ss -tano | head -20
            
            echo ""
            echo "State Summary:"
            ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
        else
            netstat -an -p tcp | head -20
            
            echo ""
            echo "State Summary:"
            netstat -an -p tcp | awk 'NR>2 {print $6}' | sort | uniq -c | sort -rn
        fi
        
        echo ""
        echo "Press Ctrl+C to exit..."
        sleep 2
    done
}

# Trap Ctrl+C to exit cleanly
trap "echo 'Monitoring stopped.'; exit 0" INT

monitor_tcp

State Summary:
  42 ESTABLISHED    # Active data transfer connections
  18 TIME_WAIT     # Recently closed connections (2MSL wait)
   5 LISTEN        # Servers waiting for connections
   2 CLOSE_WAIT    # Remote closed, local app hasn't closed yet
   1 FIN_WAIT_2    # Local closed, waiting for remote FIN

• Each closed connection sits in TIME_WAIT for 2*MSL (120 seconds default)
• High-traffic servers accumulate thousands of TIME_WAIT sockets
• They consume port numbers from the ephemeral range
• Can exhaust ports: max ~64K connections per (client IP, server IP, server port)
• Solution: Enable SO_REUSEADDR, increase ephemeral port range, use connection pooling

Example 4: TCP Connection with Raw Sockets (C)

c
// tcp_raw.c
// Demonstrates low-level TCP connection using raw sockets
// Requires root/admin privileges

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <netinet/tcp.h>
#include <netinet/ip.h>
#include <sys/socket.h>

// TCP header structure
struct tcp_header {
    uint16_t source_port;
    uint16_t dest_port;
    uint32_t seq_num;
    uint32_t ack_num;
    uint8_t  data_offset;  // 4 bits: offset, 4 bits: reserved
    uint8_t  flags;        // TCP flags (SYN, ACK, FIN, etc.)
    uint16_t window;
    uint16_t checksum;
    uint16_t urgent_pointer;
};

// Pseudo header for checksum calculation
// Why? TCP checksum includes IP addresses (pseudo header)
// Provides additional error detection
struct pseudo_header {
    uint32_t source_ip;
    uint32_t dest_ip;
    uint8_t  reserved;
    uint8_t  protocol;     // 6 for TCP
    uint16_t tcp_length;
};

// Calculate TCP checksum
uint16_t calculate_checksum(void *data, int length) {
    uint16_t *buf = (uint16_t *)data;
    uint32_t sum = 0;
    
    // Add all 16-bit words
    while (length > 1) {
        sum += *buf++;
        length -= 2;
    }
    
    // Add leftover byte if odd length
    if (length == 1) {
        sum += *(uint8_t *)buf;
    }
    
    // Fold 32-bit sum to 16 bits
    while (sum >> 16) {
        sum = (sum & 0xFFFF) + (sum >> 16);
    }
    
    // One's complement
    return ~sum;
}

void send_syn_packet(const char *dest_ip, uint16_t dest_port) {
    int sockfd;
    struct sockaddr_in dest_addr;
    char packet[4096];
    
    // Create raw socket
    // IPPROTO_TCP: We're crafting TCP packets
    // Requires CAP_NET_RAW capability (root on Linux)
    sockfd = socket(AF_INET, SOCK_RAW, IPPROTO_TCP);
    if (sockfd < 0) {
        perror("socket() failed - are you root?");
        exit(1);
    }
    
    // Tell kernel we're providing IP header
    int one = 1;
    if (setsockopt(sockfd, IPPROTO_IP, IP_HDRINCL, &one, sizeof(one)) < 0) {
        perror("setsockopt() failed");
        exit(1);
    }
    
    // Zero out packet buffer
    memset(packet, 0, sizeof(packet));
    
    // Build IP header
    struct iphdr *ip_hdr = (struct iphdr *)packet;
    ip_hdr->version = 4;
    ip_hdr->ihl = 5;           // Header length: 5 * 4 = 20 bytes
    ip_hdr->tos = 0;
    ip_hdr->tot_len = htons(sizeof(struct iphdr) + sizeof(struct tcp_header));
    ip_hdr->id = htons(54321); // Identification
    ip_hdr->frag_off = 0;
    ip_hdr->ttl = 64;          // Time to live
    ip_hdr->protocol = IPPROTO_TCP;
    ip_hdr->saddr = inet_addr("192.168.1.100");  // Source IP
    ip_hdr->daddr = inet_addr(dest_ip);          // Dest IP
    ip_hdr->check = 0;         // Kernel fills this in
    
    // Build TCP header
    struct tcp_header *tcp_hdr = (struct tcp_header *)(packet + sizeof(struct iphdr));
    tcp_hdr->source_port = htons(12345);         // Arbitrary source port
    tcp_hdr->dest_port = htons(dest_port);
    tcp_hdr->seq_num = htonl(1000);              // Initial sequence number
    tcp_hdr->ack_num = 0;                        // No ACK yet
    tcp_hdr->data_offset = (5 << 4);             // 5 * 4 = 20 bytes, no options
    tcp_hdr->flags = 0x02;                       // SYN flag
    tcp_hdr->window = htons(65535);              // Max window size
    tcp_hdr->checksum = 0;                       // Calculate below
    tcp_hdr->urgent_pointer = 0;
    
    // Calculate TCP checksum using pseudo header
    struct pseudo_header psh;
    psh.source_ip = inet_addr("192.168.1.100");
    psh.dest_ip = inet_addr(dest_ip);
    psh.reserved = 0;
    psh.protocol = IPPROTO_TCP;
    psh.tcp_length = htons(sizeof(struct tcp_header));
    
    // Create buffer with pseudo header + TCP header
    char checksum_buf[4096];
    memcpy(checksum_buf, &psh, sizeof(psh));
    memcpy(checksum_buf + sizeof(psh), tcp_hdr, sizeof(struct tcp_header));
    
    tcp_hdr->checksum = calculate_checksum(checksum_buf, 
                                           sizeof(psh) + sizeof(struct tcp_header));
    
    // Destination address
    dest_addr.sin_family = AF_INET;
    dest_addr.sin_addr.s_addr = inet_addr(dest_ip);
    
    // Send packet
    if (sendto(sockfd, packet, ntohs(ip_hdr->tot_len), 0,
               (struct sockaddr *)&dest_addr, sizeof(dest_addr)) < 0) {
        perror("sendto() failed");
        exit(1);
    }
    
    printf("SYN packet sent to %s:%d\n", dest_ip, dest_port);
    printf("  Source: 192.168.1.100:12345\n");
    printf("  Seq: 1000\n");
    printf("  Flags: SYN\n");
    
    close(sockfd);
}

int main(int argc, char *argv[]) {
    if (argc != 3) {
        fprintf(stderr, "Usage: %s <dest_ip> <dest_port>\n", argv[0]);
        exit(1);
    }
    
    const char *dest_ip = argv[1];
    uint16_t dest_port = atoi(argv[2]);
    
    send_syn_packet(dest_ip, dest_port);
    
    return 0;
}

bash
gcc -o tcp_raw tcp_raw.c
sudo ./tcp_raw 192.168.1.101 80

• Understanding packet structure at this level helps debug network issues
• Tools like tcpdump, wireshark parse these same structures
• Security tools (firewalls, IDS) inspect these fields
• Network performance tuning requires understanding header overhead

Example 5: Monitoring TCP Metrics (JavaScript/Node.js)

javascript
// tcp_metrics.js
// Monitors TCP connection metrics and performance

const net = require('net');
const { performance } = require('perf_hooks');

class TCPMetrics {
    constructor() {
        this.connections = new Map();
        this.stats = {
            totalConnections: 0,
            activeConnections: 0,
            bytesReceived: 0,
            bytesSent: 0,
            errors: 0
        };
    }
    
    // Create a monitored TCP server
    createServer(port, callback) {
        const server = net.createServer((socket) => {
            const connId = `${socket.remoteAddress}:${socket.remotePort}`;
            const connMetrics = {
                id: connId,
                connectedAt: Date.now(),
                bytesReceived: 0,
                bytesSent: 0,
                rttSamples: [],
                errors: []
            };
            
            this.connections.set(connId, connMetrics);
            this.stats.totalConnections++;
            this.stats.activeConnections++;
            
            console.log(`[CONNECT] ${connId}`);
            
            // Monitor socket buffer sizes
            // These affect TCP window size and throughput
            console.log(`  Send buffer: ${socket.bufferSize} bytes`);
            console.log(`  Receive buffer: ${socket.readableHighWaterMark} bytes`);
            
            // Data received
            socket.on('data', (data) => {
                connMetrics.bytesReceived += data.length;
                this.stats.bytesReceived += data.length;
                
                // Measure RTT by echoing with timestamp
                const pingStart = performance.now();
                
                // Echo data back
                socket.write(data, () => {
                    const rtt = performance.now() - pingStart;
                    connMetrics.rttSamples.push(rtt);
                    
                    connMetrics.bytesSent += data.length;
                    this.stats.bytesSent += data.length;
                    
                    // Keep only last 100 samples
                    if (connMetrics.rttSamples.length > 100) {
                        connMetrics.rttSamples.shift();
                    }
                });
            });
            
            // Connection closed
            socket.on('end', () => {
                const duration = Date.now() - connMetrics.connectedAt;
                const avgRTT = connMetrics.rttSamples.reduce((a, b) => a + b, 0) / 
                               connMetrics.rttSamples.length;
                
                console.log(`[DISCONNECT] ${connId}`);
                console.log(`  Duration: ${duration}ms`);
                console.log(`  Bytes RX: ${connMetrics.bytesReceived}`);
                console.log(`  Bytes TX: ${connMetrics.bytesSent}`);
                console.log(`  Avg RTT: ${avgRTT.toFixed(2)}ms`);
                console.log(`  Errors: ${connMetrics.errors.length}`);
                
                this.connections.delete(connId);
                this.stats.activeConnections--;
            });
            
            // Error handling
            socket.on('error', (err) => {
                console.error(`[ERROR] ${connId}: ${err.message}`);
                connMetrics.errors.push({
                    timestamp: Date.now(),
                    error: err.message
                });
                this.stats.errors++;
            });
            
            // Timeout handling
            // Why? Detect idle connections that should be closed
            socket.setTimeout(60000); // 60 second timeout
            socket.on('timeout', () => {
                console.log(`[TIMEOUT] ${connId} - closing`);
                socket.end();
            });
            
            if (callback) {
                callback(socket, connMetrics);
            }
        });
        
        server.listen(port, () => {
            console.log(`Server listening on port ${port}`);
            this.startMetricsReporter();
        });
        
        return server;
    }
    
    // Periodically report aggregate metrics
    startMetricsReporter() {
        setInterval(() => {
            console.log('\n=== TCP Metrics Report ===');
            console.log(`Total connections: ${this.stats.totalConnections}`);
            console.log(`Active connections: ${this.stats.activeConnections}`);
            console.log(`Total bytes received: ${this.formatBytes(this.stats.bytesReceived)}`);
            console.log(`Total bytes sent: ${this.formatBytes(this.stats.bytesSent)}`);
            console.log(`Total errors: ${this.stats.errors}`);
            
            // Per-connection details
            if (this.connections.size > 0) {
                console.log('\nActive Connections:');
                for (const [id, metrics] of this.connections) {
                    const duration = Date.now() - metrics.connectedAt;
                    const avgRTT = metrics.rttSamples.reduce((a, b) => a + b, 0) / 
                                   metrics.rttSamples.length || 0;
                    console.log(`  ${id}: duration=${duration}ms, RTT=${avgRTT.toFixed(2)}ms, ` +
                               `RX=${this.formatBytes(metrics.bytesReceived)}, ` +
                               `TX=${this.formatBytes(metrics.bytesSent)}`);
                }
            }
            console.log('========================\n');
        }, 10000); // Report every 10 seconds
    }
    
    formatBytes(bytes) {
        if (bytes < 1024) return `${bytes} B`;
        if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(2)} KB`;
        return `${(bytes / (1024 * 1024)).toFixed(2)} MB`;
    }
}

// Usage
const metrics = new TCPMetrics();
metrics.createServer(8080);

// Keep process running
process.on('SIGINT', () => {
    console.log('\nShutting down...');
    process.exit(0);
});

node tcp_metrics.js

bash
# Terminal 1
echo "Hello" | nc localhost 8080

# Terminal 2
echo "World" | nc localhost 8080

# Watch the metrics output

Visual Internals: Call Stack and System Calls

Diagram 10

Benefits & Why It Matters

Performance Benefits

• Pipelining: Sliding window allows multiple packets in flight, maximizing bandwidth utilization
• Selective acknowledgment (SACK): Recovers from multiple packet losses efficiently
• Window scaling: Supports window sizes up to 1 GB, essential for high-speed networks
• Fast retransmit: Detects loss after 3 duplicate ACKs, avoiding timeout delays

• Fast open (TFO): Sends data in initial SYN packet, saves 1 RTT
• TCP_NODELAY: Disables Nagle's algorithm for interactive applications
• Keep-alive: Detects dead connections without application-level polling

Diagram 11

• Netflix: Serves 250+ million streams daily over TCP. Uses congestion control algorithms (BBR) to maximize throughput while minimizing bufferbloat.
• Google: Developed QUIC (TCP replacement over UDP) but still uses TCP for most services. BBR congestion control increased throughput by 2-25x in some regions.
• AWS: Uses optimized TCP stacks for inter-region replication, achieving 100 Gbps+ transfers.

Developer Experience

• No manual retransmission: TCP handles it automatically
• No message boundaries: Can send/receive arbitrary chunks
• No ordering concerns: TCP delivers bytes in order
• No corruption handling: Checksums ensure integrity

• Data arrives intact or connection fails—no silent corruption
• No duplicate data delivered to application
• No gaps in data stream

• Every programming language has TCP socket libraries
• Extensive tooling: tcpdump, wireshark, netstat, ss
• Well-understood by operations teams
• Firewall-friendly (vs. UDP which is often blocked)

Scalability

• Modern servers handle millions of concurrent TCP connections
• Linux tuning: increase file descriptor limits, socket buffers, port range
• epoll/kqueue enable efficient event-driven I/O

• Billions of TCP connections active simultaneously
• Works across diverse network conditions (satellite, mobile, fiber)
• Congestion control prevents internet meltdown

Trade-offs & Gotchas

When to Use TCP

1. Reliability is critical: Financial transactions, file transfers, database queries
2. Ordered delivery required: Protocol state machines (HTTP, SSH, FTP)
3. Variable data sizes: Streaming arbitrary amounts of data
4. Firewall traversal: TCP widely allowed, UDP often blocked
5. Mature ecosystem needed: Extensive tooling and libraries

• Web applications (HTTP/HTTPS)
• API services (REST, GraphQL)
• Database connections (PostgreSQL, MySQL)
• File transfer (FTP, SFTP, rsync)
• Email (SMTP, IMAP)
• Remote access (SSH, RDP)

When NOT to Use TCP

1. Latency-sensitive real-time apps: Gaming, VoIP, video conferencing
   • Why: Head-of-line blocking—if one packet is lost, all subsequent packets wait for retransmission, causing latency spikes
   • Alternative: UDP with application-level selective retransmission
2. Broadcast/multicast: Sending to multiple recipients
   • Why: TCP is connection-oriented (one-to-one)
   • Alternative: UDP multicast
3. Simple request/response: DNS queries, SNMP, DHCP
   • Why: TCP overhead (3-way handshake) doubles latency
   • Alternative: UDP
4. Lossy networks with time-sensitive data: Live video streaming
   • Why: Retransmitting old video frames is useless; better to skip and continue
   • Alternative: UDP with forward error correction
5. Extremely high packet rate: High-frequency trading, real-time telemetry
   • Why: Per-packet TCP overhead (ACKs, state management)
   • Alternative: UDP or kernel-bypass networking (DPDK)

Common Mistakes

1. Assuming send() means data was delivered

python
# Wrong: assuming send completes delivery
sock.send(data)
# Data might not be sent yet!

# Right: use sendall and handle errors
try:
    sock.sendall(data)
except socket.error as e:
    # Connection broke before all data sent
    handle_error(e)

2. Ignoring partial reads

go
// Wrong: assuming full message received
data := make([]byte, 1024)
n, _ := conn.Read(data)
// n might be less than 1024!

// Right: loop until full message received
func readExactly(conn net.Conn, size int) ([]byte, error) {
    buf := make([]byte, size)
    offset := 0
    for offset < size {
        n, err := conn.Read(buf[offset:])
        if err != nil {
            return nil, err
        }
        offset += n
    }
    return buf, nil
}

3. Not handling TIME_WAIT exhaustion

python
# Enable socket reuse
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

# Use connection pooling
# Don't open/close connections for each request

bash
# Linux: Increase ephemeral port range
sysctl -w net.ipv4.ip_local_port_range="10000 65000"

# Reduce TIME_WAIT duration (risky!)
sysctl -w net.ipv4.tcp_fin_timeout=30

4. Small writes causing poor performance (Nagle's algorithm)

c
// Disable Nagle's algorithm
int flag = 1;
setsockopt(sockfd, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof(flag));

5. Not setting socket timeouts

python
# Set timeouts
sock.settimeout(30.0)  # 30 second timeout

# Or use SO_RCVTIMEO / SO_SNDTIMEO
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, 
                struct.pack('LL', 30, 0))

6. Misunderstanding window size limits

BashPYTHON
# Linux: Increase buffer sizes
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

7. Ignoring TCP keep-alive configuration

c
// Enable keep-alive
int optval = 1;
setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &optval, sizeof(optval));

// Set aggressive timings (Linux)
optval = 60;  // Start probing after 60s idle
setsockopt(sockfd, IPPROTO_TCP, TCP_KEEPIDLE, &optval, sizeof(optval));

optval = 10;  // Probe every 10s
setsockopt(sockfd, IPPROTO_TCP, TCP_KEEPINTVL, &optval, sizeof(optval));

optval = 3;  // 3 failed probes = dead connection
setsockopt(sockfd, IPPROTO_TCP, TCP_KEEPCNT, &optval, sizeof(optval));

8. Not handling RST packets gracefully

• Connecting to closed port
• Sending data after remote closed
• Network middle boxes (firewalls, load balancers) timeout connection

go
n, err := conn.Write(data)
if err != nil {
    // Check for connection reset
    if errors.Is(err, syscall.ECONNRESET) {
        // Handle gracefully: reconnect, log, alert
        log.Printf("Connection reset by peer")
    }
}

Performance Considerations

1. CPU: Checksum calculations, encryption (TLS), system call overhead
2. Memory: Socket buffers, per-connection state
3. Network bandwidth: Obvious but often forgotten
4. Port exhaustion: TIME_WAIT sockets consuming ephemeral ports
5. File descriptor limits: Default 1024 on many systems

Diagram 12

bash
# Test throughput
iperf3 -c server_ip -t 60 -P 4

# Test latency
ping -c 100 server_ip

# Measure packet loss
mtr -c 100 server_ip

# Monitor TCP stats
ss -tin

Security Considerations

SYN Flood Attack

bash
# Enable SYN cookies (Linux)
sysctl -w net.ipv4.tcp_syncookies=1

# Reduce SYN-RECV timeout
sysctl -w net.ipv4.tcp_synack_retries=2

Connection Hijacking

• Random ISN (implemented since 1990s)
• IPsec or TLS for authentication and encryption

RST Injection

• TCP MD5 signatures (RFC 2385) for BGP sessions
• TLS protects against injection

Amplification Attacks

• Egress filtering (BCP 38) to prevent source IP spoofing
• Rate limiting

Best Practices

1. Always use TLS for sensitive data (HTTPS, not HTTP)
2. Validate input at application layer—TCP doesn't protect against malicious data
3. Rate limit new connections to prevent resource exhaustion
4. Monitor connection states and metrics for anomalies
5. Harden OS TCP/IP stack (disable unused features, tune parameters)

Comparison with Alternatives

Diagram 13

• TCP → QUIC: Modern web applications needing low latency and multiplexing (HTTP/3)
• TCP → UDP: Real-time gaming, VoIP, live streaming (implement own reliability as needed)
• TCP → SCTP: Multi-homing, multi-streaming scenarios (telecom)
• TCP → WebSockets over TCP: Real-time web apps needing bidirectional communication

Hands-On Examples

Example 1: Simple HTTP Server (Understanding HTTP over TCP)

go
// http_tcp_server.go
// Implements a minimal HTTP server over raw TCP to understand the protocol
package main

import (
    "bufio"
    "fmt"
    "net"
    "strings"
    "time"
)

func main() {
    listener, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }
    defer listener.Close()
    
    fmt.Println("HTTP server listening on :8080")
    fmt.Println("Try: curl http://localhost:8080/")
    
    for {
        conn, err := listener.Accept()
        if err != nil {
            fmt.Println("Accept error:", err)
            continue
        }
        go handleHTTPRequest(conn)
    }
}

func handleHTTPRequest(conn net.Conn) {
    defer conn.Close()
    
    reader := bufio.NewReader(conn)
    
    // Read request line: GET /path HTTP/1.1
    requestLine, err := reader.ReadString('\n')
    if err != nil {
        fmt.Println("Error reading request:", err)
        return
    }
    
    parts := strings.Fields(requestLine)
    if len(parts) < 3 {
        sendResponse(conn, 400, "Bad Request", "Invalid request line")
        return
    }
    
    method := parts[0]
    path := parts[1]
    version := parts[2]
    
    fmt.Printf("Request: %s %s %s\n", method, path, version)
    
    // Read headers
    headers := make(map[string]string)
    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            return
        }
        
        line = strings.TrimSpace(line)
        if line == "" {
            break // Empty line indicates end of headers
        }
        
        parts := strings.SplitN(line, ":", 2)
        if len(parts) == 2 {
            key := strings.TrimSpace(parts[0])
            value := strings.TrimSpace(parts[1])
            headers[key] = value
            fmt.Printf("  Header: %s = %s\n", key, value)
        }
    }
    
    // Route handling
    switch path {
    case "/":
        sendResponse(conn, 200, "OK", "<h1>Welcome!</h1><p>TCP/IP + HTTP = Magic</p>")
    case "/time":
        timeStr := time.Now().Format(time.RFC3339)
        sendResponse(conn, 200, "OK", fmt.Sprintf("<h1>Current Time</h1><p>%s</p>", timeStr))
    case "/headers":
        body := "<h1>Your Headers</h1><ul>"
        for k, v := range headers {
            body += fmt.Sprintf("<li><b>%s:</b> %s</li>", k, v)
        }
        body += "</ul>"
        sendResponse(conn, 200, "OK", body)
    default:
        sendResponse(conn, 404, "Not Found", "<h1>404 Not Found</h1>")
    }
}

func sendResponse(conn net.Conn, statusCode int, statusText string, body string) {
    response := fmt.Sprintf("HTTP/1.1 %d %s\r\n", statusCode, statusText)
    response += "Content-Type: text/html; charset=utf-8\r\n"
    response += fmt.Sprintf("Content-Length: %d\r\n", len(body))
    response += "Connection: close\r\n" // Tell client we're closing after response
    response += "\r\n" // Empty line separates headers from body
    response += body
    
    // Write entire response
    // Under the hood: TCP segments this, handles ACKs, retransmissions
    _, err := conn.Write([]byte(response))
    if err != nil {
        fmt.Println("Error writing response:", err)
    }
}

bash
# Terminal 1: Run server
go run http_tcp_server.go

# Terminal 2: Test with curl
curl http://localhost:8080/
curl http://localhost:8080/time
curl http://localhost:8080/headers

# See raw HTTP with netcat
echo -e "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" | nc localhost 8080

• HTTP is a text protocol running over TCP
• Request/response structure
• How Content-Length determines body boundaries (TCP has no message boundaries)
• Connection management (Connection: close)

Example 2: TCP Chat Application

PYTHONPYTHON
# chat_server.py
import socket
import threading
import sys

class ChatServer:
    def __init__(self, host='0.0.0.0', port=9999):
        self.host = host
        self.port = port
        self.clients = {}  # {connection: username}
        self.lock = threading.Lock()
        
    def start(self):
        server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        server.bind((self.host, self.port))
        server.listen(5)
        
        print(f"Chat server listening on {self.host}:{self.port}")
        
        try:
            while True:
                conn, addr = server.accept()
                print(f"New connection from {addr}")
                thread = threading.Thread(target=self.handle_client, args=(conn, addr))
                thread.daemon = True
                thread.start()
        except KeyboardInterrupt:
            print("\nShutting down...")
            server.close()
    
    def handle_client(self, conn, addr):
        try:
            # Request username
            conn.send(b"Enter your username: ")
            username = conn.recv(1024).decode('utf-8').strip()
            
            if not username:
                conn.close()
                return
            
            # Register client
            with self.lock:
                self.clients[conn] = username
            
            # Announce join
            join_msg = f"*** {username} has joined the chat ***\n"
            self.broadcast(join_msg, exclude=conn)
            print(f"{username} joined from {addr}")
            
            # Send welcome message
            conn.send(f"Welcome, {username}! ({len(self.clients)} users online)\n".encode())
            
            # Handle messages
            while True:
                data = conn.recv(4096)
                if not data:
                    break  # Client disconnected
                
                message = data.decode('utf-8').strip()
                if message:
                    formatted = f"[{username}] {message}\n"
                    print(formatted.strip())
                    self.broadcast(formatted, exclude=conn)
        
        except Exception as e:
            print(f"Error with {addr}: {e}")
        finally:
            # Client disconnected
            with self.lock:
                if conn in self.clients:
                    username = self.clients[conn]
                    del self.clients[conn]
                    leave_msg = f"*** {username} has left the chat ***\n"
                    self.broadcast(leave_msg)
                    print(f"{username} disconnected")
            conn.close()
    
    def broadcast(self, message, exclude=None):
        """Send message to all clients except 'exclude'"""
        with self.lock:
            for client_conn in list(self.clients.keys()):
                if client_conn != exclude:
                    try:
                        client_conn.send(message.encode('utf-8'))
                    except:
                        # Client disconnected, remove it
                        if client_conn in self.clients:
                            del self.clients[client_conn]

if __name__ == '__main__':
    server = ChatServer()
    server.start()

bash
# Terminal 1: Start server
python chat_server.py

# Terminal 2: Client 1
python chat_client.py
Alice
Hello everyone!

# Terminal 3: Client 2
python chat_client.py
Bob
Hi Alice!

# Terminal 4: Client 3
python chat_client.py
Charlie
Hey folks!

• Multi-client server architecture
• Threading for concurrent connections
• Broadcast messaging patterns
• Connection lifecycle management
• Buffer management (partial reads)

Example 3: Performance Testing Tool

go
// tcp_bench.go
// Benchmark TCP throughput and latency
package main

import (
    "flag"
    "fmt"
    "io"
    "net"
    "sync"
    "time"
)

var (
    mode       = flag.String("mode", "server", "Mode: server or client")
    host       = flag.String("host", "localhost", "Host address")
    port       = flag.Int("port", 5001, "Port number")
    duration   = flag.Int("duration", 10, "Test duration in seconds")
    bufferSize = flag.Int("buffer", 32*1024, "Buffer size in bytes")
    parallel   = flag.Int("parallel", 1, "Number of parallel connections")
)

type Stats struct {
    bytesTransferred int64
    startTime        time.Time
    endTime          time.Time
    mu               sync.Mutex
}

func (s *Stats) add(bytes int) {
    s.mu.Lock()
    s.bytesTransferred += int64(bytes)
    s.mu.Unlock()
}

func (s *Stats) report() {
    duration := s.endTime.Sub(s.startTime).Seconds()
    throughputMbps := float64(s.bytesTransferred*8) / duration / 1_000_000
    
    fmt.Printf("\n--- Results ---\n")
    fmt.Printf("Duration: %.2f seconds\n", duration)
    fmt.Printf("Data transferred: %.2f MB\n", float64(s.bytesTransferred)/1_000_000)
    fmt.Printf("Throughput: %.2f Mbps\n", throughputMbps)
}

func runServer() {
    addr := fmt.Sprintf(":%d", *port)
    listener, err := net.Listen("tcp", addr)
    if err != nil {
        panic(err)
    }
    defer listener.Close()
    
    fmt.Printf("Server listening on %s\n", addr)
    fmt.Printf("Buffer size: %d bytes\n", *bufferSize)
    
    for {
        conn, err := listener.Accept()
        if err != nil {
            fmt.Println("Accept error:", err)
            continue
        }
        
        go handleServerConnection(conn)
    }
}

func handleServerConnection(conn net.Conn) {
    defer conn.Close()
    
    fmt.Printf("Client connected: %s\n", conn.RemoteAddr())
    
    stats := &Stats{startTime: time.Now()}
    buffer := make([]byte, *bufferSize)
    
    // Read all data from client
    for {
        n, err := conn.Read(buffer)
        if err != nil {
            if err != io.EOF {
                fmt.Println("Read error:", err)
            }
            break
        }
        stats.add(n)
    }
    
    stats.endTime = time.Now()
    
    fmt.Printf("Client %s disconnected\n", conn.RemoteAddr())
    stats.report()
}

func runClient() {
    addr := fmt.Sprintf("%s:%d", *host, *port)
    
    fmt.Printf("Connecting to %s\n", addr)
    fmt.Printf("Test duration: %d seconds\n", *duration)
    fmt.Printf("Parallel connections: %d\n", *parallel)
    fmt.Printf("Buffer size: %d bytes\n", *bufferSize)
    
    var wg sync.WaitGroup
    stats := &Stats{startTime: time.Now()}
    
    // Launch parallel connections
    for i := 0; i < *parallel; i++ {
        wg.Add(1)
        go func(id int) {
            defer wg.Done()
            runClientConnection(id, stats)
        }(i)
    }
    
    wg.Wait()
    stats.endTime = time.Now()
    
    stats.report()
}

func runClientConnection(id int, stats *Stats) {
    addr := fmt.Sprintf("%s:%d", *host, *port)
    
    conn, err := net.Dial("tcp", addr)
    if err != nil {
        fmt.Printf("Connection %d failed: %v\n", id, err)
        return
    }
    defer conn.Close()
    
    fmt.Printf("Connection %d established\n", id)
    
    buffer := make([]byte, *bufferSize)
    for i := range buffer {
        buffer[i] = byte(i % 256)
    }
    
    deadline := time.Now().Add(time.Duration(*duration) * time.Second)
    
    for time.Now().Before(deadline) {
        n, err := conn.Write(buffer)
        if err != nil {
            fmt.Printf("Connection %d write error: %v\n", id, err)
            break
        }
        stats.add(n)
    }
    
    fmt.Printf("Connection %d finished\n", id)
}

func main() {
    flag.Parse()
    
    if *mode == "server" {
        runServer()
    } else if *mode == "client" {
        runClient()
    } else {
        fmt.Println("Invalid mode. Use 'server' or 'client'")
    }
}

bash
# Compile
go build tcp_bench.go

# Terminal 1: Start server
./tcp_bench -mode server

# Terminal 2: Run client
./tcp_bench -mode client -duration 10 -parallel 4 -buffer 65536

# Test different scenarios
./tcp_bench -mode client -parallel 1 -buffer 1024    # Small buffers
./tcp_bench -mode client -parallel 10 -buffer 65536  # Multiple connections