API Gateway & Service Mesh Guide

The Problem: Traffic Management at Scale

When your application evolves from a monolith to microservices, you quickly face a traffic management challenge. How do you handle authentication, rate limiting, request routing, and observability across dozens of services?

Two complementary tools solve this: **API Gateways** for north-south traffic (external clients to your services) and **Service Meshes** for east-west traffic (service-to-service communication).

External Traffic (North-South):
Client --> API Gateway --> Service A --> Service B

Internal Traffic (East-West): Service A <--> Service B <--> Service C (Service Mesh) ```

API Gateways: Your Front Door

An API gateway sits between your clients and backend services. It handles cross-cutting concerns so individual services don't have to.

What an API Gateway Does

**Authentication and Authorization**: Validate JWT tokens, API keys, or OAuth flows before requests reach your services.

**Rate Limiting**: Protect backend services from abuse. Critical for Indian SaaS products serving global clients.

**Request Routing**: Route `/api/v2/users` to the user service and `/api/v2/orders` to the order service.

**SSL Termination**: Handle TLS at the gateway so backend services communicate over internal networks.

**Request/Response Transformation**: Add headers, modify payloads, aggregate responses from multiple services.

Kong API Gateway Configuration

Kong is one of the most popular open-source API gateways. Here's a production setup:

# Kong service definition
curl -X POST http://kong-admin:8001/services \
    --data name=order-service \
    --data url=http://order-service.internal:3000

# Route configuration curl -X POST http://kong-admin:8001/services/order-service/routes \ --data name=order-routes \ --data 'paths[]=/api/v2/orders' \ --data 'methods[]=GET' \ --data 'methods[]=POST' \ --data 'methods[]=PUT' \ --data 'methods[]=DELETE'

# Rate limiting plugin curl -X POST http://kong-admin:8001/services/order-service/plugins \ --data name=rate-limiting \ --data config.minute=1000 \ --data config.policy=redis \ --data config.redis_host=redis.internal

# JWT authentication curl -X POST http://kong-admin:8001/services/order-service/plugins \ --data name=jwt \ --data config.uri_param_names=jwt \ --data config.header_names=Authorization

# CORS plugin curl -X POST http://kong-admin:8001/services/order-service/plugins \ --data name=cors \ --data config.origins=https://app.serverraja.com \ --data config.methods=GET,POST,PUT,DELETE \ --data config.max_age=3600 ```

Nginx as a Lightweight API Gateway

For simpler setups, Nginx can serve as an effective API gateway:

http {
    # Rate limiting zones
    limit_req_zone $binary_remote_addr zone=api_general:10m rate=100r/s;
    limit_req_zone $http_api_key zone=api_key:10m rate=1000r/s;

# JWT validation via auth_request server { listen 443 ssl http2; server_name api.serverraja.com;

ssl_certificate /etc/ssl/api.crt; ssl_certificate_key /etc/ssl/api.key;

# Validate JWT before routing location /api/ { auth_request /auth; auth_request_set $user_id $upstream_http_x_user_id; auth_request_set $user_role $upstream_http_x_user_role;

# Apply rate limiting limit_req zone=api_general burst=200 nodelay;

# Route to appropriate service location /api/v2/users { proxy_pass http://user-service:3000; proxy_set_header X-User-ID $user_id; proxy_set_header X-User-Role $user_role; }

location /api/v2/orders { proxy_pass http://order-service:3001; proxy_set_header X-User-ID $user_id; }

location /api/v2/payments { proxy_pass http://payment-service:3002; proxy_set_header X-User-ID $user_id; } }

location = /auth { internal; proxy_pass http://auth-service:3003/validate; proxy_pass_request_body off; proxy_set_header Content-Length ""; proxy_set_header X-Original-URI $request_uri; } } } ```

Service Mesh: Internal Traffic Management

While an API gateway handles external traffic, a service mesh manages communication between your internal services.

Why You Need a Service Mesh

Without a service mesh, each service must implement its own: - Retry logic - Circuit breaking - mTLS encryption - Distributed tracing headers - Load balancing

That's a lot of cross-cutting logic duplicated across every service. A service mesh moves this logic to infrastructure layer.

How Service Mesh Works

With Service Mesh:

+-------------------+ +-------------------+ | Service A | | Service B | | +-------------+ | | +--------------+ | | | Application | | | | Application | | | +------+------+ | | +-------+------+ | | +------v------+ | mTLS encrypted | +-------v------+ | | | Sidecar |--+------------------->+-| Sidecar | | | | (Envoy) | | | | (Envoy) | | | +-------------+ | | +--------------+ | +-------------------+ +-------------------+ ```

Istio Service Mesh Configuration

Istio is the most feature-complete service mesh. Here's a practical setup:

# Enable Istio sidecar injection for namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    istio-injection: enabled

--- # Traffic management - canary deployment apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: order-service spec: hosts: - order-service http: - match: - headers: x-canary: exact: "true" route: - destination: host: order-service subset: canary port: number: 3000 - route: - destination: host: order-service subset: stable port: number: 3000 weight: 95 - destination: host: order-service subset: canary port: number: 3000 weight: 5

--- # Destination rules for circuit breaking apiVersion: networking.istio.io/v1beta1 kind: DestinationRule metadata: name: order-service spec: host: order-service trafficPolicy: connectionPool: tcp: maxConnections: 100 http: h2UpgradePolicy: DEFAULT http1MaxPendingRequests: 100 http2MaxRequests: 1000 outlierDetection: consecutive5xxErrors: 5 interval: 30s baseEjectionTime: 30s maxEjectionPercent: 50 subsets: - name: stable labels: version: v2.4 - name: canary labels: version: v2.5 ```

Circuit Breaking Configuration

Circuit breakers prevent cascading failures. When a downstream service fails:

Normal State:    Client --> Circuit CLOSED --> Service (healthy)
                     Requests flow normally

Failure Detected: Client --> Circuit OPEN --> Service (failing) Requests fail fast (no timeout wait)

Recovery: Client --> Circuit HALF-OPEN --> Service (testing) Limited requests to test recovery ```

API Gateway vs. Service Mesh: When to Use What

+---------------------+------------------+------------------+
| Feature             | API Gateway      | Service Mesh     |
+---------------------+------------------+------------------+
| Traffic Direction   | North-South      | East-West        |
| Primary Use Case    | External access  | Internal comms   |
| Authentication      | Yes - Primary    | No - Secondary   |
| Rate Limiting       | Yes - Primary    | No - Limited     |
| mTLS                | No - External TLS| Yes - Internal   |
| Circuit Breaking    | Yes - Basic      | Yes - Advanced   |
| Observability       | Yes - Request    | Yes - Full       |
| Canary Deployments  | No - Limited     | Yes - Native     |
| Complexity          | Low-Medium       | High             |
+---------------------+------------------+------------------+

**Use an API Gateway when**: You have external clients, need rate limiting, or want centralized authentication.

**Use a Service Mesh when**: You have 10+ microservices, need mTLS between services, or want advanced traffic management.

**Use both when**: You have external API consumers AND complex internal service communication.

Real-World Example: SaaS Platform

A Mumbai-based SaaS company serving 500+ enterprise clients implemented both:

**API Gateway (Kong)**: - External API management for 200+ API endpoints - Per-client rate limiting (free tier: 100 req/min, enterprise: 10,000 req/min) - OAuth 2.0 authentication - Request/response transformation for API versioning

**Service Mesh (Istio)**: - mTLS encryption for all internal traffic (compliance requirement) - Canary deployments with 5% traffic shifting - Circuit breaking preventing cascade failures - Distributed tracing with Jaeger

**Result**: 99.99% API uptime, 60% reduction in cross-cutting code within services, and zero-trust networking between all internal components.

Monitoring Your Traffic Infrastructure

# Prometheus metrics for API Gateway
api_gateway_requests_total{method="GET", status="200"} 150000
api_gateway_request_duration_seconds{quantile="0.99"} 0.250
api_gateway_rate_limit_exceeded_total 42

# Service mesh metrics istio_requests_total{destination_service="order-service"} 89000 istio_request_duration_milliseconds{quantile="0.99"} 45 istio_tcp_connections_opened_total 1200 ```

Set up alerts for: - Error rate exceeds 1% for 5 minutes - P99 latency exceeds 500ms - Rate limit exceeded count spikes - Circuit breaker opens

Getting Started

1. **Start with an API Gateway**: Deploy Kong or configure Nginx as a gateway. You get immediate benefits with low complexity. 2. **Add observability**: Implement request logging and basic metrics before adding a service mesh. 3. **Evaluate service mesh**: If you have 10+ services and need mTLS or advanced traffic management, consider Istio or Linkerd. 4. **Keep it simple**: Don't add complexity you don't need. A well-configured Nginx gateway handles most use cases.

Traffic management infrastructure is the nervous system of your cloud application. Invest in it proportionally to your architecture's complexity.

Key Takeaways

**An API gateway handles external traffic** (authentication, rate limiting, routing) while a **service mesh manages internal traffic** (mTLS, retries, circuit breaking) — they solve different problems at different layers.
**Start with an API gateway** (Kong, Nginx, or Traefik) — it delivers immediate value with low operational complexity and doesn't require sidecar proxies in every service.
**Add a service mesh only when you have 10+ microservices** and need mutual TLS, fine-grained traffic policies, or canary deployments — premature adoption adds overhead without proportional benefit.
**Monitor request rates, error rates, and latency** at both the gateway and mesh layers — traffic problems visible at the gateway often originate deeper in the service graph.
**Keep complexity proportional to architecture size** — a well-configured Nginx gateway handles most use cases; don't adopt Istio for a three-service deployment.

API Gateway and Service Mesh: Managing Cloud Application Traffic Like a Pro