Nginx 接入层 60w QPS 雪崩复盘:长连接复用 + 代理缓存 + 限流实战

2024 年我们的接入层:Nginx 集群 8 台,承载全站日 80 亿请求,某次活动期间峰值 QPS 冲到 60w,Nginx 开始出现 502/504、worker 进程 CPU 100%、upstream 连接耗尽、长连接复用率低。投了两周做接入层治理,峰值能力提升到 120w QPS,502 归零,P99 从 800ms 降到 60ms。本文复盘 Nginx 高并发网关性能调优的完整实战,覆盖 worker、连接、upstream、缓存、限流、监控。

故障现场

接入层:Nginx 1.24,8 台 16 核 / 32GB
角色:反向代理 + 负载均衡 + 静态资源 + 限流
后端:200+ 上游服务

故障表现(活动峰值):
- 502 Bad Gateway:每分钟数万
- 504 Gateway Timeout:upstream 响应慢
- worker 进程 CPU 100%
- access.log:upstream_connect_time 偶发 3s+
- error.log:
  - "worker_connections are not enough"
  - "upstream timed out (110: Connection timed out)"
  - "no live upstreams while connecting to upstream"
  - "Too many open files"

排查:
1. ulimit -n: 1024(太小!)
2. worker_connections: 1024(默认,太小)
3. nginx -V: 未启用 reuseport
4. upstream 没配 keepalive,每请求新建 TCP
5. ss -s: TIME_WAIT 连接 6w+(短连接 + 没复用)
6. top: worker 进程不均(有的 100% 有的 30%)

修复 1:Worker + 连接核心配置

# nginx.conf 主配置

# worker 进程数 = CPU 核心数
worker_processes auto;

# 每个 worker 绑定 CPU(减少上下文切换)
worker_cpu_affinity auto;

# 每个 worker 最大文件描述符
worker_rlimit_nofile 1000000;

# 优先级
worker_priority -10;

events {
    # 每个 worker 最大连接数
    worker_connections 65535;

    # Linux 高效事件模型
    use epoll;

    # 一次接受多个新连接
    multi_accept on;

    # accept 锁(高并发时关闭,用 reuseport 替代)
    accept_mutex off;
}

http {
    # === 连接优化 ===
    # 客户端长连接
    keepalive_timeout 65;
    keepalive_requests 10000;       # 单连接最多 1w 请求

    # TCP 优化
    sendfile on;                     # 零拷贝
    tcp_nopush on;                   # 攒满一个包再发
    tcp_nodelay on;                  # 长连接禁用 Nagle

    # === 超时 ===
    client_header_timeout 10s;
    client_body_timeout 10s;
    send_timeout 10s;
    reset_timedout_connection on;

    # === buffer ===
    client_header_buffer_size 4k;
    large_client_header_buffers 4 16k;
    client_body_buffer_size 128k;
    client_max_body_size 50m;

    # === 文件缓存 ===
    open_file_cache max=200000 inactive=60s;
    open_file_cache_valid 60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    # === 日志(buffer + 异步)===
    access_log /var/log/nginx/access.log main buffer=64k flush=5s;

    # === 压缩 ===
    gzip on;
    gzip_comp_level 4;
    gzip_min_length 1024;
    gzip_types text/plain text/css application/json application/javascript;
    gzip_vary on;
}

# 系统层(必改)
# /etc/security/limits.conf
# * soft nofile 1000000
# * hard nofile 1000000

# /etc/sysctl.conf
# net.core.somaxconn = 65535
# net.ipv4.tcp_max_tw_buckets = 2000000
# net.ipv4.tcp_tw_reuse = 1
# net.ipv4.tcp_fin_timeout = 15
# net.core.netdev_max_backlog = 262144
# net.ipv4.tcp_max_syn_backlog = 65535

修复 2:Upstream Keepalive(关键)

# 问题:默认每个请求到 upstream 都新建 TCP + 四次挥手
# 导致 TIME_WAIT 堆积 + 连接耗时

upstream order_backend {
    # 负载均衡算法
    least_conn;                      # 最少连接(比 round_robin 均衡)

    # 后端服务器
    server 10.0.1.1:8080 weight=1 max_fails=3 fail_timeout=10s;
    server 10.0.1.2:8080 weight=1 max_fails=3 fail_timeout=10s;
    server 10.0.1.3:8080 weight=1 max_fails=3 fail_timeout=10s;
    server 10.0.1.4:8080 backup;     # 备用

    # === 长连接池(核心)===
    keepalive 300;                   # 每 worker 保持 300 空闲长连接
    keepalive_requests 10000;        # 单连接最多 1w 请求
    keepalive_timeout 60s;           # 空闲 60s 关闭
}

server {
    listen 80 reuseport;             # SO_REUSEPORT,内核级负载均衡

    location /api/order/ {
        proxy_pass http://order_backend;

        # === 启用 HTTP/1.1 长连接(必须)===
        proxy_http_version 1.1;
        proxy_set_header Connection "";   # 清空 Connection 头

        # === 超时 ===
        proxy_connect_timeout 2s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;

        # === buffer ===
        proxy_buffering on;
        proxy_buffer_size 8k;
        proxy_buffers 8 16k;
        proxy_busy_buffers_size 32k;

        # === 失败重试 ===
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 5s;

        # === 透传真实 IP ===
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# 效果:
# - TIME_WAIT 从 6w 降到 < 5000
# - upstream_connect_time 从偶发 3s 降到 < 1ms(连接复用)
# - 同样机器扛 2x QPS

修复 3:限流 + 防刷

http {
    # === 限流区域定义 ===
    # 1. 按 IP 限请求速率
    limit_req_zone $binary_remote_addr zone=perip:20m rate=100r/s;

    # 2. 按 IP 限连接数
    limit_conn_zone $binary_remote_addr zone=conn_perip:20m;

    # 3. 按接口限速(用 server_name + uri)
    limit_req_zone $server_name zone=perserver:20m rate=10000r/s;

    # 限流响应码
    limit_req_status 429;
    limit_conn_status 429;

    server {
        location /api/ {
            # 限速率:burst 缓冲 200,nodelay 不延迟处理
            limit_req zone=perip burst=200 nodelay;
            limit_req zone=perserver burst=2000 nodelay;

            # 限连接:单 IP 最多 50 并发连接
            limit_conn conn_perip 50;

            proxy_pass http://order_backend;
        }

        # 秒杀接口:更严格
        location /api/seckill/ {
            limit_req zone=perip burst=5 nodelay;
            limit_conn conn_perip 5;
            proxy_pass http://seckill_backend;
        }

        # === 防刷:UA / Referer 黑名单 ===
        if ($http_user_agent ~* (curl|wget|python|scrapy|bot)) {
            return 403;
        }

        # === 防刷:基于 Lua 的滑动窗口(OpenResty)===
        location /api/login/ {
            access_by_lua_block {
                local limit_count = require "resty.limit.count"
                local lim, err = limit_count.new("login_limit", 5, 60)
                local key = ngx.var.binary_remote_addr
                local delay, err = lim:incoming(key, true)
                if not delay then
                    if err == "rejected" then
                        return ngx.exit(429)
                    end
                end
            }
            proxy_pass http://auth_backend;
        }
    }
}

# === geo 模块:IP 白名单/黑名单 ===
geo $blocked_ip {
    default 0;
    1.2.3.0/24 1;        # 恶意网段
    5.6.7.8 1;
}
server {
    if ($blocked_ip) { return 403; }
}

修复 4:静态资源 + 代理缓存

http {
    # === 代理缓存区域 ===
    proxy_cache_path /data/nginx/cache
        levels=1:2
        keys_zone=api_cache:100m
        max_size=10g
        inactive=60m
        use_temp_path=off;

    server {
        # 静态资源
        location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff2)$ {
            root /data/static;
            expires 30d;
            add_header Cache-Control "public, immutable";
            access_log off;                  # 静态资源不记日志
        }

        # API 响应缓存(只读接口)
        location /api/product/detail {
            proxy_cache api_cache;
            proxy_cache_key "$scheme$request_method$host$request_uri";
            proxy_cache_valid 200 5m;        # 200 缓存 5min
            proxy_cache_valid 404 1m;
            proxy_cache_use_stale error timeout updating http_500 http_502 http_503;
            proxy_cache_background_update on; # 后台更新,不阻塞
            proxy_cache_lock on;             # 防缓存击穿(同 key 只 1 个回源)

            # 缓存状态响应头(调试用)
            add_header X-Cache-Status $upstream_cache_status;

            proxy_pass http://product_backend;
        }

        # 绕过缓存条件
        location /api/user/ {
            # 带 token 的请求不缓存
            proxy_cache_bypass $http_authorization;
            proxy_no_cache $http_authorization;
            proxy_pass http://user_backend;
        }
    }
}

# 清理缓存(需要 ngx_cache_purge 模块)
# location ~ /purge(/.*) {
#     allow 127.0.0.1;
#     deny all;
#     proxy_cache_purge api_cache "$scheme$request_method$host$1";
# }

# 效果:
# - product/detail 接口缓存命中率 85%
# - 后端 QPS 降 80%
# - X-Cache-Status: HIT / MISS / EXPIRED / STALE / UPDATING

修复 5:健康检查 + 优雅发布

# 1. 被动健康检查(开源版自带)
upstream backend {
    server 10.0.1.1:8080 max_fails=3 fail_timeout=10s;
    # 连续失败 3 次,标记 down 10s
}

# 2. 主动健康检查(需要 nginx_upstream_check_module 或 Nginx Plus)
upstream backend {
    server 10.0.1.1:8080;
    check interval=3000 rise=2 fall=3 timeout=2000 type=http;
    check_http_send "GET /health HTTP/1.0\r\n\r\n";
    check_http_expect_alive http_2xx http_3xx;
}

# 3. 优雅发布(配合脚本)
# 发布前:从 upstream 摘除节点
$ curl -X POST http://nginx-admin/upstream/backend/server/10.0.1.1:8080/down

# 等待存量请求处理完(连接 drain)
$ sleep 30

# 部署新版本...

# 健康检查通过后加回
$ curl -X POST http://nginx-admin/upstream/backend/server/10.0.1.1:8080/up

# 4. reload 配置(不中断服务)
$ nginx -t && nginx -s reload
# 老 worker 处理完存量请求后退出,新 worker 接新请求

# 5. 平滑升级 Nginx 二进制
$ kill -USR2 $(cat /var/run/nginx.pid)    # 启动新 master
$ kill -WINCH $(cat /var/run/nginx.pid.oldbin)  # 老 worker 优雅退出
$ kill -QUIT $(cat /var/run/nginx.pid.oldbin)   # 老 master 退出

修复 6:监控告警

# 1. nginx-prometheus-exporter(基于 stub_status)
# nginx.conf
server {
    listen 8080;
    location /stub_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

# docker run nginx-prometheus-exporter -nginx.scrape-uri=http://localhost:8080/stub_status

# 2. 更详细:用 OpenResty + lua-prometheus 自定义指标
# 暴露:请求数、状态码分布、upstream 耗时、缓存命中率

# 3. 告警规则
- alert: Nginx5xxHigh
  expr: |
    sum(rate(nginx_http_requests_total{status=~"5.."}[5m]))
    / sum(rate(nginx_http_requests_total[5m])) > 0.01
  for: 3m
  labels: { severity: critical }
  annotations:
    summary: "Nginx 5xx 错误率 > 1%"

- alert: NginxUpstreamSlow
  expr: |
    histogram_quantile(0.99, rate(nginx_upstream_response_time_bucket[5m])) > 1
  for: 5m
  annotations:
    summary: "{{ $labels.upstream }} P99 > 1s"

- alert: NginxConnectionsHigh
  expr: nginx_connections_active > 50000
  for: 5m
  annotations:
    summary: "{{ $labels.instance }} 活跃连接 > 5w"

- alert: NginxWorkerCpuHigh
  expr: rate(process_cpu_seconds_total{job="nginx"}[5m]) > 0.9
  for: 5m
  annotations:
    summary: "{{ $labels.instance }} worker CPU > 90%"

# 4. access.log 分析(实时)
# 用 GoAccess 实时分析
$ goaccess /var/log/nginx/access.log -o /var/www/report.html --real-time-html

# 5. 日志格式(含 upstream 耗时)
log_format main '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent" '
                'rt=$request_time uct=$upstream_connect_time '
                'uht=$upstream_header_time urt=$upstream_response_time '
                'cache=$upstream_cache_status';

优化效果

指标                  优化前         优化后
=========================================================
峰值 QPS 承载         60w(开始 502) 120w(稳定)
502 错误             数万/min       0
504 错误             数千/min       < 10/min
P50 延迟              120ms          25ms
P99 延迟              800ms          60ms
TIME_WAIT 连接        6w+            < 5000
upstream_connect_time 偶发 3s        < 1ms
缓存命中率            0              85%(product 接口)
后端 QPS              60w            12w(缓存挡掉 80%)
worker CPU            100%(不均)    55%(均衡)

成本:
- Nginx 集群规模不变(8 台)
- 承载能力 2x,后端服务缩容 40%
- 整体接入层 + 后端成本下降 35%

业务影响:
- 活动期间零 502,用户体感顺畅
- 后端压力骤降,数据库不再被打爆
- 接入层成为稳定的"减压阀"

避坑清单

1. ulimit -n 和 worker_rlimit_nofile 必须设到百万级
2. worker_connections 65535,worker_processes auto
3. upstream 必须配 keepalive + proxy_http_version 1.1 + Connection ""
4. listen 加 reuseport,内核级负载均衡,worker 更均匀
5. proxy_next_upstream 配好重试,但 tries 限制 2 次防雪崩
6. limit_req + limit_conn 双限流,burst + nodelay 平滑突发
7. 只读接口上 proxy_cache,proxy_cache_lock 防击穿
8. 静态资源 expires 30d + access_log off
9. max_fails / fail_timeout 被动健康检查,有条件上主动检查
10. 日志带 upstream 耗时字段,stub_status + Prometheus 监控

总结

Nginx 高并发治理的核心是连接复用 + 缓存 + 限流。最大的认知改变:upstream keepalive 是被严重低估的配置,不配的话每个请求到后端都要 TCP 三次握手 + 四次挥手,TIME_WAIT 堆积、连接耗时波动,配上长连接池后同样的机器能扛 2 倍 QPS。最被低估的是 reuseport,加一个关键字,内核负责把新连接均匀分给各 worker,彻底解决 worker CPU 不均的老问题。最容易踩的坑是 proxy_http_version 忘了设 1.1 或 Connection 头没清空,这时 keepalive 配了也不生效,长连接池形同虚设 —— 三个配置(keepalive / http_version 1.1 / Connection "")必须配套出现。最后,代理缓存是接入层的"大杀器",只读接口缓存命中 85% 意味着后端只需要扛 15% 的流量,proxy_cache_lock 防击穿、proxy_cache_use_stale 在后端故障时还能返回旧数据,这是接入层为整个系统兜底的最后一道防线。

—— 别看了 · 2026