And i’m tuning a small web API on AWS and trying to keep p95 latency under 200 ms while EC2 spend stays below $150/month. Right now the ASG scales at 65% CPU, but I see queueing around 400 RPS; has anyone seen better stability using target tracking on request count per target or by tightening keep-alive/connection reuse settings on the ALB to smooth spikes?
Go target tracking on ALB RequestCountPerTarget, not CPU — set about 50 RPS per target with 60–90s warm-up; it stopped our 400 RPS queueing and kept p95 about 180 ms on t4g.mediums under ~$120/mo. > anyone seen better stability using target tracking on request count per target or by tightening keep-alive/connection — keep-alives on, but drop ALB idle_timeout to about 15–20s to dump idle conns during bursts. What’s your instance type and idle_timeout right now?
I’d enable ALB slow start (30–60s) to absorb “400 RPS” spikes; ASG warm pool helps. What’s your deregistration delay?
Quick win: put CloudFront in front of the ALB with a tiny TTL (30–60s) and Brotli; if your GETs are cacheable, it soaked our bursts and dropped p95 to about 170 ms while cost barely moved — a cheap turbo. Also ensure ALB HTTP/2 is on so a few connections can multiplex; docs: Manage how long content stays in the cache (expiration) - Amazon CloudFront. @OP which routes are safe to cache right now?
Double-check that your app server’s keep-alive/idle timeouts are set slightly below the ALB idle timeout (default 60s) so the app, not the LB, closes first — cut our connection churn and shaved p95 without touching scaling. If you’re on Node, set keepAliveTimeout about 55s and headersTimeout a bit higher (ref: Application Load Balancers - Elastic Load Balancing); what runtime are you on?
We shaved p95 by turning on TCP_NODELAY in the app (disabling Nagle) so small responses don’t hit the classic ‘200 ms delayed ACK’ blip between ALB and the instance; quick to try and cheap — can you toggle it in your framework and watch a spike run?
Enable ALB target group ‘slow start’ to tame 400 RPS surges; cut p95 measurably: Target groups for your Application Load Balancers - Elastic Load Balancing.
Switch to target tracking on ‘ALBRequestCountPerTarget’ with a conservative target (around 50–80 per instance) and add a warm pool of 1 so scale-outs don’t start cold — like keeping one burner preheated; this kept our p95 <200 ms without busting $150 (docs: Amazon EC2 Auto Scaling). Are you on T-class by chance? Credit wobble can mimic queueing; c7g/c6g smalls were steadier for us.