Node.js handles concurrency beautifully — until a single slow query or an unbounded loop blocks the event loop and everything grinds to a halt. Scaling an API is mostly about removing those bottlenecks one measured step at a time.
Why measure before you optimize?
Add structured logging and tracing first. Most latency hides in one or two endpoints making N+1 database calls; you can't fix what you can't see. Load-test with realistic traffic to find the real ceiling.
What should you cache first?
- Put Redis in front of hot, read-heavy queries with a sensible TTL and explicit invalidation.
- Cache at the edge/CDN for anonymous, cacheable responses.
- Pool database connections — opening a connection per request will sink you under load.
async function getProduct(id: string) {
const cached = await redis.get(`product:${id}`);
if (cached) return JSON.parse(cached);
const product = await db.product.findUnique({ where: { id } });
await redis.set(`product:${id}`, JSON.stringify(product), 'EX', 300);
return product;
}How do queues take slow work off the request path?
Sending email, generating PDFs or calling third-party APIs inside a request makes users wait and ties up the event loop. Push that work to a queue (BullMQ) and respond immediately; process it in a separate worker.
A fast API isn't one that does everything quickly — it's one that does the slow things somewhere else.
Should your API responses be cached at the edge?
For public, cacheable GETs, a CDN layer in front of Node removes load before it ever reaches you: set explicit Cache-Control headers with s-maxage and stale-while-revalidate, version cacheable payloads in URLs rather than relying on headers alone, and purge on write via your platform's API. Compression matters too — enable brotli for JSON responses; large list endpoints shrink 5–10x. What must never sit at the edge: authenticated responses without strict vary rules and anything user-specific unless you key the cache by session. Rate limiting at the edge (per IP and per API key) completes the picture — it protects your origin from both abuse and buggy client retry loops. Measure cache hit-rate per endpoint weekly; a hit-rate that silently drops from 90% to 40% is usually a header regression, and it shows up in your Redis bill before your latency dashboards.
What do you monitor once the API is fast?
Speed is a property you maintain, not one you achieve once. Four signals catch almost every production regression:
- p95/p99 latency per endpoint, not averages — averages hide exactly the tail your loudest customers live in.
- Event-loop lag: if it climbs past tens of milliseconds, some handler is blocking and no amount of caching will save you.
- Pool saturation and Redis hit-rate — the two numbers that explain most sudden slowdowns.
- Queue depth and oldest-job age for every BullMQ queue; workers silently dying is the classic 3 a.m. page.
How does stateless design make scaling boring?
Horizontal scaling only works when any instance can serve any request. Keep sessions in Redis (or signed cookies), never in module-level memory; treat local disk as disposable; and make health checks verify dependencies, not just liveness. Then adding capacity is literally adding containers behind the load balancer — and rolling deploys stop being terrifying because no instance owns state that dies with it.
- Graceful shutdown: on SIGTERM, stop accepting new connections, finish in-flight requests, close pool/Redis — or deploys drop live requests.
- Idempotent handlers: retries happen, especially under load; make POST /charge safe to replay with an idempotency key.
- Backpressure: cap queue depth per worker so overload degrades predictably instead of cascading.
What kills a Node process under load?
- Synchronous CPU work — JSON-parsing megabyte payloads or hashing passwords on the event loop blocks every other request.
- Unbounded concurrency: firing 5,000 database calls in parallel exhausts the pool and stacks latency behind connection waits.
- Missing timeouts: one hung downstream API holds sockets open until the process runs out of file descriptors.
- Memory leaks from unbounded in-process caches — the OOM kill at 3 a.m. is always an LRU away from being fixed.
Each has a standard countermeasure: move CPU-heavy work to worker threads, bound concurrency with a semaphore or p-limit, set explicit request timeouts and circuit breakers, and cap any in-memory cache. Load-test after each change so you know what the fix actually bought you.
Load-testing deserves more rigor than it usually gets. Record real traffic shapes from logs — endpoint mix, payload sizes, think time between calls — and replay them with k6 or autocannon rather than hammering one URL with flat concurrency. Include a spike profile (0 to peak in seconds), because production traffic arrives in bursts, not ramps; an API that survives steady 10k but folds at a sudden 3k is the one that pages you at launch.
References
Caching, pooling and queues are unglamorous, but together they take an API from buckling at a hundred concurrent users to comfortably serving a hundred thousand.