Test your Understanding of Ruby Concurrency

Let’s break this down:

Let’s start with the number of Puma master processes. On a single box, two masters × 4 workers is strictly more complexity for zero throughput, and slightly worse on memory. The reasoning is worth spelling out because it hinges on a detail of Puma’s architecture people routinely get wrong.

The master is not in the request path. People carry an nginx-shaped mental model where the master process accepts connections and dispatches to workers — a proxy that could become a bottleneck worth duplicating. Puma doesn’t work that way. In cluster mode, the master opens the listening socket once, forks the workers, and each worker inherits the file descriptor and calls accept() on it directly. The kernel distributes incoming connections across whichever workers are blocked in accept. The master’s remaining job is babysitting: forking, reaping, signaling, restarts. Duplicating a babysitter doesn’t add throughput; there was no contention to relieve.

So one master with W=8 already gives you 8 independent processes pulling from one shared accept queue with kernel-level distribution. Two masters gives you 8 independent processes pulling from two accept queues — plus a new problem you didn’t have before: something now has to split traffic between the two ports. An nginx upstream, a load balancer, iptables tricks. Naive round-robin between the two ports is worse than the single shared socket, because the kernel’s shared-queue behavior is naturally work-conserving (an idle worker grabs the next connection regardless of which “half” it’s in), whereas round-robin across two sockets can queue a request behind a busy group while the other group sits idle. You’d be re-implementing, badly, in userspace, what the kernel already does well.

Now let’s look at the number of workers. W is the number of workers per master process.

Since MRI Ruby can only max out a single CPU core, one process limits us to a single core. Therefore we typically want to run at least the same number of MRI Ruby processes as we have CPU cores available to us.

However any realistic Rails application spends a portion of it’s time waiting on IO, which means waiting for DB queries to return, waiting on Redis network calls, even logging to a file, counts as IO, as well as caching (via filesystem or memcached) we are not able to push each process to the max, and that typically means that by increasing the number of threads we can utilize that IO wait time doing something on CPU, like rendering a view or processing incoming JSON requests.

This brings us to the last value: T — the number of threads per worker.

To properly determine how many threads per process we should run, we need some sort of monitoring like DataDog or NewRelic (it can also be done without the monitoring giants, but it’s much more involved).

What we want to know is how much time each process (running with 1 thread) spends in CPU versus IO, and what latency that achieves. A typical ratio is 1:1 IO to CPU or even 2:1 IO to CPU. That gives us the ratio to figure out the T in question. If the ration is 3:1 IO to CPU, then we can run 3 threads per process to leverage that remaining CPU time. However, once we change this, we need to monitor the new ratio, especially the resulting latency. If the latency goes up significantly, we need to reduce the number of threads from 3 to 2 and retest.

In my personal experience, running 3-4 threads per Puma process is typically a good balance between CPU, IO utilization, and the resulting latency, but once again, that really depends on how well your database is tuned and how quickly your queries return, because if they’re not returning fast enough, it means your time spent in the database calls is going to be much higher, resulting in likely high latency (anything more than 150ms is already considered high latency). And therefore your I/O weight is going to be high compared to CPU, and therefore you can afford to have much higher number of threads to compensate for that. Your requests won’t get any faster, but at least you’ll be able to process more of them concurrently on the same box.

Thanks for playing! And hope this exercise helps you understand the trade-offs of concurrency and how to properly configure Puma for your application.

Let’s break this down:

Because the task at hand is CPU-bound, MRI Ruby (with its GIL) will not be able to take advantage of multiple cores with just one Ruby process. In fact, if we did introduce a thread-pool, Ruby will have to do more work, since it will need to spend some CPU cycles on context switching: voluntary and involuntary.

JRuby, on the other hand, is able to fully utilize a multi-core environment with Java’s green threads.