Learning objectives
What you will be able to explain
- Match each Ray primitive to its definition
- Why does Ray's shared-memory object store matter for RL throughput?
- When do you need a Ray actor rather than a task?
- What is RLlib?
- Match each RLlib abstraction to its role
- In deep RL, why is environment sampling usually the throughput bottleneck?
Section 01
Match each Ray primitive to its definition to When do you need a Ray actor rather than a task?
01
Match each Ray primitive to its definition
Ray (Moritz et al., OSDI 2018) exposes tasks (stateless) and actors (stateful) on top of a shared-memory object store and a global control store. RL maps naturally: env simulators and policies become actors, and the object store moves large sample batches with little copying.
Guided checkpoint
Ray's core model underpins RLlib.
02
Why does Ray's shared-memory object store matter for RL throughput?
RL shuffles big arrays (observations, trajectories) between many workers. Plasma's immutable, shared-memory objects let workers on the same node read without copying, cutting the serialisation overhead that would otherwise dominate at high sample rates.
03
When do you need a Ray actor rather than a task?
Tasks are stateless and ideal for embarrassingly parallel one-shot work. An environment that must persist between steps, or a learner holding weights and optimiser state, is inherently stateful, so it is an actor.
Section 02
What is RLlib? to In deep RL, why is environment sampling usually the throughput bottleneck?
01
What is RLlib?
RLlib (Liang et al., 2018) provides production-grade distributed RL on Ray, with PPO, IMPALA/APPO, DQN/Ape-X, SAC and more, plus multi-agent support. It is the concrete tool behind the ad's explicit mention of Ray.
02
Match each RLlib abstraction to its role
An Algorithm coordinates many rollout workers (each stepping envs) producing SampleBatches that feed a learner. Being able to name these maps your data-pipeline thinking from dmTECH onto RL infrastructure.
Guided checkpoint
These are the moving parts of an RLlib training run.
03
In deep RL, why is environment sampling usually the throughput bottleneck?
A single env may produce only thousands of steps per second while a GPU can consume far more, so the standard scaling move is to parallelise environments across many workers. This is the heart of every distributed-RL architecture below.
Section 03
What is the actor-learner architecture? to Match each distributed-RL architecture to its defining idea
01
What is the actor-learner architecture?
Separating acting from learning lets you scale the bottleneck (sampling) independently by adding actors, while the learner batches updates on a GPU. IMPALA, Ape-X and SEED RL are all variations on this split.
02
Synchronous vs asynchronous distributed RL: true or false
Synchronous is simple, reproducible and on-policy, but straggler-bound. Asynchronous gives high throughput, but staleness must be handled (V-trace in IMPALA, replay in Ape-X). Async does not remove the need for correction; it is precisely what creates it.
Guided checkpoint
Mark each statement.
03
Match each distributed-RL architecture to its defining idea
A3C (Mnih, 2016) shares gradients; Ape-X (Horgan, 2018) shares prioritised experience; IMPALA (Espeholt, 2018) shares trajectories plus V-trace; SEED RL (Espeholt, 2020) centralises inference to keep accelerators busy; R2D2 (Kapturowski, 2019) adds recurrence to distributed replay.
Guided checkpoint
Distinguish the canonical distributed actor-learner architectures.
Section 04
In IMPALA, what do the actors send to the learner, and what problem does that create? to SEED RL moves policy inference to the central learner. What IMPALA inefficiency does this fix?
01
In IMPALA, what do the actors send to the learner, and what problem does that create?
Sending trajectories rather than gradients lets a single learner batch large updates on a GPU. The cost is policy lag, the actors used an older policy, so IMPALA applies V-trace truncated importance sampling to keep the value targets and policy gradient sound.
02
What makes Ape-X scale so well?
Ape-X (distributed prioritised experience replay) is off-policy, so stale actor data is fine, replay already breaks the on-policy assumption. Diverse per-actor exploration plus prioritisation means more actors simply yields richer, higher-priority data for the learner. It powered distributed DQN and DDPG.
03
SEED RL moves policy inference to the central learner. What IMPALA inefficiency does this fix?
IMPALA actors do their own CPU inference and need weight updates shipped to them, wasting accelerator capacity. SEED RL has actors send observations over gRPC to a central, batched inference-and-learning loop on the accelerator, sharply improving hardware utilisation and frames/sec at scale.
Section 05
What is 'policy lag' (staleness) in distributed RL? to Once sampling is no longer the bottleneck, how is the learner itself scaled?
01
What is 'policy lag' (staleness) in distributed RL?
Asynchronous actors are always a few updates behind the learner, so their experience is mildly off-policy. Left uncorrected this biases updates; V-trace (IMPALA/APPO) and replay-based off-policy methods (Ape-X) are the two standard ways to cope, the same lag AlphaStar's V-trace handles.
02
Why did synchronous A2C often replace asynchronous A3C in practice?
A3C's asynchronous gradient application creates stale-gradient noise and underuses GPUs. A2C waits for all workers and then does one large batched update, simpler, more reproducible and often better on a GPU. A neat example that 'async' is not automatically better.
03
Once sampling is no longer the bottleneck, how is the learner itself scaled?
When actors saturate a single learner, you go data-parallel (multiple GPUs, gradient all-reduce). Then learner throughput and interconnect bandwidth dominate, the same distributed-training concerns as large supervised models, which is where CI/CD and cloud experience pays off.
Section 06
Throughput and bottlenecks: true or false to Ray Tune supports Population Based Training (PBT). Why is that interesting for an RL / AlphaStar-style team?
01
Throughput and bottlenecks: true or false
Frames per second is the standard scaling metric. Balance matters: too many actors for one learner just increases staleness and queueing. Batched inference (the SEED RL idea) helps utilisation, but eventually the learner or interconnect caps the speed-up, no free lunch.
Guided checkpoint
Mark each statement.
02
How is a Ray cluster typically run on Kubernetes, as the ad implies?
KubeRay is the standard operator: you declare a RayCluster (and RayJob/RayService) and Kubernetes schedules the head and autoscaling worker pods. This provides a reproducible way to run Ray workloads on containerized, autoscaling infrastructure.
03
Ray Tune supports Population Based Training (PBT). Why is that interesting for an RL / AlphaStar-style team?
RL is notoriously hyperparameter-sensitive. PBT (Jaderberg et al.) runs many trials, exploits good ones and explores perturbations on the fly. The population framing rhymes with league training and is a natural fit on Ray Tune.
Section 07
How does Ray help with the long, failure-prone runs typical of large RL training? to Why might a distributed RL job use Ray 'placement groups' (gang scheduling)?
01
How does Ray help with the long, failure-prone runs typical of large RL training?
At AlphaStar-like scale (days, many nodes) failures are routine. Ray can reconstruct lost objects from lineage and restart actors, and you checkpoint learner/optimiser state so a crash costs minutes, not days, core MLOps hygiene.
02
Why distribute RL training for expensive multi-agent simulations? Select all that apply.
Expensive sims, breadth of scenarios, and population/hyperparameter search all push toward distributed training on Ray/Kubernetes. What distribution never buys you is skipping validation, the sim-to-real and trust questions remain.
03
Why might a distributed RL job use Ray 'placement groups' (gang scheduling)?
When a job needs several components running simultaneously, scheduling them piecemeal can deadlock (some get resources, others wait forever). Placement groups gang-schedule the bundle. A more advanced operational detail that signals real cluster experience.
Section 08
How does a fast C++ simulation fit into a Ray / Python RL stack?
01
How does a fast C++ simulation fit into a Ray / Python RL stack?
You keep the performance-critical simulator in C++ (matching the ad's C++ requirement and Qt/OpenCV stack), expose a Python env API via bindings such as pybind11, and let Ray fan out many instances across the cluster. That C++/Python boundary is exactly where this role lives.
Knowledge check
Turn understanding into recall.
The quiz now follows the same concepts in scored form. You can return to this lesson from the quiz whenever a gap appears.