Quizmaster

Learning objectives

What you will be able to explain

What formally defines a Markov Decision Process (MDP)?
What does the Markov property assume?
What is the role of the discount factor $\gamma$ ?
Compute the discounted return
Match each RL quantity to its definition
What does the Bellman equation express?

Section 01

What formally defines a Markov Decision Process (MDP)? to What is the role of the discount factor $\gamma$ ?

What formally defines a Markov Decision Process (MDP)?

An MDP is the tuple

(S, A, P, R, \gamma)

. The agent observes a state, picks an action, the environment transitions according to

P

and emits a reward

R

. The goal is a policy that maximises expected discounted return. Reference: Sutton & Barto, Reinforcement Learning: An Introduction, ch. 3.

Guided checkpoint

RL problems are commonly framed as MDPs. Start by identifying the formal components and objective.

What does the Markov property assume?

The Markov property says the current state is a sufficient statistic for the future:

P(s_{t+1} \mid s_t, a_t)

does not improve if you also condition on

s_{t-1}, s_{t-2}, \dots

. When the true state is hidden by occlusion or noisy sensors, the agent faces a POMDP, which motivates recurrent policies.

What is the role of the discount factor $\gamma$ ?

The return is

G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k}

. A smaller

\gamma

makes the agent myopic (favours near-term reward); a larger

\gamma

values long-term consequences. Because

\gamma < 1

, the geometric series converges, so the infinite sum stays finite.

Guided checkpoint

Assume $0 \le \gamma < 1$ .

Section 02

Compute the discounted return to What does the Bellman equation express?

Compute the discounted return

Case A is a geometric series:

\sum_{t=0}^{\infty} \gamma^t = \frac{1}{1-\gamma} = \frac{1}{0.1} = 10

. Case B is finite:

2 + 0.5 \cdot 0 + 0.5^2 \cdot 8 = 2 + 0 + 2 = 4

. The calculation directly checks the discounted-return definition.

Guided checkpoint

Use $G = \sum_{t=0}^{\infty} \gamma^t r_t$ , discounting reward at time $t$ by $\gamma^t$ .

Case A: the agent receives reward $+1$ at every timestep forever, with $\gamma = 0.9$ .
Case B: rewards are $r_0 = 2$ , $r_1 = 0$ , $r_2 = 8$ , then the episode ends, with $\gamma = 0.5$ .

Match each RL quantity to its definition

V

scores a state,

Q

scores a state-action pair,

\pi

is the behaviour itself, and the advantage

A(s,a) = Q(s,a) - V(s)

measures relative gain. Advantages are central to policy-gradient methods like PPO because they reduce gradient variance (see the advantage question below).

Guided checkpoint

Pick the term each description defines.

What does the Bellman equation express?

V^\pi(s) = \mathbb{E}[\,r + \gamma V^\pi(s')\,]

. This recursion is the backbone of value-based RL: dynamic programming, temporal-difference learning, and Q-learning all bootstrap value estimates from successor states using a Bellman update.

Section 03

Sort algorithms: on-policy vs off-policy to Match each algorithm to its method family

Sort algorithms: on-policy vs off-policy

Off-policy methods (Q-learning, DQN, DDPG, SAC) reuse old experience from a replay buffer, so they tend to be more sample-efficient. On-policy methods (SARSA, REINFORCE, A2C/A3C, PPO) must collect fresh data with the current policy after each update, which is simpler and often more stable but less sample-efficient.

Guided checkpoint

On-policy methods learn about the same policy that generates the data. Off-policy methods can learn a target policy from data produced by a different behaviour policy (which is what makes replay buffers possible).

What distinguishes off-policy from on-policy RL?

The decoupling of behaviour and target policy is the defining feature. It lets off-policy algorithms reuse stored transitions (sample efficiency) but introduces a distribution mismatch that must be corrected (e.g. importance sampling, or the stability tricks in DQN/SAC). It does not make them universally faster or more stable.

Match each algorithm to its method family

Value-based methods learn

Q

and act greedily (DQN). Pure policy-gradient methods optimise the policy directly with Monte-Carlo returns (REINFORCE). Actor-critic methods combine both: an actor (policy) and a critic (value baseline), e.g. A3C, and PPO too. Model-based methods learn or use a dynamics model to plan (MuZero, Dyna-Q). PPO sits in the actor-critic family.

Guided checkpoint

Choose the family that best describes each representative algorithm.

Section 04

Model-free vs model-based, and exploration to Which statements about PPO are correct?

Model-free vs model-based, and exploration

Model-based methods (sample-efficient, but a wrong model hurts) plan with learned dynamics; model-free methods (DQN, PPO) skip the model.

\epsilon

-greedy explores by occasionally acting randomly. An entropy bonus rewards a less-peaked action distribution, so it encourages exploration, not discourages it. PPO and SAC both use entropy terms.

Guided checkpoint

Mark each statement true or false.

What is the exploration-exploitation trade-off?

Explore too little and you lock onto a suboptimal habit; explore too much and you waste reward. Strategies:

\epsilon

-greedy and optimistic initialisation (value-based), entropy bonuses and stochastic policies (policy-gradient), and curiosity / intrinsic rewards for sparse-reward tasks.

Which statements about PPO are correct?

PPO (Schulman et al., 2017) is on-policy actor-critic. Its clipped objective bounds the ratio

r_t(\theta) = \frac{\pi_\theta(a \mid s)}{\pi_{\theta_{old}}(a \mid s)}

[1-\epsilon, 1+\epsilon]

, and it commonly uses GAE for low-variance advantages. It is not off-policy and does not use a DQN-style replay buffer (it reuses a batch for only a few epochs before discarding it).

Guided checkpoint

PPO is the workhorse the team is most likely to ask about. Select all that apply.

Section 05

Why does PPO clip the probability ratio? to How was AlphaStar's policy initialised before reinforcement learning?

Why does PPO clip the probability ratio?

If a single update moves the policy too far, the new policy can be much worse and the on-policy data becomes useless. TRPO enforced a hard KL trust region (expensive, second-order); PPO approximates the same effect cheaply by clipping the ratio so the objective stops rewarding moves beyond

[1-\epsilon, 1+\epsilon]

. That stability-with-simplicity trade-off is why PPO is so widely used.

Why use an advantage function or baseline in policy-gradient methods?

Raw returns make the policy-gradient estimator very high-variance. Subtracting a state-dependent baseline (typically

V(s)

, giving the advantage

A = Q - V

) keeps the estimator unbiased while cutting variance, so learning is faster and more stable. GAE trades off bias and variance with a parameter

\lambda

How was AlphaStar's policy initialised before reinforcement learning?

AlphaStar (Vinyals et al., 2019, Nature) was first trained by supervised learning on human replays, then improved with multi-agent RL. The imitation bootstrap solves the cold-start problem: in an enormous action space, random exploration almost never stumbles onto competent play. This 'pretrain on demonstrations, then refine with RL' pattern is exactly the kind of research-to-system transfer the role is about.

Guided checkpoint

AlphaStar is a useful case study in large-scale multi-agent learning, league training, and structured policies.

Section 06

Which statements about AlphaStar's League training are correct? to Partial observability and decision-making under uncertainty

Which statements about AlphaStar's League training are correct?

The league mixes main agents (generalists aiming to be robust), main exploiters (hunt weaknesses in the main agents), and league exploiters (hunt weaknesses across the whole population). Prioritised fictitious self-play (PFSP) weights opponents by difficulty. Specialised exploiters are the whole point, so the last statement is false.

Guided checkpoint

Select all that apply.

Why use a diverse league instead of naive self-play against the latest agent?

If A beats B, B beats C, and C beats A, training only against your newest self can chase the cycle forever and forget counters to old tactics. Keeping a frozen, varied population forces robustness across the strategy space, approximating a best response to a mixture of opponents.

Partial observability and decision-making under uncertainty

When the agent cannot see the full state because observations are occluded or sensors are limited, a single observation is no longer Markov. The agent can summarise history into an approximate belief, often with a recurrent core; AlphaStar used a deep LSTM for this purpose.

Guided checkpoint

The ad stresses 'Entscheidungsfindung unter unsicheren Bedingungen'. How is hidden state handled?

Section 07

Distributed RL: match each component to its role to Why do large actor-learner setups (e.g. IMPALA) need an off-policy correction like V-trace?

Distributed RL: match each component to its role

Ray is the distributed runtime (originally from UC Berkeley's RISELab, now maintained by Anyscale); RLlib is the scalable RL library built on it. The typical actor-learner split has many CPU rollout workers feeding a central GPU learner, with a replay buffer for off-policy algorithms. Being able to sketch this architecture maps straight onto the 'MLOps-getriebene KI-Trainings auf Kubernetes / Ray' bullet.

Guided checkpoint

The ad names Ray explicitly. Match each piece of a Ray/RLlib-style training stack to what it does.

Why is distributed computing (e.g. Ray) especially valuable for deep RL?

Deep RL is data-hungry and the simulator is frequently slower than the gradient step. Parallelising rollouts across many CPU workers (with the learner on a GPU) is the standard way to scale, as in IMPALA and APPO. This is exactly why a heavy simulation + RL shop reaches for Ray and Kubernetes.

Why do large actor-learner setups (e.g. IMPALA) need an off-policy correction like V-trace?

When dozens of actors run asynchronously, the data they send is from an older policy than the one the learner now holds (policy lag). V-trace (Espeholt et al., 2018) applies truncated importance-sampling weights so the value targets stay (approximately) unbiased despite the mismatch. This is a good 'I understand scaling RL' detail to have ready.

Section 08

MLOps and getting models into production to Why are sparse, delayed rewards a core RL challenge?

MLOps and getting models into production

Reproducible containers, cluster orchestration (Kubernetes, and KubeRay for Ray), and experiment/model versioning are the MLOps backbone. Simulation success never removes the need for real-world validation because of the sim-to-real gap (next question). Frame your CI/CD and cloud-deployment work here.

Guided checkpoint

This is where your dmTECH Docker / Terraform / CI-CD experience is directly relevant. Mark each statement.

What is the sim-to-real (reality) gap?

Simulators approximate reality, so a policy can overfit to quirks that do not exist in the field. Mitigations include domain randomisation (vary physics, sensors, appearance during training so the policy is forced to generalise), higher-fidelity models, and real-world fine-tuning. This is a central concern when 'übertragung akademischer Forschung auf reale Systeme' is your job.

Why are sparse, delayed rewards a core RL challenge?

If reward only arrives at the end of an episode, the agent struggles to know which of the hundreds of earlier actions mattered: that is temporal credit assignment. Reward shaping, curriculum learning, hierarchical RL, and intrinsic motivation are common remedies.

Section 09

Why train AI in simulation rather than on real operations? to Typical C++ / Python division of labour in a simulation + RL stack

Why train AI in simulation rather than on real operations?

Simulation is valuable for RL when real data are scarce, exploration is costly or unsafe, and thousands of parallel episodes can be generated. It does not remove the sim-to-real gap, so validation under distribution shift remains essential.

Guided checkpoint

The ad highlights 'sicher, datensparsam und zuverlässig'. Select all reasons that apply.

What makes StarCraft II a useful benchmark for complex multi-agent RL?

Partial observability, real-time pressure, combinatorial action spaces, multi-agent dynamics, and long-horizon planning make StarCraft II a demanding RL benchmark. There is no precomputed optimal action per state; that is what the agent has to learn.

Guided checkpoint

Select all properties that carry over to the operational setting.

Typical C++ / Python division of labour in a simulation + RL stack

The common pattern is a fast C++ simulation/engine (often a Qt front-end) exposed to Python via bindings (e.g. pybind11), with PyTorch/TensorFlow and RLlib on the Python side and NumPy/OpenCV for data and vision. Being comfortable crossing that boundary is a concrete strength to mention given your production-Python background.

Guided checkpoint

The ad asks for both C++ and Python, plus NumPy, OpenCV and Qt. How are they usually split?

Knowledge check

Turn understanding into recall.

The quiz now follows the same concepts in scored form. You can return to this lesson from the quiz whenever a gap appears.

Start the quiz →Back to course

What you will be able to explain

What formally defines a Markov Decision Process (MDP)? to What is the role of the discount factor γ\gammaγ?

What formally defines a Markov Decision Process (MDP)?

What does the Markov property assume?

What is the role of the discount factor γ\gammaγ?

Compute the discounted return to What does the Bellman equation express?

Compute the discounted return

Match each RL quantity to its definition

What does the Bellman equation express?

Sort algorithms: on-policy vs off-policy to Match each algorithm to its method family

Sort algorithms: on-policy vs off-policy

What distinguishes off-policy from on-policy RL?

Match each algorithm to its method family

Model-free vs model-based, and exploration to Which statements about PPO are correct?

Model-free vs model-based, and exploration

What is the exploration-exploitation trade-off?

Which statements about PPO are correct?

Why does PPO clip the probability ratio? to How was AlphaStar's policy initialised before reinforcement learning?

Why does PPO clip the probability ratio?

Why use an advantage function or baseline in policy-gradient methods?

How was AlphaStar's policy initialised before reinforcement learning?

Which statements about AlphaStar's League training are correct? to Partial observability and decision-making under uncertainty

Which statements about AlphaStar's League training are correct?

Why use a diverse league instead of naive self-play against the latest agent?

Partial observability and decision-making under uncertainty

Distributed RL: match each component to its role to Why do large actor-learner setups (e.g. IMPALA) need an off-policy correction like V-trace?

Distributed RL: match each component to its role

Why is distributed computing (e.g. Ray) especially valuable for deep RL?

Why do large actor-learner setups (e.g. IMPALA) need an off-policy correction like V-trace?

MLOps and getting models into production to Why are sparse, delayed rewards a core RL challenge?

MLOps and getting models into production

What is the sim-to-real (reality) gap?

Why are sparse, delayed rewards a core RL challenge?

Why train AI in simulation rather than on real operations? to Typical C++ / Python division of labour in a simulation + RL stack

Why train AI in simulation rather than on real operations?

What makes StarCraft II a useful benchmark for complex multi-agent RL?

Typical C++ / Python division of labour in a simulation + RL stack

Turn understanding into recall.

What formally defines a Markov Decision Process (MDP)? to What is the role of the discount factor $\gamma$ ?

What is the role of the discount factor $\gamma$ ?