What is the difference between off-policy and on-policy algorithms?

On-policy methods attempt to evaluate or improve the policy that is used to make decisions. In contrast, off-policy methods evaluate or improve a policy different from that used to generate the data.

What is the difference in value iteration and policy iteration methods in reinforcement learning?

In policy iteration, we start with a fixed policy. Conversely, in value iteration, we begin by selecting the value function. Then, in both algorithms, we iteratively improve until we reach convergence.

What is an off-policy algorithm?

Off-Policy learning algorithms evaluate and improve a policy that is different from Policy that is used for action selection. In short, [Target Policy != Behavior Policy]. Some examples of Off-Policy learning algorithms are Q learning, expected sarsa(can act in both ways), etc.

What is the basic difference between Sarsa and Q learning can you think of a real life learning analogy that could help explain the difference between these two strategies?

The most important difference between the two is how Q is updated after each action. SARSA uses the Q’ following a ε-greedy policy exactly, as A’ is drawn from it. In contrast, Q-learning uses the maximum Q’ over all possible actions for the next step.

Is PPO on-policy or off-policy?

TRPO and PPO are both on-policy. Basically they optimize a first-order approximation of the expected return while carefully ensuring that the approximation does not deviate too far from the underlying objective.

Is Monte Carlo on-policy?

The Monte Carlo ES method with exploring starts described above is an example of an on-policy method.

How do I do a policy iteration?

In policy iteration algorithms, you start with a random policy, then find the value function of that policy (policy evaluation step), then find a new (improved) policy based on the previous value function, and so on.

Is policy iteration on policy?

Policy Iteration takes an initial policy, evaluates it, and then uses those values to create an improved policy. These steps of evaluation and improvement are then repeated on the newly generated policy to give an even better policy. This process continues until, eventually, we end up with the optimal policy.

Is PPO on policy or off-policy?