Reinforcement learning

Bandit

Learner only observes the reward for the actual action taken, instead of the reward of all possible actions. The reward is often noisy since the learner interacts with an environment.

Example: advertisement placement. learner shows the user an advertisement, want the user to click on it and generate revenue

• missing rewards: learner does not get to see if the user would have clicked on other ads that are not shown
• noisy rewards: the user might click on a bad ad by accident, or not click on the good ones

Exploration exploitation trade-off. An agent can exploit by taking what seems to be the best action, or explore and gain more information on alternative actions. A particular simple form is the $\epsilon$-greedy method, that always explore with probability $\epsilon$.

One can reduce supervised learning to the contextual bandit setting by paying a factor of $|Y|$. Given dataset ${x_i, y_i}_i$, we will withhold $y$ from the learner and instead only tell the learner if it predicted correctly.

• learner is given $x$ and makes a prediction $\hat{y}$.
• learner receives $R(y, \hat{y})$.

With agents that explore and learn under noisy and partial rewards, it is natural to consider agents that make a sequence of actions, and only receive a reward sometimes.

Markov decision process

At step $t$ the environment is in state $s_t \in S$, agent takes actions $a_t \in A$ in an environment and receive a reward $r_t$ based on the action. When this is rolled out in time steps,

$$s_0, a_0, r_1, s_1, a_1, r_2, s_2, a_3, r_3, \ldots$$

When the environment is specified by the transition probability $p(s' \mid s, a) = \operatorname{Pr}(s_{t+1}=s' \mid s_t = s, a_t = a)$ and the expected reward $r(s, a, s')$, the problem of computing what action to take in each state to maximize the total reward is called a Markov decision process (MDP). The solution $\pi(a | s)$ is called a policy, which is a distribution of actions for each given state. When the policy is deterministic, it is just function from state to action $\pi(s) \in A$.

Need for random policy. Suppose the states are arranged spatially as left, next-to-mid, mid, next-to-mid, right, and the reward at mid is high.

Dice game. actions are quit and stay

• quit: get reward 10, and end
• stay: get reward 4, and end if dice rolls 1 or 2 Should you always play stay?

Need for discounting. For example, we might want to maximize the lifetime earning of some ad placement method. The lifespan might be long, and in any event there is no natural ending. In this case, we are forced to consider the time value as well, and discount future rewards by a factor of $\gamma$.

Value of a policy

The value function $v_\pi(s)$ of a policy $\pi$ is is the (expected) sum of all future reward starting with state $s$ and following policy $\pi$. The value function and policy together satisfy the Bellman equation.

$$\begin{aligned} v_\pi(s) &=\sum_{s'} p(s' \mid s, \pi(s)) (r(s, \pi(s), s') + \gamma v_\pi(s')) \quad \text{deterministic policy} \\ v_\pi(s) &= \sum_a \pi(a|s) \sum_{s'} p(s' \mid s, a) (r(s, a, s') + \gamma v_\pi(s')) \quad \text{stochastic policy} \end{aligned}$$

• This gives a set of linear equations, could just solve for $v$
• alternatively, because the simple form, one can just do iterative updates until convergence

$$v'_{\pi}(s) \gets \sum_{s'} p(s' \mid s, \pi(s)) (r(s, \pi(s), s') + \gamma v_\pi(s')).$$

The complexity per round is $O(|S|^2)$.

Value iteration

The Bellman equation gives us the value function for a given policy. However, actually solving an MDP means we would like to compute the policy. We want

$$v_*(s) = \max_\pi{v_\pi(s)} = \max_a q_*(s,a)$$

Instead, we can consider the value as a function of a particular action taken, and the Bellman equation for the optimal policy

$$\begin{aligned} q_*(s,a) &=\sum_{s'} p(s' \mid s, a) (r(s, a, s') + \gamma v_*(s'))\\ v_*(s) &= \max_a q_*(s, a)\\ &= \max_a \sum_{s'} p(s' \mid s, a) (r(s, a, s') + \gamma v_*(s')) \end{aligned}$$

As before, this can be computed iteratively, but the complexity per round is now a bit higher, at $O(|S|^2 |A|)$.

Exercise

• value of the dice game
• value of a grid game

Games. Reward (negative if the other player wins), only when the win condition is satisfied (3 in a row, column or diagonal).

In a symmetric game, an RL algorithm take both sides and play against itself with both sides learning. With a game that has clearly defined rules, RL is very attractive since infinite data is available.

• What are the states? is discounting necessary?

Limitations. Which problems are not captured by a MDP?

RL in NLP

Since RL is a very general framework, many NLP problems can be formulated in terms of RL. To better understand when we might need RL, it is worth considering why we did not need RL in many problems.

• language modeling
• pasing

Dialogue. Agent and customer take turns in a dialogue to solve a customer problem. Receive reward in the end based on if the problem was solved. Unfortunately, there is no good simulator for the customer since the only working NL processor is the human, and human in the loop means slow iterations and expensive data. Some important feedback is not easily converted to a reward e.g. “No, I meant …”, “Yes, please”.

TBC