10.2:Off-policy策略梯度定理的证明
由于\(d_\beta\)与\(\theta\)无关，\(J(\theta)\)的梯度满足
\[\nabla_\theta J(\theta)=\nabla_\theta\sum_{s\in\mathcal{S}}d_\beta(s)v_\pi(s)
=\sum_{s\in\mathcal{S}}d_\beta(s)\nabla_\theta v_\pi(s).\tag{10.12}\]
根据引理9.2，
\[\nabla_\theta v_\pi(s)
=\sum_{s'\in\mathcal{S}}\Pr_\pi(s'|s)\sum_{a\in\mathcal{A}}\nabla_\theta\pi(a|s',\theta)q_\pi(s',a),\tag{10.13}\]
其中
\[\Pr_\pi(s'|s)\doteq \sum_{k=0}^{\infty}\gamma^k[P_\pi^k]_{ss'}=[(I_n-\gamma P_\pi)^{-1}]_{ss'}.\]
将式\((10.13)\)代入式\((10.12)\)可得
\[\begin{aligned}
\nabla_\theta J(\theta)
&=\sum_{s\in\mathcal{S}}d_\beta(s)\sum_{s'\in\mathcal{S}}\Pr_\pi(s'|s)
\sum_{a\in\mathcal{A}}\nabla_\theta\pi(a|s',\theta)q_\pi(s',a)\\
&=\sum_{s'\in\mathcal{S}}\left[\sum_{s\in\mathcal{S}}d_\beta(s)\Pr_\pi(s'|s)\right]
\sum_{a\in\mathcal{A}}\nabla_\theta\pi(a|s',\theta)q_\pi(s',a)\\
&\doteq \sum_{s'\in\mathcal{S}}\rho(s')\sum_{a\in\mathcal{A}}\nabla_\theta\pi(a|s',\theta)q_\pi(s',a)\\
&=\sum_{s\in\mathcal{S}}\rho(s)\sum_{a\in\mathcal{A}}\nabla_\theta\pi(a|s,\theta)q_\pi(s,a)\\
&=\mathbb{E}_{S\sim\rho}\left[\sum_{a\in\mathcal{A}}\nabla_\theta\pi(a|S,\theta)q_\pi(S,a)\right].
\end{aligned}\]
利用重要性采样，上式进一步改写为
\[\begin{aligned}
&\mathbb{E}_{S\sim\rho}\left[\sum_{a\in\mathcal{A}}\nabla_\theta\pi(a|S,\theta)q_\pi(S,a)\right]\\
&=\mathbb{E}_{S\sim\rho}\left[\sum_{a\in\mathcal{A}}\beta(a|S)\frac{\pi(a|S,\theta)}{\beta(a|S)}\nabla_\theta\ln\pi(a|S,\theta)q_\pi(S,a)\right]\\
&=\mathbb{E}_{S\sim\rho,A\sim\beta}\left[
\frac{\pi(A|S,\theta)}{\beta(A|S)}
\nabla_\theta\ln\pi(A|S,\theta)q_\pi(S,A)
\right].
\end{aligned}\]
证明完毕。该证明与定理9.1的证明类似。