10.5:定理10.4的证明
由于策略是确定性的，有
\[v_\mu(s)=q_\mu(s,\mu(s)).\]
因为\(q_\mu\)和\(\mu\)都是\(\theta\)的函数，所以
\[\nabla_\theta v_\mu(s)
=\nabla_\theta q_\mu(s,\mu(s))
=\nabla_\theta q_\mu(s,a)|_{a=\mu(s)}
+\nabla_\theta\mu(s)\nabla_aq_\mu(s,a)|_{a=\mu(s)}.\tag{10.21}\]
在无折扣情形下，根据第\(9.3.2\)节中行动值的定义，有
\[\begin{aligned}
q_\mu(s,a)
&=\mathbb{E}[R_{t+1}-\bar{r}_\mu+v_\mu(S_{t+1})|s,a]\\
&=\sum_r p(r|s,a)(r-\bar{r}_\mu)+\sum_{s'}p(s'|s,a)v_\mu(s')\\
&=r(s,a)-\bar{r}_\mu+\sum_{s'}p(s'|s,a)v_\mu(s').
\end{aligned}\]
由于\(r(s,a)=\sum_r rp(r|s,a)\)与\(\theta\)无关，
\[\nabla_\theta q_\mu(s,a)
=-\nabla_\theta\bar{r}_\mu+\sum_{s'}p(s'|s,a)\nabla_\theta v_\mu(s').\]
代入式\((10.21)\)可得
\[\nabla_\theta v_\mu(s)
=-\nabla_\theta\bar{r}_\mu
\sum_{s'}p(s'|s,\mu(s))\nabla_\theta v_\mu(s')
+\underbrace{\nabla_\theta\mu(s)\nabla_aq_\mu(s,a)|_{a=\mu(s)}}_{u(s)}.
\]
该式对所有\(s\in\mathcal{S}\)都成立，因此可写成矩阵-向量形式：
\[\nabla_\theta v_\mu=u-\mathbf{1}_n\otimes\nabla_\theta\bar{r}_\mu+(P_\mu\otimes I_m)\nabla_\theta v_\mu.\]
于是
\[\mathbf{1}_n\otimes\nabla_\theta\bar{r}_\mu
=u+(P_\mu\otimes I_m)\nabla_\theta v_\mu-\nabla_\theta v_\mu.\tag{10.22}\]
由于\(d_\mu\)是稳态分布，有
\[d_\mu^TP_\mu=d_\mu^T.\]
在式\((10.22)\)两边同时左乘\(d_\mu^T\otimes I_m\)，可得
\[\begin{aligned}
(d_\mu^T\mathbf{1}_n)\otimes\nabla_\theta\bar{r}_\mu
&=(d_\mu^T\otimes I_m)u+(d_\mu^TP_\mu\otimes I_m)\nabla_\theta v_\mu-(d_\mu^T\otimes I_m)\nabla_\theta v_\mu\\
&=(d_\mu^T\otimes I_m)u.
\end{aligned}\]
由于\(d_\mu^T\mathbf{1}_n=1\)，上式变为
\[\begin{aligned}
\nabla_\theta\bar{r}_\mu
&=(d_\mu^T\otimes I_m)u\\
&=\sum_{s\in\mathcal{S}}d_\mu(s)u(s)\\
&=\sum_{s\in\mathcal{S}}d_\mu(s)\nabla_\theta\mu(s)\nabla_aq_\mu(s,a)|_{a=\mu(s)}\\
&=\mathbb{E}_{S\sim d_\mu}\left[\nabla_\theta\mu(S)\nabla_aq_\mu(S,a)|_{a=\mu(S)}\right].
\end{aligned}\]
证明完毕。