Gradient issue

[GoingMyWay]

When setting

"actor_feature_reg": None,

and run the simple env, it will report

ValueError: No gradients provided for any variable: ["<tf.Variable 'policy_0/actor/policy_0/actor_feature/dense/kernel:0' shape=(4, 64) dtype=float32_ref>", "<tf.Variable 'policy_0/actor/policy_0/actor_feature/dense/bias:0' shape=(64,) dtype=float32_ref>", "<tf.Variable 'policy_0/actor/policy_0/actor_feature/dense_1/kernel:0' shape=(64, 64) dtype=float32_ref>", "<tf.Variable 'policy_0/actor/policy_0/actor_feature/dense_1/bias:0' shape=(64,) dtype=float32_ref>", "<tf.Variable 'policy_0/actor/dense/kernel:0' shape=(64, 5) dtype=float32_ref>", "<tf.Variable 'policy_0/actor/dense/bias:0' shape=(5,) dtype=float32_ref>"].

So, I think it quite confusing.

        act_n = act_ph_n.copy()
        act_n[agent_id] = act_sampler
        critic, _, _, _ = self._build_critic_network(
            obs_ph_n,
            act_n,
            obs_space_n,
            act_space_n,
            config["use_state_preprocessor"],
            config["critic_hiddens"],
            getattr(tf.nn, config["critic_hidden_activation"]),
            scope="critic")

Since act_sampler is set and the gradient will be computed.