EnvPool advertisement

[Trinkle23897]

Hi, I just came across this repo. I'm quite surprised that you use envpool to achieve 2 min Pong and 20min Breakout, nice work!

I'm wondering if you'd like to open a pull request at EnvPool to link with your result (like the CleanRL ones), and if it is possible for us to include your experiment result in our incoming arXiv paper. Also, it would be great if you can make more amazing results based on EnvPool mujoco tasks (which has aligned with gym's implementation and can also get a free speedup). Thanks!

BTW, isn't it a typo?
https://github.com/Denys88/rl_games/blame/master/docs/ATARI_ENVPOOL.md#L9

-* **Breakout-v3** 20 minutes training time to achieve 20+ score.
+* **Breakout-v3** 20 minutes training time to achieve 400+ score.

[Denys88]

Oh yes it is a typo! Thanks.
Envpool is reallyt fast! We will prepare pr till the end of the week.
We already got very good results with envpool mujoco humanoid:

We are going to post updated results today/tomorrow.
Right now I am working on the google colab example which trains walker2d with envpool in a few minutes. And I need to switch to the original mujoco-py for visualization.
Also It would be nice to have a render function for envpool envs.
Humanoid policy doesnt work with original mujoco-py env. Other envs are good.
Mujoco-py used outdated version of the mujoco 210 vs 215 in envpool.

[Trinkle23897]

Btw, could you please use the newest version (0.6.1.post1) to verify the final reward on ant-v3 and humanoid-v3? Some changes have been made but I'm not sure whether it will break the consistency.

[ViktorM]

@Trinkle23897 the Breakout-v3 typo was fixed. And btw this Mujoco humanoid result I got by training on my laptop with 11th Gen Intel® Core™ i9-11980HK @ 2.60GHz × 16 and RTX 3080, it was not even a desktop. Training with envpool was extremely fast.

Just started training Mujoco Humanoid with the latest envpool. You updated it really fast to the new opensourced Mujoco version!

[ViktorM]

@Trinkle23897 humanoid works the same well, blue is the new run.

[ViktorM]

Ant also works well!

[Trinkle23897]

Great! Would you like to be one of the authors of our paper?

[ViktorM]

I and Denys would happy to contribute to being co-authors of the paper with you.

[Denys88]

btw, you can join our discord too https://discord.gg/hnYRq7DsQh

[Trinkle23897]

Another request: I'm trying to use mujoco source code to build envpool. However, there are some small precision issues (google-deepmind/mujoco#294). The corresponding wheels are in https://github.com/sail-sg/envpool/actions/runs/2381544251
Not sure if it will affect the benchmark result. If possible, could you please also run some experiments to verify?

[ViktorM]

@Trinkle23897 I can test ant and humanoid after finishing with the ongoing experiments. Btw do you plan to support DM_control multi-agent envs: https://github.com/deepmind/dm_control/blob/main/dm_control/locomotion/soccer/README.md ?

If yes we can run self-play experiments with rl_games and envpool as well, for the simplest env.

[Benjamin-eecs]

@ViktorM Yes, envpool plans to support all tasks in dm_control.locomotion, the multi-agent soccer will be supported too. It can be one of the multi-agent env that envpool supported.

[ViktorM]

@Benjamin-eecs thank you! Looking forward for o soccer with envpool. We already had some interesting results with the simples boxhead version 1x1. With envpool speed up we’ll be able to train 2x2 and maybe even ant version!