Chunking in vector environments

[tristandeleu]

Chunking (ie. having synchronous environments inside vector environments) is impossible with the current implementations of AsyncVectorEnv and SyncVectorEnv. The benefits were discussed in openai/baselines#608. What would be the preferred way to go to add this functionality into Gym?

• Fix AsyncVectorEnv and SyncVectorEnv to allow nested vector environments, where you can use SyncVectorEnv inside either AsyncVectorEnv and SyncVectorEnv. This shouldn't break anything because VectorEnv already handles any observation/action space, but this would still make the shape of the observations with "2 batch dimensions". Transform that back to a single batch dimension could be easily done with a wrapper though. This would look like:

import gym

def make_vector_env():
    return gym.vector.make('CartPole-v1', num_envs=5, asynchronous=False)

env = gym.vector.AsyncVectorEnv([make_vector_env for _ in range(3)])
observations = env.reset()
print(observations.shape)
# (3, 5, 4)

• Add an argument to AsyncVectorEnv, like the one added in SubprocVecEnv in Make SubprocVecEnv works with DummyVecEnv baselines#908, to keep a single batch dimension.

What do you think @pzhokhov @christopherhesse ?

[jkterry1]

I'm going to close this just for issue management since a PR exists. I generally like this idea, I just don't want to merge this (or other PRs) until things are sorted out wrt the vector API. I plan to deal with this after the website and ALE-Py upgrade for a sense of time scale.