👷 Deploy request for catanatron-staging pending review.

Visit the deploys page to approve it

Hey! Thanks for thinking about this! I think the Pickle+Filesystem idea is interesting.

Let's not remove the database infrastructure because its needed to deploy catanatron as a web app in the cloud (most application deployment solutions out-of-the-box provide ephemeral filesystems), plus might as well keep it as an option for users.

Can you thus instead make this pull request just add the --pickle option to use the filesystem for persistance (in addition to --json, --db, ...)? Also, I'd like for Pull Requests to be small and single-focused, so I'd say lets first add the --pickle backend, and then we worry about the Frontend. It can be in a separate PR if it helps. Let's also not change the default use of it to be pickle just yet; again lets present it as a new option.

How much faster is using the filesystem than the database?

I also try to keep pull requests as small as possible, but the reason I grouped these together is because implementing state-navigation requires saving all intermediary states, which is with a db simply not feasible. Using pickle requires practically no overhead, in other words it runs almost as fast as no saving at all (you can try it on the pr branch). Using a db, however, is so slow it's unusable (>2s per state).

As for the web app, I can find a way to keep the app itself connected to a db, yet I ultimately believe the focus should be on training a catan AI first, rather than optimising a web app.

I'll look into breaking this into a pickle specific PR first.

what's the performance of pickling the game vs some other serialization format? (ideally one that doesn't allow arbitrary code execution)

what's the performance of pickling the game vs some other serialization format? (ideally one that doesn't allow arbitrary code execution)

I haven't tested different methods, you can suggest something if you know. Again, given that this is a research project, security isn't the main concern and anyone using the script should know what they are doing.

Feels like we should lean away from pickling as a long-term solution bc it allows arbitrary code execution. I haven't messed around with the db much, but it seems like that would be the better direction to take. Anybody have a clearer idea on what the scope would be to make that happen?
Also, in order to test those changes locally, could you run the db instance thru Docker and run the front/backend from a terminal? Or is it necessary to use docker-compose to work on them locally?