Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create boto session once per thread #89

Open
iandees opened this issue Nov 3, 2017 · 1 comment
Open

Create boto session once per thread #89

iandees opened this issue Nov 3, 2017 · 1 comment
Labels

Comments

@iandees
Copy link
Collaborator

iandees commented Nov 3, 2017

In render_pyramid.py I am creating one boto3 session per tile in order to handle the fact that boto3's session is not threadsafe. https://github.com/mojodna/marblecutter/blob/tilezen/examples/render_pyramid.py#L193

I was working on code to copy tiles out of a md5-hash prefixed bucket to a "regular" bucket and noticed that creating a session once per tile was eating up a lot of time, so instead I used a thread.local to store the boto session and initialized it once in the initializer for the Pool.

This took 75%+ off the time to do the simple copy operation. It might not take quite as much time off the pyramid renderer because there's more time spent on the actual rendering, but it would still be useful to do in the rendering scripts.

@nvkelso nvkelso added this to the Mapzen aerial composite milestone Dec 14, 2017
@nvkelso nvkelso added the p2 label Dec 15, 2017
@nvkelso
Copy link
Collaborator

nvkelso commented Dec 22, 2017

This is part of bulk renderer. This goes away using the MBTiles route.

@nvkelso nvkelso removed this from the Mapzen aerial composite milestone Dec 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants