You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I was working on code to copy tiles out of a md5-hash prefixed bucket to a "regular" bucket and noticed that creating a session once per tile was eating up a lot of time, so instead I used a thread.local to store the boto session and initialized it once in the initializer for the Pool.
This took 75%+ off the time to do the simple copy operation. It might not take quite as much time off the pyramid renderer because there's more time spent on the actual rendering, but it would still be useful to do in the rendering scripts.
The text was updated successfully, but these errors were encountered:
In
render_pyramid.py
I am creating one boto3 session per tile in order to handle the fact that boto3's session is not threadsafe. https://github.com/mojodna/marblecutter/blob/tilezen/examples/render_pyramid.py#L193I was working on code to copy tiles out of a md5-hash prefixed bucket to a "regular" bucket and noticed that creating a session once per tile was eating up a lot of time, so instead I used a
thread.local
to store the boto session and initialized it once in theinitializer
for thePool
.This took 75%+ off the time to do the simple copy operation. It might not take quite as much time off the pyramid renderer because there's more time spent on the actual rendering, but it would still be useful to do in the rendering scripts.
The text was updated successfully, but these errors were encountered: