WARC lookup for remote zipfile (WACZ)

[leewesleyv]

Resolves #21

To account for large zipfiles (WACZ) we are implementing lookup of the index/WARC record using range requests for remote files.

• Add a storage handler class that abstracts logic to allow handling local and remote zipfiles
• Implement logic for requesting partial zip data and parsing the data low level instead of using Python's build-in zipfile module (does not seem to be compatible without having the entire file).
• Add unit-tests

Implement different storages handlers

• Implement local storage handler class
• Implement S3 storage handler class
  Implement GC storage handler class
  Implement Azure storage handler class
  Implement FTP storage handler class

Useful links

• The ZIP specification https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
• Interesting blog article https://blog.yaakov.online/zip64-go-big-or-go-home/

[wvengen]

Wow, impressive you got this, with a streaming zip-reading implementation! Looking good, at first glance.

I'm still a bit surprised that there is no existing streaming-zip-over-http implementation, but it happens that sometimes some real engineering is still necessary :) 🚀

[wvengen]

I would say, GC/Azure/FTP are nice to haves - so feel free to add them, but I would not spend a long long time if you come across blocking issues.