Support Streaming

[raphaeljolivet]

When using Flask Streaming capabilities, flask-compress gathers all the data with get_data() and returns everything at once, breaking the "streaming" paradigm.

You may integrate streaming compression as described in this answer (for deflate here) :
https://stackoverflow.com/a/44387566/254061

def generate_zip():
    compressor = zlib.compressobj()
    for x in range(10000):
        chunk = compressor.compress(f"this is my line: {x}\n".encode())
        if chunk:
            yield chunk
    yield compressor.flush()

[gilad9366]

Is this on the roadmap?
We are using flask-compress in our product and it breaks "streaming" for us as well.
It would be great if this was fixed as @raphaeljolivet suggested above, or at least skip the compression in case of a streamed response:

def after_request(self, response):
    app = self.app or current_app

    accept_encoding = request.headers.get('Accept-Encoding', '')
    chosen_algorithm = self._choose_compress_algorithm(accept_encoding)

    if (chosen_algorithm is None or
        response.mimetype not in app.config["COMPRESS_MIMETYPES"] or
        response.status_code < 200 or
        response.status_code >= 300 or
        "Content-Encoding" in response.headers or
        (response.content_length is not None and
         response.content_length < app.config["COMPRESS_MIN_SIZE"]) or
+++     response.is_streamed):
        return response

[raphaeljolivet]

@gilad9366 :

For the record, I ended up bypassing flask-compress for the few requests in which I do streaming.

def stream_compress(chunks) :
    compressor = zlib.compressobj()
    for data in chunks:
        out = compressor.compress(data.encode())
        if out:
            yield out
    yield compressor.flush()


@route("/rest/data")
def get_data() :
  
  # Generates an iterator with "yield"
  chunks = stream_data()

  headers = dict()
  if "deflate" in request.headers.get('Accept-Encoding', ''):
      headers["Content-Encoding"] = "deflate"
      out = stream_compress(chunks)
  
  return Response(
        out,
        mimetype=mimetype,
        headers=headers)

[alexprengere]

If anyone comes up with a PR that implements streaming without breaking existing code, I will be happy to review it.
Or we could indeed just drop compression if response.is_streamed is True, as @gilad9366 suggested.

[gilad9366]

@alexprengere Working on supporting streaming.
I'll send a PR once I'm done.

[gilad9366]

@alexprengere @raphaeljolivet
So after some research, it seems like implementing compression during streaming might not be possible.
The issue is that flask (and basically any python framework) doesn't support Transfer Encoding:
Flask implemented a workaround with generators but doesn't follow the HTTP streaming protocol
For that same reason, clients won't be able to decompress "streamed" responses from flask since they don't follow the streaming and compression RFC.
I tried implementing streaming with flask, and tested with curl but it broke the stream.

Instead, I think we should just skip compression if response.is_streamed is True as I originally suggested.

[alexprengere]

Thanks a lot for your investigation, I merged your PR.

[raphaeljolivet]

@gilad9366 : Well I disagree.
The solution I implemented here works fine. There is nothing specific to Flask that prevents this : setting the proper Http Headers and then sending the proper bytes in chunks enables streaming to compressed content.

[gilad9366]

@raphaeljolivet Are you sure it's actually streaming to the client and not streaming only to the WSGI middleware and then whole thing compressed at once to the client?
I tried running your code and the streaming broke.

[raphaeljolivet]

It is really streaming. I can see the size growing on the client size. ------- Original Message ------- Le mercredi 13 avril 2022 à 3:02 PM, gilad9366 ***@***.***> a écrit :

…

***@***.***(https://github.com/raphaeljolivet) Are you sure it's actually streaming to the client and not streaming only to the WSGI middleware and then whole thing compressed at once to the client? I tried running your code and the streaming broke. — Reply to this email directly, [view it on GitHub](#30 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAJFLQF3HSRYD73H2MYGPP3VE3AXLANCNFSM5QJWUXUA). You are receiving this because you were mentioned.Message ID: ***@***.***>

[alexprengere]

@raphaeljolivet if you want to open a PR, I can take a look at it.

[raphaeljolivet]

Here we go !
#32