[Go] Estimate physical size of data accumulated in array.Builder

[vitalyisaev2]

Describe the usage question you have. Please include as many useful details as possible.

I'm working on a streaming service that accumulates blocks of Arrow data, dumps them into Arrow Streaming IPC format and then writes them into network socket. The data is accumulated in []array.Builder, which converts into []arrow.Array, then into a single arrow.Record and finally into bytes.Buffer.

The key part of this algorithm is the separation of a new block: we need to know when the previous block is large enough (but not too large - because it could exceed message size limits in this case) to let us stop data accumulation in []array.Builder, serialize existing data into bytes.Buffer and send over the network.

So is it possible to estimate the number of bytes accumulated physically in array.Builder? Or may be it even possible to forecast the size of a future bytes.Buffer containing serialized data from []array.Builder?

Component(s)

Go