Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-37756: [Format][Docs] Document IPC Compression #43950

Merged
merged 18 commits into from
Sep 17, 2024
Merged

GH-37756: [Format][Docs] Document IPC Compression #43950

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
2f7620b
Add Compression paragraph to the Columnar.rst IPC section
AlenkaF Sep 4, 2024
bc75ce5
Update docs/source/format/Columnar.rst
AlenkaF Sep 4, 2024
b4375f5
Update docs/source/format/Columnar.rst
AlenkaF Sep 5, 2024
01aa4c8
Use only buffer instead of memory buffer
AlenkaF Sep 5, 2024
83257b3
Add info about endian and signed for the length in the msg body
AlenkaF Sep 5, 2024
d9ad1b2
Fix links, add link to RecordBatch msg section and add note about lz4…
AlenkaF Sep 5, 2024
cb50c9d
Update docs/source/format/Columnar.rst
AlenkaF Sep 9, 2024
3eb3792
Add missing newline
ianmcook Sep 9, 2024
6df2797
Add missing newline
ianmcook Sep 9, 2024
4e06abb
Apply suggestions from code review
AlenkaF Sep 10, 2024
bcb614d
Update docs/source/format/Columnar.rst
AlenkaF Sep 11, 2024
6fb80b0
Add info about where compression is stored
AlenkaF Sep 11, 2024
99b7db5
Update body info for first 8 bytes
AlenkaF Sep 11, 2024
b87a17c
Remove reference to empty 8 bytes
AlenkaF Sep 12, 2024
fb03ef0
Apply suggestions from code review
AlenkaF Sep 12, 2024
739f610
Add compression default
AlenkaF Sep 12, 2024
e684f54
Update docs/source/format/Columnar.rst
AlenkaF Sep 13, 2024
e4e9629
Remove errant backtick
ianmcook Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions docs/source/format/Columnar.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1388,11 +1388,12 @@ have two entries in each RecordBatch. For a RecordBatch of this schema with
Compression
-----------

There are three different options for record batch body
buffers compression: buffers can be uncompressed, can use
``lz4`` or ``zstd`` compression codec. All buffers in the flat
sequence of the message body are compressed separately with the
same codec.
There are three different options for compression of record batch
body buffers: Buffers can be uncompressed, buffers can be
compressed with the``lz4`` compression codec, or buffers can
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved
be compressed with the ``zstd`` compression codec. Buffers in
the flat sequence of a message body must be either all
uncompressed or all compressed separately using the same codec.
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved

The difference between compressed and uncompressed buffers in the
serialized form is as follows:
Expand Down
Loading