Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modules/zstd: Add initial ZSTD Encoder procs #1846

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from

Conversation

lpawelcz
Copy link
Contributor

This PR adds initial procs of the ZSTD Encoder. It's based on #1654.

Changes in this PR consist of:

  • Adding AlignedParallelRam proc - Responsible for handling Read/Write operations on multiple RAM models in parallel.
  • Adding HashTable proc - Generic Hash Table implementation. It uses underlying RAM models for operation and is controlled with a Read/Write Request/Response channels interface. In the ZSTD Encoder, it is used for fast lookup of repeating symbols in the input data buffer.
  • Adding HistoryBuffer proc - Data buffer capable of storing historical data and outputting said data from arbitrary chosen offset up to the top of the buffer. Uses AlignedParallelRam proc underneath.
  • Adding MatchFinder proc - Processes data stream from the input buffer in the system memory. Searches for repeating symbol patterns in the HashTable proc. Symbols that are not found are stored directly as literals in the literals buffer, while those that have matches are stored as sequences in the sequence buffer, using the matched offset and length information. Sequence and Literals Buffers are implemented in the system memory accessible via the AXI interface.

This PR is currently a WIP.
The following work needs to be finished to mark this PR as ready for review:

  • Fix MatchFinder codegen
  • Enable back the IR benchmarks, codegen, synthesis and P&R targets in the CI

koblonczek and others added 30 commits January 13, 2025 12:29
- XLSStruct for easier handling and serializing/deserializing XLS structs
- XLSChannel that serves as a dummy receiving channel
- XLSMonitor that monitors transactions on an XLS channel
- XLSDriver that can send data on an XLS channel
- LatencyScoreboard that can measure latency between corresponding transactions on input and output buses
- File-backed AXI memory python model

Internal-tag: [#64075]
Signed-off-by: Krzysztof Obłonczek <[email protected]>
Co-authred-by: Pawel Czarnecki <[email protected]>
Co-authred-by: Robert Winkler <[email protected]>
Signed-off-by: Michal Czyz <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Remove references to buffer structs as those are not used anywhere

Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]>
Co-authored-by: Robert Winkler <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Krzysztof Oblonczek <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]>
Co-authored-by: Robert Winkler <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
This reverts commit 04ad379225b706ddf492d440c673e77348d7a409.
Internal-tag: [#67096]
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Signed-off-by: Krzysztof Obłonczek <[email protected]>
Internal-tag: [#66955]
Signed-off-by: Pawel Czarnecki <[email protected]>
lpawelcz and others added 16 commits January 13, 2025 12:35
* Fix byte ordering when receiving a series of non-empty packets
* Adjust MemReader DSLX tests

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
…es function into a separate proc

* Extract the operation of removing not-strobed bytes from input frames
  to a separate proc
* Extract control logic to AxiStreamRemoveEmptyInternal proc
* Optimize strobe calculation

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
Fix paramaterization of the proc

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
…ackets

Add AxiStreamRemoveEmpty proc to the processing pipeline. It removes non-strobed
bytes from the input AXI Stream frames and forms full frames (ensures that only
the last input data packet won't be full).

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
* Extract control logic to MemWriterInternal proc
* Create alias for the MemWriter response type

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
SequenceExecutor:
  * Add output channel in the format compliant with MemWriter input data channel type

ZstdDecoder:
  * Add MemWriter proc:
    * Write request formed based on the address of the OutputBuffer CSR and
      FrameContentSize field from the Frame Header
    * Data to write is sent out to the proc by the SequenceExecutor
  * Transition to the FINISH state (and triggers notify channel) only after
    receiving the response from the MemWriter
  * DSLX tests:
    * Receive decoded data sent out on the AXI interface by the MemWriter proc
    * Mock the output memory buffer as a DSLX array
  * Cocotb tests:
    * Move third-party verilog modules (AXI Interconnect) to external directory
    * Replace AXI Interconnect with AXI Crossbar that handles simultaneous AXI Read
      and Write transactions
    * Add reference memory and fill it with expected data for comparison
      against testbench memory at the end of the decoding

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
* Remove Repacketizer proc
* Remove stream-based output channels from
  * SequenceExecutor
  * ZstdDecoder

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
* Decode multiple ZSTD frames in a single cocotb testbench
* Add one cocotb testbench per type of the ZSTD frames:
  * Frames with RAW blocks only
  * Frames with RLE blocks only
  * Frames with Compressed blocks only (disabled)
  * Frames with mixed blocks (disabled)

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <[email protected]>
…on for verilog library

Signed-off-by: Pawel Czarnecki <[email protected]>
…st cases

Caused by timeouts after rebase - looks like regression in the
performance of codegen

Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants