Avro codec enhancements + Avro Writer Schema Generator

[jecsand838]

Which issue does this PR close?

Part of #4886

Rationale for this change

The primary objective of this PR is to enhance the arrow-rs Avro implementation by introducing full support for Avro data types, support for Avro aliases, and laying the foundations for an Avro writer. These enhancements are crucial for several reasons:

1. Enhanced Data Interoperability:
By supporting these additional types, the Avro reader becomes more compatible with a wider range of Avro schemas. This ensures that users can ingest and process diverse datasets without encountering type-related limitations.

2. Foundational Support for an Avro Writer:
This PR lays the groundwork for writing Arrow data structures to Avro format by introducing foundational support for an Avro writer. Previously, the focus was primarily on reading and converting Avro data into Arrow's in-memory format. Extending this functionality to include writing ensures bidirectional interoperability between Arrow and Avro.

What changes are included in this PR?

Avro Codec

1. Support for new types:
   • Decimals
   • Maps
   • Enums
   • UUID (in-progress)
2. Extended namespace support to all supported types
3. Added Alias support (in-progress)
4. Added Arrow Schema -> Avro Schema mapping logic
5. Extended support for local timestamps
6. Added Unit tests

Avro Record Decoder

1. Support for new types:
   • Lists
   • Fixed
   • Interval
   • Decimal
   • Map
   • Enum
   • UUID (in-progress)
2. Expanded nullability support
3. Added Unit tests

Avro Writer Foundations

1. Setup basic initial foundation for writer:
   • mod file
   • VLQ Encoder
2. Added initial support for Arrow -> Avro JSON Schema builder
3. Added Unit tests

Are there any user-facing changes?

No, none of these types are public yet

[tustvold]

👋 thank you for working on this, it is very exciting to see arrow-avro getting some love. I think what would probably help is to break this up into smaller pieces that can be delivered separately. Whilst I accept this is more work for you, it makes reviewing the code much more practical, especially given our relatively limited review bandwidth. Many of the bullets in your PR description would warrant a separate PR IMO.

[alamb]

Thank you @jecsand838 I triggered CI for this PR

I haven't had a chance to review the code yet, but I did start to look at the test coverage. What would you think about adding tests using the existing avro testing data in https://github.com/apache/arrow-testing/tree/master/data/avro (already a submodule in this repo)

Key tests in my mind would be:

1. Read the avro testing files and verify the schema and data read (and leave comments for tests that don't pass)
2. For the writer implement round trip tests: create a RecordBatch (es), write it to an .avro file and then read it back in and ensure the round tripped batches are equal

You might be able to find code / testing that you can reuse in the datafusion copy: https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs

I also wonder how/if this code is related to the avro rust reader/decoder in https://github.com/apache/avro-rs?

FYI @Jefffrey

[alamb]

After some review of this PR, I think I could likely find some additional time to review / help it along if/when it has end to end tests of reading existing avro files (that would give me confidence that the code being reviewed did the right thing functionally).

To make a specific proposal for splitting up this PRs' functionality as suggested by @tustvold -- #6965 (comment), one way to do so would be:

1. First PR: Reader improvements / tests showing reading existing .avro files
2. Second PR: Minimal Writer support (with tests showing round tripping for a few basic data types)
3. Subsequent PRs: Additional PRs to support writing additional data types

The rationale for breaking it up this way would be to have better confidence in the read path to ensure round tripping also works as needed

Let me know what you think

And thanks again