Add support for complex nested types in List Arrays and Struct Arrays in avro_to_arrow

[ameyc]

Is your feature request related to a problem or challenge?

We are currently working on a stream processing system built atop DataFusion and as such Avro is a major format for us given its ubiquity in the Kafka world. We tried using the the existing Avro Reader in data fusion, however found it lacking in some critical ways that make not terribly useful for us in its present state.

The reader currently does not support complex nested datatypes such as -

1. The List arrays only support primitive types
2. Dictionary arrays only support Utf8 as its value types.

Lastly, the reader seems to rely on decode_internal method on the apache-avro crate and seems to implement some of the Avro decoding "by hand". We ended up rolling our reader to support and we're able to use decode_from_avro datum and entirely pass on the avro decoding responsibility to the avro package.

Would love to work with @tustvold who seems to contributed here the most to augment the existing limitations here.

Describe the solution you'd like

Addition of support for parsing complex datatypes.

Describe alternatives you've considered

Convert avro > json then rely on json_to_arrow conversion, but this leads to inevitable loss of type information.

Additional context

No response

[alamb]

Started collecting this and other avro related items in #14096