GH-43902: [Java] Support for Long memory addresses

[vibhatha]

Rationale for this change

The usage of Integer instead of Long must be encouraged with the usage of memory sizing, indexing and addresses.

What changes are included in this PR?

This PR refactors the usage of Integer into Long along with utilities refactors.

Are these changes tested?

Existing test cases.

Are there any user-facing changes?

Yes, certain API calls may subject changes.

• GitHub Issue: [Java] Support for Long memory addresses #43902
• GitHub Issue: [Java] Support for Long memory addresses #43902

[lidavidm]

I think this is the right idea but we should be adding the long version as an overload in most cases (and if it's in return position we need to add a separate method), then we need to deprecate the int variants

[vibhatha]

I think this is the right idea but we should be adding the long version as an overload in most cases (and if it's in return position we need to add a separate method), then we need to deprecate the int variants

Right. I will update. Thank you!

[vibhatha]

@lidavidm I updated. Sorry I have missed a few things earlier.

[lidavidm]

Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  The following options were not recognized by any processor: '[skipDefs, atfDoNotCache]'
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[995,19] [removal] reserve(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[894,19] [removal] add(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[64,46] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[77,16] Unsafe is internal proprietary API and may be removed in a future release

Can we ensure we fix the warnings? (Also can we enable warnings-as-errors?)

[vibhatha]

Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  The following options were not recognized by any processor: '[skipDefs, atfDoNotCache]'
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[995,19] [removal] reserve(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[894,19] [removal] add(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[64,46] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[77,16] Unsafe is internal proprietary API and may be removed in a future release

Can we ensure we fix the warnings? (Also can we enable warnings-as-errors?)

Right, we discussed this earlier as well. We can create a ticket to follow up with this. But in relation to this context, I think we need to sort of organize the warning -> error migration as Unsafe components probably need some waiting. But for the rest and for each module I can enable the warnings to errors iteratively.

I created a parent issue apache/arrow-java#59, and I will update each module upon feasibility and create new issues for cases like Unsafe to be fixed when we are supporting minimal JDKs in future.

[lidavidm]

At least here, let's fix any warnings about deprecation of things we deprecated in this PR

[vibhatha]

@lidavidm fixed the warnings and locally they don't appear, but let's check via the CI logs as well.

[laurentgo]

Sorry for not having commented earlier but I wonder how much the changes do increase the memory footprint overall. When many buffers are created, this has a non negligeable impact abd I wonder if we should not adapt the fields based on the actual need for more than 2gb (4gb if we consider it unsigned) allocations

[lidavidm]

Hmm, is there a use case where you'd expect many tens of thousands of buffers allocated but memory is constrained (measured in megabytes, I suppose)?

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit b2e0668.

There were 2 benchmark results with an error:

• Commit Run on amd64-m5-4xlarge-linux at 2024-09-04 05:56:25Z
  • dataframe-to-table (R) with dataset=type_nested, language=R
  • dataframe-to-table (R) with dataset=type_nested, language=R

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[laurentgo]

Hmm, is there a use case where you'd expect many tens of thousands of buffers allocated but memory is constrained (measured in megabytes, I suppose)?

The query engine I work with runs multiple fragments concurrently and we are regularly seing hundreds of thousands of buffers allocated and being live. We try to keep the heap size at a minimum size (4g/8g) and use as much direct memory as possible and so arrow bufs footprint (along with related objects) have a large impact for us

[lidavidm]

But even for 1 million buffers, 4 bytes of overhead per buffer is 4 megabytes (or we can say 8 bytes of overhead: 8 megabytes).

I suppose we could continue storing int but require all interfaces to use long...