Skip to content

Commit

Permalink
GH-40900: [Go] Fix Mallocator Weirdness (#40902)
Browse files Browse the repository at this point in the history
### Rationale for this change
With help from @ lidavidm and @ bkietz digging into the linked issue, we found the following:

* Using `mtrace` and `strace` didn't produce much enlightenment to what was happening.
* If the python adbc_driver_manager was built so that the cython lib is built using `CMAKE_BUILD_TYPE=Debug` then the crash/failure goes away
* If the env var `MALLOC_MMAP_THRESHOLD_` is set to 128MB, the crash/failure goes away
* It is only reproducible when calling through python, I haven't been able to reproduce it using pure Go
* Calling `calloc` again after it fails, still fails
* Calling `malloc` + `memset` immediately after the failing `calloc` works perfectly and doesn't fail anymore

### What changes are included in this PR?
Adding a comment describing the situation and falling back to `malloc` + `memset` if `calloc` returns an error. If the pointer returned from `malloc` is `nil` then we surface the error.

* GitHub Issue: #40900

Authored-by: Matt Topol <[email protected]>
Signed-off-by: Matt Topol <[email protected]>
  • Loading branch information
zeroshade authored Apr 1, 2024
1 parent 17a5368 commit 7132184
Showing 1 changed file with 10 additions and 1 deletion.
11 changes: 10 additions & 1 deletion go/arrow/memory/mallocator/mallocator.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,19 @@ func (alloc *Mallocator) Allocate(size int) []byte {
}
ptr, err := C.calloc(C.size_t(size), 1)
if err != nil {
panic(err)
// under some circumstances and allocation patterns, we can end up in a scenario
// where for some reason calloc return ENOMEM even though there is definitely memory
// available for use. So we attempt to fallback to simply doing malloc + memset in
// this case. If malloc returns a nil pointer, then we know we're out of memory
// and will surface the error.
if ptr = C.malloc(C.size_t(size)); ptr == nil {
panic(err)
}
C.memset(ptr, 0, C.size_t(size))
} else if ptr == nil {
panic("mallocator: out of memory")
}

atomic.AddUint64(&alloc.allocatedBytes, uint64(size))
return unsafe.Slice((*byte)(ptr), size)
}
Expand Down

0 comments on commit 7132184

Please sign in to comment.