Are ELF oddities worth handling?

[thixotropist]

ELF object files can hold oddities that Ghidra flags as errors. These errors don't seem to harm importing and analyzing object files, but they do add confusion and clutter. For example:

Single byte sections

ERROR Failed to process GOT at 00100028: Insufficent memory at address 00100028 (length: 8 bytes) (ElfDefaultGotPltMarkup) ghidra.program.model.util.CodeUnitInsertionException: Insufficent memory at address 00100028 (length: 8 bytes)

This is thrown because the object file contains .plt, .got, and .got.plt sections of only one byte in length. Ghidra tries to load 8 bytes at the beginning of at least two of these segments, throwing the error. Ghidra should probably ignore such anomalously short sections instead.

Symbol name validation

Error creating symbol: .L1�12 - Symbol name contains invalid characters: .L1�12
Error creating symbol: .L0�1 - Symbol name contains invalid characters: .L0�1

Symbols like ".L1\x0212" and ".L0\x021" are not necessarily invalid in ELF, and appear to be used by an assembler or compiler to make local symbols unique. Ghidra marks these as invalid in SymbolUtilities.containsInvalidCharacters.

This behavior is seen in the current release and in 10.4.DEV. with RISCV compilers and kernel load modules. The oddities likely don't appear in RISCV user binaries or x86_64 load modules.

[ryanmkurtz]

I think Ghidra should definitely handle any runnable binary produced by legitimate compiler. If that is the case here, opening a new ticket with the sample attached would be good.

[thixotropist]

OK, I'll work up a ticket. How do you want the sample binary attachment? In general, what RISCV ELF binaries do you routinely import during integration testing?

Also, https://stackoverflow.com/questions/65879012/what-do-pcrel-hi-and-pcrel-lo-actually-do suggests that symbols like ".L1\x0212" are required to support RISCV PC relative relocations, so this might snowball into a broader symbol refactoring.

[ryanmkurtz]

Just zipping up a small sample and attaching it to the issue itself should be fine.

[thixotropist]

Thanks for the guidance - I really appreciate the responsiveness. The next discussion point leads into some murky areas:

What steps should contributors follow in proposing additions to things like ElfRelocationContext. Ghidra's external repo appears to have no integration tests involving that class, and no implicit or explicit documentation on how you detect release-blocking import regressions.

Collecting a large set of reference binaries to import in a CI test is probably doable, but how would you prove that this collection contains no anti-RE license violations?

[ryanmkurtz]

Collecting a large set of reference binaries to import in a CI test is probably doable, but how would you prove that this collection contains no anti-RE license violations?

We cannot, so we do not retain any binaries submitted by users. I use the word retain because we do sometimes accept sample binaries that users attach to help us diagnose an issue, but we effectively treat them as malware and then discard them when done.

What steps should contributors follow in proposing additions to things like ElfRelocationContext?

This code is actively maintained by @ghidra1, so the best way to propose additions is to open an issue that we can assign to @ghidra1 for triage. Part of the dialogue that occurs in that ticket might be an agreement that the user will submit a PR, but more times than not (at least in that part of the codebase) we'd prefer to just fix it internally.

[thixotropist]

Thank you for the discussion. Would @ghidra1 be willing to triage an issue to expose your CI test framework for developer submitted binaries? We wouldn't need the actual binaries you use, just broad metadata describing their type and toolchain. AbstractGhidraHeadlessIntegrationTest might be a useful component.

Another approach might collect C and C++ sources, then get the user community to provide their own toolchain to build the binaries and run them through headless analysis with C export for relocation sanity checking.

[ryanmkurtz]

Unfortunately, we don't have a testing framework in place like the one you describe. In the past we have done one-off testing with binaries produced by llvm/binutils, but honestly we have nothing to share.

[thixotropist]

Would you be receptive to a community-developed testing framework? I'm thinking of something limited in scope, perhaps sticking with RISCV ELF import tests rather than broader x86_64 or ARM applications. Your team would still have the last word on policy, so we don't add tests for situations you have no interest in ever supporting.

[ryanmkurtz]

What did you have in mind for where binaries would be stored?

[thixotropist]

One option is to collect binaries in a Docker container hosted in an external repo, something not directly tied to your Ghidra team and project.

Since there aren't a lot of RISCV-64 binaries out there, another option is to just publish scripts to pull Fedora and Ubuntu system images as needed, then break them down into sample kernel, load module, sharable library, and linked executable binaries for testing. A third option is to do that when developing the tests, then generating assembly or C source code that invokes the major relocation codes with section sizes big enough to test 32 bit immediate calculations. There are no bundled binaries at all in that scenario, just an imported (by URL reference) binutils and/or C toolchain.

In all of these cases the test case binaries would need associated metadata, showing what toolchain, memory model, and compiler options were used in the test.

The tests themselves could run in a Docker container, using an imported analyzeHeadless environment. I'm not sure there is much point in integrating them into the existing Ghidra gradle integration test framework.

The current scope is testing ELF relocation codes. As a hobby project, I'd probably include some minimal tests to see if any RISCV ISA extension code is present, e.g., vector, crypto, or coherency instructions.

[ghidra1]

Symbols like ".L1\x0212" and ".L0\x021" are not necessarily invalid in ELF, and appear to be used by an assembler or compiler to make local symbols unique. Ghidra marks these as invalid in SymbolUtilities.containsInvalidCharacters.

@thixotropist what compiler toolchain is producing these symbols? Are such symbols only produced for object modules and never within a fully linked binary?

[thixotropist]

It's probably binutils 2-40, possibly as used in the riscv-gnu-toolchain. See https://sourceware.org/binutils/docs-2.20/as/Symbol-Names.html for more details. The -L option during compilation and linking may be enough to generate and retain similar local symbols in object files, and possibly in debugging info.

I've seen these symbols in riscv-64 object files and linux kernel modules. Riscv toolchains apparently generate a lot of local symbols to support link-time or load-time relaxation and optimization, especially in support of pcrel_hi and pcrel_lo relocations. Fully linked binaries don't usually retain these symbols. Kernel load modules built with the -L option for debugging support apparently can.

An ASCII string like .L0^B1 associated with a symbol doesn't have much significance. These local symbols must be retained and processed by Ghidra if importing an object file or kernel module that has not yet been fully linked. The %pcrel_lo relocation code uses such symbols to find the matching %pcrel_hi relocation code that actually names the relocation target.

I've started [email protected]:thixotropist/ghidra_import_tests.git as a hobby-grade import test framework to collect RISCV-64 import samples - you might find some of the reference packages used there useful. This repo avoids retaining any executable or object binaries, per your policy. It fetches Fedora-Developer-37-20221130.n.0-nvme.raw.img from a public repo and breaks it down into kernel, kernel module, system lib, and user executable exemplars which are then imported into Ghidra and tested for relocation correctness.