Can a Loader get info from the FileSystem such as the file's metadata?

[hippietrail]

When making Loader and FileSystem extensions with GhidraDev is there any way for the Loader to get more info from the FileSystem besides just the ByteProvider?

For older systems like I'm working with in RetroGhidra it's very common that the file itself has little or no structure but the disk or archive format the file comes from does have useful extra info such as file type, load address, entry point, etc.

In most modern systems I know much about, files are pretty self-contained. Sometimes just knowing the filename extension is all the extra info you need in addition to info in the file's header, which is known to exist due to a magid number or signature.

But maybe some modern formats also have useful info stored outside the raw files themselves?

In a FileSystem there is a metadata type created in mount() and queried in getFileAttributes() and getByteProvider() but it seems there are three generations of the FileSystem APIs and I'm using the newest one. The oldest one might not provide this, or it might be in a less convenient way.

But a Loader seems to only get a raw ByteProvider, which is the stream of bytes.

Maybe I'm missing a way to get the "file" associated with a ByteProvider which I could use to get the metadata from the FileSystem that might be possible, even if perhaps no built-in Loaders make use of it?

If not, which is what I suspect, how possible would it be to add a way to do this? What is missing that would need to be designed and implemented?

I understand it might not be worthwhile if it doesn't already exist, it would be difficult to implement, and no modern files require it.

1. Is it there and I've overlooked it.
2. If not do we know of modern files where it would be useful to add?
3. Maybe there's some tricky way to achieve most of or part of what I want? Can we get an FSRL from a ByteProvider? I don't think a ByteProvider has an FSRL field, but maybe there's some external map? Or maybe I can easily make one that both my FileSystem and my Loader can access?

[hippietrail]

Typing that all out helped me think about this more clearly, which led me to find this much already:

• ByteProvider has a getFSRL() method that returns an FSRL
• FSRL has a getFS() method that returns an FSRLRoot

I'm now looking if I can get from an FSRL or an FSRLRoot to a GFile...

OK I can then do

String path = fsrl.getPath();
FSRLRoot fsroot = fsrl.getFS();

FileSystemService fss = FileSystemService.getInstance();

FileSystemRef fsref = fss.getFilesystem(fsroot, null);
GFileSystem gfs = fsref.getFilesystem();

GFile file = gfs.lookup(path);

But to get the metadata for the GFile I normally use fsIndex.getMetadata(file); but fsIndex comes from AbstractFileSystem ... but this is all part of the newer way of making a FileSystem so I might have to see how the old way worked.

[ryanmkurtz]

I've had to do something like this a couple of times before. I ended up just making a public method in my GFileSystem that does the operation I needed to get done from some other place in Ghidra. The public method presumably will have access to the fsIndex.

[dev747368]

I'm now looking if I can get from an FSRL or an FSRLRoot to a GFile...

Probably the easiest is:

fsService.getRefdFile(fsrl, monitor)

RefdFile is a Closable that keeps the file system open while you deal with the GFile.

From there, you can fileRef.fsRef.getFilesystem().getFileAttributes(fileRef.file), which is what you were looking for, right?

[hippietrail]

I've been meaning to post a followup with the details of how I got it working, and to ask if it was a good solution.

I changed all my FileSystems like this

@FileSystemInfo(type = Apple2ProDosDskFileSystem.FS_TYPE, description = "Apple II ProDOS",
        factory = Apple2ProDosDskFileSystemFactory.class)
public class Apple2ProDosDskFileSystem extends AbstractFileSystem<ProDosEntry> {

    public static final String FS_TYPE = "prodos"; // ([a-z0-9]+ only)

and added a public method to retrieve the metadata for a file:

    public ProDosEntry getMetadata(GFile file) {
        return fsIndex.getMetadata(file);
    }

Then in my Loader I do this:

    @Override
    public Collection<LoadSpec> findSupportedLoadSpecs(ByteProvider provider) throws IOException {
        List<LoadSpec> loadSpecs = new ArrayList<>();

        FSRL fsrl = provider.getFSRL();

        FSRLRoot root = fsrl.getFS();

        String protocol = root.getProtocol();

        try {
            GFileSystem gfs = FileSystemService.getInstance().getFilesystem(root, null).getFilesystem();

            GFile gfile = gfs.lookup(fsrl.getPath());

            if (protocol == Apple2ProDosDskFileSystem.FS_TYPE) {
                   ProDosEntry metadata = ((Apple2ProDosDskFileSystem) gfs).getMetadata(gfile);
                   // process the raw binary file based on info in the fields of the metadata

So it seems we have some similarities and some differences. Am I doing something wrong that I should change?

[dev747368]

Your code is workable, but as pasted above would leak reference handles to the file system instance and prevent orderly closing of the file system.

If you wanted the 'correct' way, I would say something like (completely untested):

public class Apple2ProDosDskFileSystem extends AbstractFileSystem<ProDosEntry> {
  public FileAttributes getFileAttributes(GFile file, TaskMonitor monitor) {
    ProDosEntry metadata = fsIndex.getMetadata(file);
    return FileAttributes.of(
      // plus any others
      FileAttribute.create("ProDos File Metadata", metadata)
    );

class ProDosEntry {
  // will be called by the GUI to display this property in the file info dialog
  public String toString() {
    return whatever;
  }
}

and

  ProDosEntry metadata = fileRef.fsRef.getFilesystem()
       .getFileAttributes(fileRef.file)
       .get(FileAttributeType.UNKNOWN, ProDosEntry.class, null);

should work. 🤞

Alternatively, something closer to what you pasted above, more java-correct:

        try ( FileSystemRef fsRef = FileSystemService.getInstance().getFilesystem(root, null) ) {
            GFileSystem gfs = fsRef.getFileSystem();
            if ( gfs instanceof Apple2ProDosDskFileSystem appleFS ) {
              GFile gfile = appleFS.lookup(fsrl.getPath());
               ProDosEntry metadata = appleFS.getMetadata(gfile);
               // process the raw binary file based on info in the fields of the metada

[hippietrail]

Thanks for the great feedback! As a non-Java guy this is exactly what I need.

In looking at the built-in FileSystems it seemed to me there are three successive patterns and extends AbstractFileSystem<> seems to be the newest. I like the fact that it has separate concepts of metadata vs file attributes that the original pattern didn't have.

I've been putting things into FileAttribute that I consider user-facing, such as file type, permission flags, real name if characters like / or : were allowed. But the metadata can include internal things such as offsets into files that don't really belong in the 'Info' window when you right-click on a file in the filesystem browser. But of course FileAttributes is more core component available everywhere and doesn't require casting.

Let me know if my understanding/analysis is off.

---

Update

• Am I missing something or are you not using getRefdFile in the alternative version by mistake or on purpose?

• I wonder if it's worth proposing something like a new version of FileAttribute.create() that takes an isHidden argument or similar? Or if the "Info about" dialog could be changed to not include any attribute whose value lacks a toString method etc?

[dev747368]

Seems like you have a pretty good understanding.

Since your loader and fs are so tightly coupled, I don't see any downsides to inspecting the GFilesystem via instanceof and calling your getMetadata() method directly. My other suggestion about exposing the info via getFileAttributes() is almost the same, but using instanceof is probably simpler to understand and maintain.

I would recommend using FileSystemService.getRefdFile() instead of manually opening the filesystem and doing a lookup. I wasn't sure how much you wanted to change your existing code and wanted to stick with your example.

[ryanmkurtz]

Here are some samples.

• I query the FS when loading libraries during import. This is similar in that I started with just a ByteProvider.
  

  ghidra/Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/AbstractLibrarySupportLoader.java

  Lines 1170 to 1197 in a1f243e

  	/**
  	* Resolves the given library path to an existing {@link FSRL}. Some {@link Loader}s
  	* have relaxed requirements on what counts as a valid library filename match. For example,
  	* case-insensitive lookup may be allowed, and filename extensions may be optional.
  	*
  	* @param fs The {@link GFileSystem file system} to resolve in
  	* @param libraryParentPath The {@link Path} of the library's parent directory, relative to the
  	* given file system (could be null)
  	* @param libraryName The library name
  	* @return The library resolved to an existing {@link FSRL}, or null if it did not resolve
  	* @throws IOException If an IO-related problem occurred
  	*/
  	protected FSRL resolveLibraryFile(GFileSystem fs, Path libraryParentPath, String libraryName)
  	throws IOException {
  	String lpp = libraryParentPath != null
  	? FilenameUtils.separatorsToUnix(libraryParentPath.toString())
  	: null;
  	String targetPath = FSUtilities.appendPath(lpp, libraryName);
  	Comparator<String> baseNameComp = getLibraryNameComparator();
  	Comparator<String> nameComp = isOptionalLibraryFilenameExtensions() &&
  	FilenameUtils.getExtension(libraryName).isEmpty()
  	? (s1, s2) -> baseNameComp.compare(FilenameUtils.getBaseName(s1),
  	FilenameUtils.getBaseName(s2))
  	: baseNameComp;
  	GFile foundFile = fs.lookup(targetPath, nameComp);
  	return foundFile != null && !foundFile.isDirectory() ? foundFile.getFSRL() : null;

• I made a Plugin that lets you "Add to Program" on missing references for a dyld_shared_cache. Here i have a Program instead of a ByteProvider, but this is the one where I added some helper code to my the DyldCacheFileSystem to get what I need:
  

  ghidra/Ghidra/Features/FileFormats/src/main/java/ghidra/macosx/plugins/DyldCacheBuilderPlugin.java

  Lines 87 to 160 in a1f243e

  	/**
  	* Attempts to add the DYLD Cache component that resides at the given {@link ProgramLocation}'s
  	* "referred to" address
  	*
  	* @param location The {@link ProgramLocation} where the action took place
  	* @param monitor A {@link TaskMonitor}
  	*/
  	private void addMissingDyldCacheComponent(ProgramLocation location, TaskMonitor monitor) {
  	Program program = location.getProgram();
  	Address refAddress = location.getRefAddress();
  	if (refAddress == null) {
  	Msg.showInfo(this, null, name, "No referenced address selected");
  	return;
  	}
  	if (refAddress.getAddressSpace().isExternalSpace()) {
  	Msg.showInfo(this, null, name, "External locations are not currently supported");
  	return;
  	}
  	if (program.getMemory().contains(refAddress)) {
  	Msg.showInfo(this, null, name, "Referenced address already exists in memory");
  	return;
  	}
  	try (FileSystemRef fsRef = openDyldCache(program, monitor)) {
  	DyldCacheFileSystem fs = (DyldCacheFileSystem) fsRef.getFilesystem();
  	SplitDyldCache splitDyldCache = fs.getSplitDyldCache();
  	long refAddr = refAddress.getOffset();
  	String fsPath = findInDylibSegment(refAddr, splitDyldCache);
  	if (fsPath == null) {
  	fsPath = findInStubs(refAddr, splitDyldCache);
  	}
  	if (fsPath == null) {
  	fsPath = findInDyldData(refAddr, splitDyldCache);
  	}
  	if (fsPath != null) {
  	ImporterUtilities.showAddToProgramDialog(fs.getFSRL().appendPath(fsPath), program,
  	tool, monitor);
  	}
  	else {
  	Msg.showInfo(this, null, name,
  	"Address %s not found in %s".formatted(refAddress, fs.toString()));
  	}
  	}
  	catch (CancelledException e) {
  	// Do nothing
  	}
  	catch (MachException | IOException e) {
  	Msg.showError(this, null, name, e.getMessage(), e);
  	}
  	}
  	/**
  	* Attempts to open the given {@link Program}'s originating {@link DyldCacheFileSystem}
  	*
  	* @param program The {@link Program}
  	* @param monitor A {@link TaskMonitor}
  	* @return A {@link FileSystemRef file system reference} to the open {@link DyldCacheFileSystem}
  	* @throws IOException if an FSRL or IO-related error occurred
  	* @throws CancelledException if the user cancelled the operation
  	*/
  	private FileSystemRef openDyldCache(Program program, TaskMonitor monitor)
  	throws IOException, CancelledException {
  	FSRL fsrl = FSRL.fromProgram(program);
  	if (fsrl == null) {
  	throw new IOException("The program does not have an FSRL property");
  	}
  	String requiredProtocol = DyldCacheFileSystem.DYLD_CACHE_FSTYPE;
  	if (!fsrl.getFS().getProtocol().equals(requiredProtocol)) {
  	throw new IOException("The program's FSRL protocol is '%s' but '%s' is required"
  	.formatted(fsrl.getFS().getProtocol(), requiredProtocol));
  	}
  	FSRLRoot fsrlRoot = fsrl.getFS();
  	return FileSystemService.getInstance().getFilesystem(fsrlRoot, monitor);
  	}