Manifest file for all graphs and subgraphs

[caufieldjh]

In its role as an index of KGs, it would be useful for KG-HUB to provide a list (e.g., a manifest file) of all graphs and their component subgraphs (in most cases besides KG-OBO, the source transforms) and then have this list be publicly viewable. It should include metadata such as graph descriptions. These could be pulled from the download.yaml for each project in theory.

See also the draft linkml dataset distribution schema:
https://github.com/linkml/linkml-model/blob/main/linkml_model/model/schema/datasets.yaml

[caufieldjh]

As per KG-OBO would also like to keep track of broken links - this will mean the manifest should not be written anew with each update, at least not without reading it first to store all obsolete IDs.

[caufieldjh]

As per Slack discussion: also keep track of KGX sources, CURIE namespaces, and Biolink types.
This would support user knowledge of how KGs overlap.
It would also serve as error-checking to see if Biolink types match expectations - types are listed in KGX stats output so we can read those here

(Split into issue #8)

[caufieldjh]

Other misc TODOs:

• assign description and was_derived_from to objects - This may need to be extracted on a per-project basis
• get version for projects other than KG-OBO
• consider using other LinkML class (DataResource) for uncompressed files

[caufieldjh]

Will also need to set up Jenkinsfile to run this weekly or so (I'll make that its own PR so it can have its own Jenkins test branch)

(And its own issue - see #7)

[caufieldjh]

There is a degree of structural validation the manifest step can accomplish, with items like the following:

• Verify that projects follow the expected file structure (dated builds, raw and transforms in their own dirs, stats in their own dir)
• Graph tar.gz files contain only node and edge list
• Files are, in fact, tsvs in KGX format

[caufieldjh]

Need to set click arg to write to bucket or not, in case we just want a local version
As this will be called on a Jenkinsfile anyway, we can handle the writing there.

[caufieldjh]

Each DataPackage object (here, compressed files) should include a list of its components (here, node and edge lists).
The validation step should collect the names of these files to ensure they are merged-kg_nodes.tsv and merged-kg_edges.tsv, then pass that to the object creator to include their names.