Add singularity inspect metalad extractor #200

asmacdo · 2023-03-07T22:06:00Z

Fixes: #198

This PR is the result of Datalad Metalad hackathon last week!

TODOs

add some rudimentary unittest
make CI happy

codeclimate · 2023-03-07T22:06:25Z

Code Climate has analyzed commit 2340c1c and detected 4 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Security	4

View more on Code Climate.

changelog.d/20230307_164111_austin_add_metalad_extractor.md

yarikoptic

Also will need a rudimentary unittest on extracting metadata from a sample container. There is already some which use singularity etc, so should be similar I guess. Test should skip if no metadata or singularity available. But metalad should be installed in CI to actually do run that test.

datalad_container/extractors/metalad_container.py

setup.cfg

requirements-devel.txt

yarikoptic · 2023-03-08T16:02:34Z

re appveyor -- I did that fix in #190 already which I merged today. Just rebase your branch on current master and it should be gone.

setup.cfg

datalad_container/extractors/metalad_container.py

yarikoptic · 2023-03-10T18:40:57Z

re test

let's use image from tests/test_run.py:testimg_url = 'shub://datalad/datalad-container:testhelper'
create a session scope fixture. More advanced example is in dandi-cli https://github.com/dandi/dandi-cli/blob/HEAD/dandi/tests/fixtures.py#L342 but in your case it would be simpler -- it would singularity pull that image into temporary location, and give that path as what it yields as a "fixture" to underlying test. Then multiple tests could reuse that download.
for running external commands use DataLad's WitlessRunner , smth like out = WitlessRunner().run(cmd, protocol=StdOutCapture) but see other uses in the tests of datalad.

yarikoptic · 2023-03-10T18:58:12Z

re external_versions -- just "patch" it to provide function to load version for singularity and apptainer. Here is how

In [8]: external_versions.CUSTOM['cmd:singularity'] = lambda : '0.1.1'

In [9]: external_versions['cmd:singularity']
Out[9]: LooseVersion ('0.1.1')

so replace lambda with yours. For that probably just do in from datalad.support.external_versions import external_versions and do that patching. This would make external_versions["cmd:singularity"] available throughout for all uses of external_versions in that process, including I think reporting in datalad wtf output as long as datalad_container is imported (via datalad.api or elsehow)

asmacdo · 2023-03-15T17:51:48Z

skipif test if datalad metalad not installed
install metalad for travis

datalad_container/extractors/metalad_container.py

codecov · 2023-03-15T20:38:57Z

Codecov Report

Patch coverage: 93.26% and project coverage change: +0.01 🎉

Comparison is base (c3bb747) 93.82% compared to head (3771e2b) 93.84%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #200      +/-   ##
==========================================
+ Coverage   93.82%   93.84%   +0.01%     
==========================================
  Files          19       25       +6     
  Lines         956     1072     +116     
==========================================
+ Hits          897     1006     +109     
- Misses         59       66       +7

Impacted Files	Coverage Δ
datalad_container/utils.py	`85.71% <85.71%> (ø)`
...container/extractors/_load_singularity_versions.py	`90.00% <90.00%> (ø)`
...ntainer/extractors/tests/test_metalad_container.py	`91.66% <91.66%> (ø)`
...alad_container/tests/fixtures/singularity_image.py	`94.73% <94.73%> (ø)`
datalad_container/extractors/metalad_container.py	`96.55% <96.55%> (ø)`
datalad_container/__init__.py	`100.00% <100.00%> (ø)`
datalad_container/conftest.py	`100.00% <100.00%> (ø)`
datalad_container/tests/fixtures/__init__.py	`100.00% <100.00%> (ø)`

... and 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

.appveyor.yml

datalad_container/extractors/_load_singularity_versions.py

datalad_container/extractors/metalad_container.py

datalad_container/extractors/tests/test_metalad_container.py

datalad_container/tests/test_add.py

datalad_container/tests/fixtures/singularity_image.py

datalad_container/extractors/tests/test_metalad_container.py

datalad_container/extractors/_load_singularity_versions.py

Fixes: datalad#198 - Adds "singularity inspect path/to/file.sing" to metadata - Adds "apptainer --version || singularity version" to metadata find_executable will be removed in 3.12 Code that imports distutils will no longer work from Python 3.12. Necessary for me to run locally. Update file docstring to not lie Handle both singularity and apptainer

yarikoptic · 2023-03-29T16:46:44Z

CI is read on docs and travis ... from a single travis run

=========================== short test summary info ============================
FAILED datalad_container/extractors/tests/test_metalad_container.py::test__container_inspect_nofile - FileNotFoundError: [Errno 2] No such file or directory: 'apptainer': 'apptainer'
FAILED datalad_container/extractors/tests/test_metalad_container.py::test__container_inspect_valid - FileNotFoundError: [Errno 2] No such file or directory: 'apptainer': 'apptainer'

yarikoptic

The main point is to make tests "execute" (not just skip) and pass on CI

changelog.d/20230307_164111_austin_add_metalad_extractor.md

datalad_container/extractors/tests/test_metalad_container.py

yarikoptic · 2023-03-30T13:37:48Z

datalad_container/tests/fixtures/singularity_image.py

+
+@pytest.fixture(scope="session")
+def container_command():
+    """Not a very useful function other than to add session scope."""


could have just made that function "lazy" via using e.g. @functools.cache but ok for now

yarikoptic reviewed Mar 8, 2023

View reviewed changes

changelog.d/20230307_164111_austin_add_metalad_extractor.md Outdated Show resolved Hide resolved

asmacdo commented Mar 8, 2023

View reviewed changes

changelog.d/20230307_164111_austin_add_metalad_extractor.md Outdated Show resolved Hide resolved

yarikoptic requested changes Mar 8, 2023

View reviewed changes

yarikoptic reviewed Mar 8, 2023

View reviewed changes

requirements-devel.txt Outdated Show resolved Hide resolved

yarikoptic mentioned this pull request Mar 9, 2023

Introduce {cont_dspath} to contain what {img_dspath} was and fix {img_dspath} #201

Draft

1 task

asmacdo force-pushed the add-metalad-extractor branch from 1d8fc67 to ee00701 Compare March 9, 2023 16:30

asmacdo commented Mar 9, 2023

View reviewed changes

setup.cfg Outdated Show resolved Hide resolved

yarikoptic reviewed Mar 10, 2023

View reviewed changes

datalad_container/extractors/metalad_container.py Outdated Show resolved Hide resolved

asmacdo force-pushed the add-metalad-extractor branch 2 times, most recently from 2fea20f to f8b61bf Compare March 15, 2023 16:29

asmacdo commented Mar 15, 2023

View reviewed changes

datalad_container/extractors/metalad_container.py Outdated Show resolved Hide resolved

yarikoptic requested changes Mar 16, 2023

View reviewed changes

yarikoptic reviewed Mar 20, 2023

View reviewed changes

datalad_container/extractors/_load_singularity_versions.py Outdated Show resolved Hide resolved

yarikoptic reviewed Mar 20, 2023

View reviewed changes

datalad_container/extractors/_load_singularity_versions.py Outdated Show resolved Hide resolved

asmacdo force-pushed the add-metalad-extractor branch from 2340c1c to 441825d Compare March 24, 2023 13:34

asmacdo added 9 commits March 24, 2023 11:31

skipif no metalad

9e6efb7

install metalad on travis

c55cf02

install metalad in appveyor

7ccdb47

squashing evil

775467c

simplify version retrieve

0fc76d7

expand my tabs please

8dfaa29

do not assume apptainer adapter

7a58419

assert exact results

bdd4f1e

add full extract test

8be0812

asmacdo force-pushed the add-metalad-extractor branch from abe8613 to 8be0812 Compare March 24, 2023 15:31

fixup: align docs to code changes

5808fb4

yarikoptic requested changes Mar 29, 2023

View reviewed changes

asmacdo added 4 commits March 29, 2023 13:30

/double-face-palm

ac51729

typo

0569e0f

fixdocs

6af523f

add generated manpage to gitignore

6e81213

asmacdo force-pushed the add-metalad-extractor branch from b2da0ee to 6e81213 Compare March 29, 2023 17:45

yarikoptic reviewed Mar 30, 2023

View reviewed changes

datalad_container/extractors/tests/test_metalad_container.py Outdated Show resolved Hide resolved

yarikoptic reviewed Mar 30, 2023

View reviewed changes

Tiny strengthening of a test

3771e2b

yarikoptic merged commit 10f5d1f into datalad:master Mar 30, 2023

yarikoptic added the minor Increment the minor version when merged label Mar 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add singularity inspect metalad extractor #200

Add singularity inspect metalad extractor #200

asmacdo commented Mar 7, 2023 •

edited by yarikoptic

Loading

codeclimate bot commented Mar 7, 2023 •

edited

Loading

yarikoptic left a comment

yarikoptic commented Mar 8, 2023

yarikoptic commented Mar 10, 2023

yarikoptic commented Mar 10, 2023

asmacdo commented Mar 15, 2023 •

edited

Loading

codecov bot commented Mar 15, 2023 •

edited

Loading

yarikoptic commented Mar 29, 2023

yarikoptic left a comment

yarikoptic Mar 30, 2023

Add singularity inspect metalad extractor #200

Add singularity inspect metalad extractor #200

Conversation

asmacdo commented Mar 7, 2023 • edited by yarikoptic Loading

codeclimate bot commented Mar 7, 2023 • edited Loading

yarikoptic left a comment

Choose a reason for hiding this comment

yarikoptic commented Mar 8, 2023

yarikoptic commented Mar 10, 2023

yarikoptic commented Mar 10, 2023

asmacdo commented Mar 15, 2023 • edited Loading

codecov bot commented Mar 15, 2023 • edited Loading

Codecov Report

yarikoptic commented Mar 29, 2023

yarikoptic left a comment

Choose a reason for hiding this comment

yarikoptic Mar 30, 2023

Choose a reason for hiding this comment

asmacdo commented Mar 7, 2023 •

edited by yarikoptic

Loading

codeclimate bot commented Mar 7, 2023 •

edited

Loading

asmacdo commented Mar 15, 2023 •

edited

Loading

codecov bot commented Mar 15, 2023 •

edited

Loading