Update to match neurodatascience/nipoppy:main #47
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* create bids_db_dir if it does not exist * add --cleanenv to mriqc Singularity call * add --regenerate_bids_db flag * make scripts executable * remove --bids_db_path argument * log message if BIDS database is found * revert logfile name
bcmcpher
approved these changes
Feb 29, 2024
* added logic to check outputs, remove symlinks, and clean unnecessary directories. * edits from pr request
* bind tractoflow-specific derivatives folder * fix tractogram loading with Nibabel * do not move non-symlink files (e.g. READMEs)
* Add dcm2bids conversion * dcm2bids cleanup * dcm2bids updates
* add tests for mriqc and tractoflow * add expected string * add expected string * blacify * lint * rm tractoflow * fix fmriprep tests * fix mriqc tests * start fixing tests for tractoflow * update flake8 configf * reset some linted files to master * reset some linted files to master * refix * rm test tractoflow * reset tractoflow to main * fix MRIQC test --------- Co-authored-by: Michelle Wang <[email protected]>
* move everything to legacy folder * copy previous pyproject.toml * bump version and clean up pyproject.toml * add/update fields in pyproject.toml * add minimal README * remove empty .gitmodules * add license * add linting to pre-commit * add dev setup to README * add shields for license and black * add flake8 config * add very basic parsers * add isort to pre-commit * add very basic CLI * add very basic logger * add rich formatting for CLI * ignore VS Code config files * fix isort and black conflict * add verbosity argument * add dataset_root arg and do light cleanup * add init subparser and refactor args/parsers * add DatasetLayout class * add method to create empty dataset * add base class * remove DatasetLayout.create(), to be used in workflow/runner * add name parameter to logger * add _Workflow class * remove function docstrings from test files * make _Base an abstract class * refactor test function names * add run methods to _Workflow class * add DatasetInitWorkflow for nipoppy init * fix typo in log_prefix_run_stdout * catch all workflow/command exceptions in run.py * add --dry-run flag * add common/global option group * use rich exception tracebacks * improve default __str__ method and add tests for _Base * add logging to file (but not for init) * add CLI test for init * add/update docstrings * remove redundant license file * add Pydantic config model for reading global configs * add layout validation * copy sample config file during init * move workflows * fix bad import * try implementing a Manifest class with a schema (model) * refactor manifest using abstract _Tabular class * remove useless "after" validator * implement basic doughnut class * copy sample manifest file during init * update warning message in init and remove cli return statement * add config and manifest properties to _Workflow * validate manifest visits/sessions against list * add save_json util function (not used currently) * add save() method to Config * validate manifest sessions/visits (based on config) in _Workflow * add add_record method to _Tabular * add _Tabular.add_records() for list of records * add tabular utilities for checking equality, sorting, saving to file * add preliminary work on doughnut update workflow * add working DoughnutWorkflow * fix verbosity not parsed correctly * minor changes to package structure and other things * add _Workflow.doughnut property that automatically generates the doughnut if is missing * add preliminary work on PipelineWorkflow * make minor changes to logger, logger typing and workflow init arguments * add generic pipeline utilities * rename nipoppy.workflows.workflow -> nipoppy.workflow.base * add boutiques and implement PipelineRunner * update pipeline workflows and related things (MRIQC working) * add/fix tests * add `PipelineRunner` to CLI * update pipeline-related workflows (fMRIPrep working) * clean up _Workflow.run_command() * light refactoring of package structure * move PipelineRunner and PipelineTracker into their own files * catch exceptions in run_single (don't crash) * add basic tracking (only supports SUCCES and FAIL, only for pipeline_complete) * add DicomReorgWorkflow * add BIDS conversion workflow and descriptor for HeuDiConv * fix Singularity/Boutiques config templating not being processed in runner * add DESCRIPTOR_FILE field to config and make dcm2bids steps work * move refactored code into subdirectory and move legacy code back out * change new version to 1.0.0 * refactor DatasetLayout to use config and add custom layout option to workflows * add top-level __init__.py * fix workflows failing due to partial datasets * add test checking that built-in pipelines and sample config work in simulate mode * fix BIDS database tests after updating FIDS * allow manifest to have extra columns and rename test data files * minor improvements to tabular data handling, logging, and path utilities * make DicomReorgWorkflow more modular so that it can be easily subclassed if needed * add helpers for filesystem utilities for workflows so that they respect dry_run * fix symlink bug and pass participant and session to DICOM reorg renaming function * add workflow for running tests in `nipoppy_new` * add __init__.py to tests directory * fix conftest import statement so that tests hopefully run in CI * update parser so that it renders better in docs * apply precommit * apply sourcery refactor suggestions * accept suggestions for pytest options Co-authored-by: Remi Gau <[email protected]> * specify pytest version Co-authored-by: Remi Gau <[email protected]> * update tests based on Rémi comments * rename nipoppy_new -> nipoppy_cli * rename sample_data -> examples * rename convert/converted -> bidsify/bidsfied * remove "copy" log messages in InitWorkflow * rename nipoppy.config.base -> nipoppy.config.main * make VISITS (not SESSIONS) mandatory in Config * rename Singularity things to "container" and use "apptainer" instead of "singularity" * update author list * rename data/layout -> data/layouts * revert derivatives subdirectories to <pipeline_name>/<pipeline_version> * address comments related to docstrings * add Codecov for new test suite * add tests for tracker --------- Co-authored-by: Remi Gau <[email protected]>
* switch from sphinx-autoapi to sphinx-autodoc2 extension to use markdown syntax in docstrings * add installation instructions * add Contributing guide and include README in index page * add notes about associated workflow class in CLI subcommand pages * add preliminary work on Quickstart page * add schema reference and link to it from Quickstart page * add tests/docs sections to contribution guide * minor changes * rename nipoppy_new -> nipoppy_cli * rename sample_data -> examples * update gitignore after rename * remove old files that didn't get merged correctly * add Codecov badge to README * move docs to correct directory * more fixes * revert back to sphinx-autoapi since sphinx-autodoc2 does not seem maintained anymore * add additional schemas (LayoutConfig, BoutiquesConfig) * add description string to all Pydantic model fields * update/move README * make README images appear correctly in both GitHub repo and Sphinx website * modify README text to match figure * remove empty tutorial section * mention "nipoppy_cli" in install instructions and add next steps in some pages * move images under docs/source/_static/img * reduce term references in installation instructions, and point to GitHub issues under Troubleshooting * set up ReadtheDocs * delete old README * fix ReadtheDocs config and move new README file to top * update path to README in pyproject.toml * update ReadtheDocs config to install the package so that the CLI reference works * fix typo in .readthedocs.yml and clean up .gitignore * update README after Nikhil comments * minor change
make install command more robust
* unpin doc dependencies * add details furo config * maybe if I commited my stash things would show up in the PR * Apply suggestions from code review Co-authored-by: Michelle Wang <[email protected]> * Update nipoppy_cli/docs/source/conf.py --------- Co-authored-by: Michelle Wang <[email protected]>
…poppy#218) * build schema before doc * fail on warnings * ignore warnings
…oppy#196) * moving sample_global_configs.json and tree.json to the nipoppy subdir * fixed import paths after refactor * fixed import paths after refactor * refactored and cleaned up mriqc run script * refactored and cleaned up mriqc run script * Started tracker refactoring to use doughnut * added a catalog function to identify new proc-participants from bagel and doughnut * added a catalog function to identify new proc-participants from bagel and doughnut * added custom (new subjects only) pybids layout (sqldb) generation * fixed sessions loop and incorporated utils.load_status into run_tracker.py * fixed import path errors and incorporated tracker.py for status flags * fixed global var imports and logging levels * updated sample_run_nipoppy to set log-level and prototypical mriqc run with tracker * updated bids_tracker to match proc_pipe tracker schema * minor fixes and comments * fixed Pandas future warning on setting an item of incompatible dtype * fixed another Pandas future warning on setting an item of incompatible dtype * 1) Updated mriqc and fmirpre run scripts to bind complete bids_dir path, 2) added sqldb generation with ignore list for subjects and datatype+acq, 3) updated sample_run_nipoppy.py to show the these two functionalities. * fixed fmriprep pytest * fixed codespell * fixed NM filename pattern * added functionality to custom map participant_id to bids_id * fixed minor codespell errors * fixed errors from previous merge conflict resolution * updated sample_run_nipoppy to run tractoflow, renamed and moved check_dicom_status, and fixed minor bugs and logging in run_tracker * fixed session_id typo and optimized tracker runs * fixed FS utils function * added acq option to all trackers * added acq option to all trackers (fixed merge conflict) * fixed typos and added support for acq tag in mriqc tracker * fixed tractoflow subject dir path issues and added INCOMPLETE status * refactored FS extraction using brainload package * fixed hemisphere naming * fixed aseg extraction * remove version (i.e. v) prefix from run scripts to match trackers * fixed a typo and removed legacy FS extractor script * fixed bugs related to maget-brain data prep and tracking options * minor changes to extractor scripts * minor changes to extractor scripts
use official fids
|
@nikhil153 same comment here, though I see that the README does have additional information. I can rename the README to e.g. Also not sure what that other conflict in |
|
Yes, this one has the |
* make tests run for Python 3.9 to 3.12 * set requires-python >=3.9 * fix 3.10 being interpreted as 3.1 * add "from __future__ import annotations" in files import Self from typing * use typing.Union instead or "|" (introduced in Python 3.10) * revert to using "|" instead of typing.Union * use typing_extensions to resolve version incompatibility issues * silence Sphinx warnings * change fpath_layout annotation from Optional[Path] to Optional[StrOrPathLike] * pin `sphinx-autoapi==3.0.0` to avoid errors with `3.1.0`
…low` (nipoppy#229) * add dicom-checking functionality and fix logging things for testing * split console logging to stdout/stderr instead of just stdout * add options for using custom mapping or session-first DICOM directories for reorg/doughnut * fix Sphinx warnings * add tests for manifest validation * add tests for DicomDirMap * more tests + light refactor of participant/session strict validation * add test for invalid path to DicomDirMap * use Myst substitutions and default layout object for paths in docs * slightly update CLI argument descriptions * import Self from typing_extensions * also do `from __future__ import annotations`... * add Codecov config file
* add NAME, VERSION, and VARIANT to PipelineConfig * remove VARIANT and add BidsPipelineConfig (with STEP) * apply new pipeline config schemas to Config * apply changes to workflows and workflow tests * improve test for duplicate pipelines * rename BIDS -> BIDS_PIPELINES * add GLOBALS field to config to minimize repetition * make sample global config file only contain the "latest" version of each supported pipelines * rename global_configs.json -> global_config.json (singular) * update config file insert in docs (quickstart) * rename global_configs.json -> global_config.json in documentation and test files too * increase test coverage * fix tests so that dummy Templateflow file is written in tmp_path * put CONTAINER and URI inside CONTAINER_INFO in PipelineConfig * implement pipeline steps with INVOCATION_FILE/DESCRIPTOR_FILE instead of INVOCATION/DESCRIPTOR (non-config tests failing) * make workflows use pipeline steps with invocation/descriptor files (all tests pass except test_supported_pipelines.py) * make supported pipeline tests pass * rename GLOBALS field to SUBSTITUTIONS * apply substitutions to descriptor/invocation file content * make `nipoppy init` copy sample invocation files * apply layout path replacement right after loading config file * add `--pipeline-step` argument to `nipoppy run` * make `pipeline_version` argument optional * do not propagate container configs on load (fix triple --cleanenv issue) * move container subcommand field to Boutiques descriptor instead of CONTAINER_CONFIG * make ContainerConfig propagation optionally include COMMAND propagation * move manifest and config file to dataset root directory * use PYBIDS_IGNORE_FILE instead of PYBIDS_IGNORE * move tracker config(s) to separate file too * add "CUSTOM" field to Config and do not allow extra fields * update file schemas doc page * fix path in docs * uncomment caplog assert statements now that caplog is working * add "container store" substitution by default * add [[NIPOPPY_PIPELINE_VERSION]] and [[NIPOPPY_PIPELINE_NAME]] substitutions to config * fix incorrect freesurfer output paths * add descriptor file paths to global config and move descriptors in DatasetInitWorkflow * remove flaky test_capture_warnings test * fix dcm2bids/heudiconv default value for sourcedata directory * fix DICOM_DIR_MAP_FILE validation error caused by substitution logic * fix typo * make default values explicit for container command and CUSTOM field * rename CONTAINER_INFO.PATH -> CONTAINER_INFO.FILE * turn get_boutiques_config() into a cached property boutiques_config * make sure highlighted lines in docs are correct * add second visit/session example to sample global config * add note in tabular model classes about "model" terminology * update highlighted lines in example config in docs
…hat already have expected output (nipoppy#259) * add get_participants_sessions_to_run in DicomReorgWorkflow * update get_participants_sessions_to_run in BidsConversionRunner * update get_participant_sessions_to_run for PipelineRunner and PipelineTracker * do not use | for union in type annotations * increase test coverage * make sure PipelineRunner.get_participants_sessions_to_run() still works even if pipeline_version is not explicitly given * make DicomReorgWorkflow update the doughnut in `run_setup()`
…oppy#263) * update manifest/doughnut columns (and DicomDirMap partially) * rename column `session` -> `session_id` in DicomDirMap * remove dicom_id and bids_id fields from doughnut * change sourcedata expected subdirectory names * rename check_session -> session_id_to_bids_id and participant_id_to_bids_id -> participant_id_to_bids_participant * refactor participant/session ID checking functions * rename bids_id -> bids_participant and session -> bids_session in invocations * update CLI arguments for participant/session * rename variables participant -> participant_id and session -> session_id * make test_default_config tests fail if replacement didn't work * rename column attributes in tabular classes (except Bagel) * rename VISIT/SESSION fields in global config * rename doughnut status columns * update raw dicom/imaging data directory to scratch/raw_imaging * add test for session_id_to_bids_session * address code review comments
…rkflow` and `BidsConversionWorkflow` (nipoppy#260) * write doughnut in DicomReorgWorkflow's cleanup step * write doughnut in BidsConversionWorkflow's cleanup step * address code review comment about docstrings
…nd update default invocation files (nipoppy#265) * move Nipoppy-specific default values from descriptors to invocation files * use a single descriptor file for Heudiconv * fix descriptors not having CLI entrypoints... * add default tracker config files for fMRIPrep * add default tracker config files for Freesurfer * fix failing tests after merge * add output_spaces to fMRIPrep invocations * rename dcm2bids descriptors * add `converter` field to heudiconv convert invocation (dcm2niix, default)
* fix doughnut path to be in correct directory * do not update doughnut in BidsConversionWorkflow if simulate is True * refactor PipelineConfig and PipelineStepConfig to add UPDATE_DOUGHNUT field for BIDS pipelines * use Union instead of | in Pydantic annotation * add comments for clarity * remove unused property * fix docs warnings
* add descriptors and sample invocations for BIDSCOIN * rename descriptors * expand on descriptions in Boutiques descriptors * update BIDScoin config/descriptor to not use container (difficult because of GUI components) * fix test
…ipoppy#275) fix command-line for fMRIPrep, MRIQC and HeuDiConv descriptors
… later) loop iteration (nipoppy#283) * fix pybids ignore pattern bug * update tests
* test for model field names * add `pipeline_starttime` column to bagel
…pelineWorkflow` (nipoppy#288) * test for model field names * add `pipeline_starttime` column to bagel * do not remove working directory in tracker
) * [ENH] [REFACTOR] imports to improve performance of the cli. * [FIX] type annotation. * [ENH] lazy import to improve cli performance. * [FIX] `sphinxarg.ext` release `0.5.0` breaks doc build. Issue described at sphinx-doc/sphinx-argparse#56 * [UPDATE] dependency requirement for sphinx-argparse.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR created/updated automatically by create-pull-request GitHub action. Will be merged automatically by automerge-action if there are no merge conflicts.