Code

Reduce pipeline processes#298

Code

Open

christian-monch wants to merge 68 commits intomaint_0.4datalad/datalad-metalad:maint_0.4from enh-reduce-pipeline-processesdatalad/datalad-metalad:enh-reduce-pipeline-processesCopy head branch name to clipboard

+1,538-295

Commits

Commits on Apr 3, 2023

• add package tools and clean up requirements

  Show description for 5d54c40
  christian-monch

  committed
  5d54c40

  Copy full SHA for 5d54c40

• update release-pypi goal in Makefile

  christian-monch

  committed
  6568591

  Copy full SHA for 6568591

• use dataclasses for annex-status info

  Show description for 15b1168
  christian-monch

  committed
  15b1168

  Copy full SHA for 15b1168

• add --file-info parameter

  Show description for e362521
  christian-monch

  committed
  e362521

  Copy full SHA for e362521

• clean up traverser and use --file-info

  Show description for b389caf
  christian-monch

  committed
  b389caf

  Copy full SHA for b389caf

• more cleanup

  christian-monch

  committed
  2e505e4

  Copy full SHA for 2e505e4

• distinguish between GitRepo and AnnexRepo

  christian-monch

  committed
  492804b

  Copy full SHA for 492804b

• fix dataset path returned by traverser

  christian-monch

  committed
  1e6760a

  Copy full SHA for 1e6760a

• handle id-less datasets

  Show description for 6585ff3
  christian-monch

  committed
  6585ff3

  Copy full SHA for 6585ff3

• update required versions

  christian-monch

  committed
  af9b45a

  Copy full SHA for af9b45a

• fix some typos and pep-8 warnings

  christian-monch

  committed
  f07616e

  Copy full SHA for f07616e

• add probe pipeline elements

  Show description for e9e8353
  christian-monch

  committed
  e9e8353

  Copy full SHA for e9e8353

• improve probe elements

  Show description for 5aa694d
  christian-monch

  committed
  5aa694d

  Copy full SHA for 5aa694d

• add an invocation count to processor probe

  Show description for b68c1f1
  christian-monch

  committed
  b68c1f1

  Copy full SHA for b68c1f1

• backport a fix for traversal root path handling

  christian-monch

  committed
  0acee9d

  Copy full SHA for 0acee9d

• send stop pipeline-data to consumer

  christian-monch

  committed
  e40c26d

  Copy full SHA for e40c26d

• remove an unused import

  christian-monch

  committed
  6fe8e74

  Copy full SHA for 6fe8e74

• fix a bug in relative item path determination

  christian-monch

  committed
  e581244

  Copy full SHA for e581244

• add a test for pipeline extract adapter

  christian-monch

  committed
  254ba86

  Copy full SHA for 254ba86

• add nose to requirements

  christian-monch

  committed
  7f928d0

  Copy full SHA for 7f928d0

• fix parameter names

  christian-monch

  committed
  ff73673

  Copy full SHA for ff73673

• require dataset IDs, rebase on maint_0.4

  Show description for a601e85
  christian-monch

  committed
  a601e85

  Copy full SHA for a601e85

• fix dataset reporting

  Show description for d2cb104
  christian-monch

  committed
  d2cb104

  Copy full SHA for d2cb104

• update requirements-files

  christian-monch

  committed
  8dee26d

  Copy full SHA for 8dee26d

• [TEMP] fix traversal code

  christian-monch

  committed
  909bba5

  Copy full SHA for 909bba5

• copy code from datalad to prevent pulling in nose

  christian-monch

  committed
  90609b3

  Copy full SHA for 90609b3

• fix use of to_json() instead of to_dict()

  christian-monch

  committed
  46ffe7c

  Copy full SHA for 46ffe7c

• finalize rebase on maint

  christian-monch

  committed
  8a29f5b

  Copy full SHA for 8a29f5b

• implement a faster dataset traverser

  christian-monch

  committed
  da6a7c6

  Copy full SHA for da6a7c6

• add a known flag

  christian-monch

  committed
  66d9901

  Copy full SHA for 66d9901

• fix git ls-file interpretation

  Show description for 2dbc0a9
  christian-monch

  committed
  2dbc0a9

  Copy full SHA for 2dbc0a9

• cache dataset version

  Show description for 95d38fe
  christian-monch

  committed
  95d38fe

  Copy full SHA for 95d38fe

• yield a relative dataset path and simplify code

  Show description for de02bc5
  christian-monch

  committed
  de02bc5

  Copy full SHA for de02bc5

• [TEMP]

  christian-monch

  committed
  ff45963

  Copy full SHA for ff45963

• [TEMP2]

  christian-monch

  committed
  940e4a0

  Copy full SHA for 940e4a0

• use annex find to add annex info to FileInfo

  Show description for 0ff6598
  christian-monch

  committed
  0ff6598

  Copy full SHA for 0ff6598

• [temp3] add a stdin-based adding consumer

  christian-monch

  committed
  46bd69e

  Copy full SHA for 46bd69e

• [temp4] use --json-lines

  christian-monch

  committed
  55c6ad3

  Copy full SHA for 55c6ad3

• provide proper JSON lines to stdin-adder

  Show description for 0eb804f
  christian-monch

  committed
  0eb804f

  Copy full SHA for 0eb804f

• log metadata sent to stdin-adder on debug level

  christian-monch

  committed
  b2b91b6

  Copy full SHA for b2b91b6

Commits on Apr 5, 2023

• add size information for non-annex files

  Show description for 1ec78ad
  christian-monch

  committed
  1ec78ad

  Copy full SHA for 1ec78ad

Commits on Apr 17, 2023

• fix a bug where file-size was returned as string

  christian-monch

  committed
  a5f03e6

  Copy full SHA for a5f03e6

Commits on Apr 24, 2023

• [temp] consolidate listing code

  christian-monch

  committed
  20349b5

  Copy full SHA for 20349b5

Commits on Apr 26, 2023

• handle directory attributes properly

  Show description for 43409cc
  christian-monch

  committed
  43409cc

  Copy full SHA for 43409cc

• add key to status dict returned by ls_struct

  christian-monch

  committed
  f3916a9

  Copy full SHA for f3916a9

• adapt conduct test to new traverser

  christian-monch

  committed
  7e2b5ab

  Copy full SHA for 7e2b5ab

Commits on Jun 8, 2023

• add a demo of a possible dataset iterator

  christian-monch

  committed
  ee6f9c5

  Copy full SHA for ee6f9c5

Commits on Jun 9, 2023

• add a first example with parallel execution

  christian-monch

  committed
  f994320

  Copy full SHA for f994320

Commits on Jun 14, 2023

• use xargs instead of subprocess

  christian-monch

  committed
  75b8088

  Copy full SHA for 75b8088

• working parallel example

  christian-monch

  committed
  3dcfd91

  Copy full SHA for 3dcfd91

• add an example for adding from parallel

  christian-monch

  committed
  7e4b62b

  Copy full SHA for 7e4b62b

• use ls-tree options supported by older git

  christian-monch

  committed
  db341c7

  Copy full SHA for db341c7

• fix a faulty format spec in git-annex-find

  christian-monch

  committed
  dfb0b53

  Copy full SHA for dfb0b53

Commits on Jun 16, 2023

• compare ls_struct for empty or None

  christian-monch

  committed
  388cfa2

  Copy full SHA for 388cfa2

• improve git output parsing

  christian-monch

  committed
  cce26a2

  Copy full SHA for cce26a2

• improve extract adaptor

  christian-monch

  committed
  5e1246c

  Copy full SHA for 5e1246c

Commits on Jun 19, 2023

• add DatasetInfo creation from file-info parameter

  christian-monch

  committed
  820d719

  Copy full SHA for 820d719

• add individual adaptors for file and dataset

  christian-monch

  committed
  b7dc8dd

  Copy full SHA for b7dc8dd

Commits on Jun 22, 2023

• add a metadata iterator

  christian-monch

  committed
  499da79

  Copy full SHA for 499da79

Commits on Jul 4, 2023

• add a trivial traversal pipeline

  christian-monch

  committed
  018515c

  Copy full SHA for 018515c

• add location info to traverser result

  christian-monch

  committed
  b587f9b

  Copy full SHA for b587f9b

Commits on Jul 13, 2023

• add root dataset info to output

  christian-monch

  committed
  50f42bf

  Copy full SHA for 50f42bf

Commits on Jul 28, 2023

• add an adaptor to emit files for tabby-file TSVs

  christian-monch

  committed
  c2ffeeb

  Copy full SHA for c2ffeeb

Commits on Aug 15, 2023

• fix creation of AnnexedFileInfo in get_path_info

  christian-monch

  committed
  cf75265

  Copy full SHA for cf75265

• improve report of unrecognized extractor classes

  christian-monch

  committed
  a34b05a

  Copy full SHA for a34b05a

• remove unused code

  christian-monch

  committed
  48eacde

  Copy full SHA for 48eacde

• improve reporting of unrecognized extractor class

  christian-monch

  committed
  b80f94c

  Copy full SHA for b80f94c

• ignore classes from datalad-deprecated

  christian-monch

  committed
  c76dfeb

  Copy full SHA for c76dfeb