Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No curation ephys AND kilosort triggering #36

Closed
wants to merge 89 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
f740aef
added NWB export
Aug 18, 2021
042cc46
copied `ephys` to `ephys-no-curation`, added `recording_duration`, make
Sep 19, 2021
a39a9b1
remove `Curation`
Sep 19, 2021
590310e
make `clustering_output_dir` user-input optional, auto infer
Sep 19, 2021
adfad95
minor bug fix
Sep 19, 2021
b7c0845
improve kilosort loading routine - add `validate()` method
Sep 20, 2021
de84ce0
helper for `ProbeInsertion` - `auto_generate_entries(session_key)`
Sep 20, 2021
4397dd7
no-curation, store processed data in user-specified `processed_data_d…
Sep 21, 2021
5420ae0
using `find_full_path` for session dir as well - improve robustness
Sep 21, 2021
70e93b3
Update ephys_no_curation.py
Sep 21, 2021
a7e7554
bugfix
Sep 22, 2021
0d56456
automate ClusteringTask insertion
Sep 23, 2021
047bfa1
triggering Kilosort with ecephys_spike_sorting package
Sep 30, 2021
60105da
remove `_required_packages_paths`
Sep 30, 2021
a9326ea
minor stylistic improvements
Sep 30, 2021
149ef3f
Apply suggestions from code review
Sep 30, 2021
9f98195
typo fix
Oct 1, 2021
a24bd1a
Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…
Oct 1, 2021
ba28637
handles a weird windows/unix path incompatibility (even with pathlib)
Oct 1, 2021
5cdc994
fix missing `clustering_method`
Oct 1, 2021
6c646bb
flag to create spike sorting output dir
Oct 1, 2021
dd01fd2
Update kilosort_trigger.py
Oct 1, 2021
f699ce7
minor tweak/improvements in kilosort triggering
Oct 1, 2021
6bf0eb1
bugfix in triggering ecephys_spike_sorting
Oct 1, 2021
6788180
handle cases where `fileTimeSecs` is not available
Oct 1, 2021
d63561f
bugfix
Oct 2, 2021
375e437
missing generate module json
Oct 2, 2021
c593baf
change default `noise_template_use_rf` to False
Oct 2, 2021
69c5e51
minor bugfix
Oct 4, 2021
a6cae12
add kilosort 2.5 as default content
Oct 4, 2021
a0ea9f7
make variable naming consistent
Oct 4, 2021
acdab12
include `clustering_method` into the calculation of `param_set_hash`
Oct 4, 2021
38c5be6
updating `kilosort_repository` depending on which KSVer to be used
Oct 4, 2021
28c4452
bugfix
Oct 4, 2021
94aade7
bugfix, convert path to string
Oct 4, 2021
ae966aa
log the folder creation
Oct 5, 2021
b9f4e92
add debugging lines
Oct 5, 2021
044c389
bugfix - no dir created
Oct 5, 2021
0f0c212
Update kilosort.py
Oct 6, 2021
67341d8
new version 0.1.0b1
Oct 6, 2021
c2ee64f
improve clusteringtask and waveform ingestion routine
Oct 7, 2021
dc7ddd9
minor cleanup
Oct 7, 2021
408532c
stage tracking and resumable kilosort run
Oct 7, 2021
55bec01
minor bugfix
Oct 7, 2021
f34e95d
bugfix logging for kilosort triggering
Oct 7, 2021
19b704b
duration for each module run
Oct 7, 2021
beaf765
check `ap.bin` file validity before triggering kilosort (based on fil…
Oct 8, 2021
74e3ead
Apply suggestions from code review
Oct 11, 2021
e8ffe17
stylistic improvements, addressing code review comments
Oct 11, 2021
d07f830
handle NWB export with multiple curated clusterings from one session
Nov 2, 2021
19616ef
Update nwb.py
Nov 2, 2021
09c1e60
minor bugfix
Nov 2, 2021
3ebdf23
Merge pull request #2 from ttngu207/nwb-export
Nov 2, 2021
b25f065
add nwb export to `no-curation` ephys
Nov 2, 2021
49ca0be
first prototype for running the ecephys_pipeline with OpenEphys
Nov 9, 2021
199a2ba
bugfix for running kilosort for Open Ephys data
Nov 9, 2021
58b5984
specify additional recording-info as part of the `params`
Nov 11, 2021
ddc3b94
add `neuropixels UHD` probe type
Nov 18, 2021
df599fb
triggering kilosort analysis for open-ephys
Nov 18, 2021
819ff19
first prototype for pykilosort
Nov 19, 2021
81d99c8
use_C_waves=False for OpenEphys
Nov 19, 2021
96931a4
using fork of pyopenephys
Nov 22, 2021
b6f8f99
minor bugfix in running pykilosort
Nov 22, 2021
4250220
update openephys loader
Nov 22, 2021
af2b18b
fix package requirement formatting error
Nov 22, 2021
e325a30
handles format differences between npx1 vs 3A
Nov 22, 2021
db3027b
supporting `neuropixels UHD` in `ephys.EphysRecording`
Nov 23, 2021
f55a6a7
bugfix in electrode sites design for Neuropixels UHD probe
Nov 23, 2021
d105419
search recording channels for Open Ephys based on channel names
Nov 23, 2021
0d16e7e
bugfix openephys loader
Nov 23, 2021
c213325
bugfix - extract recording datetime (instead of using experiment date…
Nov 23, 2021
adffe34
bugfix, timedelta as seconds
Nov 23, 2021
43d6614
handles extraction of `connected` channels for NP_PROBE format in Ope…
Nov 23, 2021
da10c66
clusters extraction - check `cluster_group.tsv` and `cluster_KSLabel.…
Nov 25, 2021
02069c9
bugfix in triggering pykilosort
Nov 25, 2021
142459d
handle missing `sample_rate` from pykilosort params.py
Nov 25, 2021
1c39568
for pykilosort's probe, provide both Nchan and NchanTOT
Nov 25, 2021
2022e91
openephys loader - handles signalchain and processor as single elemen…
Nov 26, 2021
02c4b67
handles multi-probe for older OE version
Nov 26, 2021
8dff08b
NWB export fix, specifying which ephys module
Nov 29, 2021
6107e8e
version bump
Nov 29, 2021
dae36d1
fix NWB export - null `waveforms` - back to version 0.1.0b1
Nov 29, 2021
8657d58
version bump
Nov 29, 2021
aa47c8a
include probe as part of the electrode_group name for uniqueness
Nov 29, 2021
31bba8c
bugfix in assigning unit electrode indices
ttngu207 Dec 7, 2021
f9a4754
Merge pull request #3 from ttngu207/nwb-export
Dec 7, 2021
f2881ca
all three ephys files up to date
A-Baji Jan 24, 2022
c56cd18
applied requested changes
A-Baji Jan 24, 2022
9dd6b42
Merge pull request #4 from A-Baji/no-curation
Jan 24, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.


## [0.1.0b4] - 2021-11-29
### Added
+ NWB export
+ Processing with Kilosort and pyKilosort for Open Ephys and SpikeGLX


## [0.1.0b0] - 2021-05-07
### Added
+ First beta release
Expand Down
38 changes: 27 additions & 11 deletions element_array_ephys/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,39 @@
import pathlib
import uuid
import hashlib
import logging
import os


dj.config['enable_python_native_blobs'] = True


def get_logger(name):
log = logging.getLogger(name)
log.setLevel(os.getenv('LOGLEVEL', 'INFO'))
return log


def find_full_path(root_directories, relative_path):
"""
Given a relative path, search and return the full-path
from provided potential root directories (in the given order)
:param root_directories: potential root directories
:param relative_path: the relative path to find the valid root directory
:return: root_directory (pathlib.Path object)
:return: full-path (pathlib.Path object)
"""
relative_path = pathlib.Path(relative_path)
relative_path = _to_Path(relative_path)

if relative_path.exists():
return relative_path

# turn to list if only a single root directory is provided
if isinstance(root_directories, (str, pathlib.Path)):
root_directories = [root_directories]
root_directories = [_to_Path(root_directories)]

for root_dir in root_directories:
if (pathlib.Path(root_dir) / relative_path).exists():
return pathlib.Path(root_dir) / relative_path
if (_to_Path(root_dir) / relative_path).exists():
return _to_Path(root_dir) / relative_path

raise FileNotFoundError('No valid full-path found (from {})'
' for {}'.format(root_directories, relative_path))
Expand All @@ -37,27 +45,35 @@ def find_root_directory(root_directories, full_path):
Given multiple potential root directories and a full-path,
search and return one directory that is the parent of the given path
:param root_directories: potential root directories
:param full_path: the relative path to search the root directory
:return: full-path (pathlib.Path object)
:param full_path: the full path to search the root directory
:return: root_directory (pathlib.Path object)
"""
full_path = pathlib.Path(full_path)
full_path = _to_Path(full_path)

if not full_path.exists():
raise FileNotFoundError(f'{full_path} does not exist!')

# turn to list if only a single root directory is provided
if isinstance(root_directories, (str, pathlib.Path)):
root_directories = [root_directories]
root_directories = [_to_Path(root_directories)]

try:
return next(pathlib.Path(root_dir) for root_dir in root_directories
if pathlib.Path(root_dir) in set(full_path.parents))
return next(_to_Path(root_dir) for root_dir in root_directories
if _to_Path(root_dir) in set(full_path.parents))

except StopIteration:
raise FileNotFoundError('No valid root directory found (from {})'
' for {}'.format(root_directories, full_path))


def _to_Path(path):
"""
Convert the input "path" into a pathlib.Path object
Handles one odd Windows/Linux incompatibility of the "\\"
"""
return pathlib.Path(str(path).replace('\\', '/'))


def dict_to_uuid(key):
"""
Given a dictionary `key`, returns a hash string as UUID
Expand Down
Loading