less data preparation time #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Sorry I find there is some bug for SS, let me look into this further. |
leo19941227
commented
Sep 21, 2021
There was a problem hiding this comment.
Hi @ftshijt and @HuangZiliAndy ,
I made some updates and now two scripts for generating mixtures both work.
Please help me proof-read the change if you have time. Thanks!
Comment on lines
+79
to
+81
| --freqs 16k \ | ||
| --modes min \ | ||
| --types mix_clean |
There was a problem hiding this comment.
@HuangZiliAndy Please help me proof-read this! Thanks!
Comment on lines
+66
to
+69
| print("[Warning] - train-clean-360 is ignored in create_librimix_from_metadata.py for less data preparation time."\ | ||
| " Please note that in S3PRL we only use the train-clean-100 for downstream tasks.") | ||
| md_filename_list = [file for file in os.listdir(metadata_dir) | ||
| if 'info' not in file] | ||
| if 'info' not in file and '360' not in file] |
There was a problem hiding this comment.
This is for skipping the mixture creation for trian-clean-360 which will take much more time.
| @@ -79,5 +78,5 @@ for n_src in 2; do | |||
| --n_src $n_src \ | |||
| --freqs 16k \ | |||
| --modes max \ | |||
| --types mix_clean mix_both | |||
| --types mix_both | |||
There was a problem hiding this comment.
Can @ftshijt please help me proof-read this?
Thanks!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey @ftshijt and @HuangZiliAndy !
As we have discussed this very long time ago, that it will be nice we have a centralized LibriMix repo instead of one repo for a task, and we just use different data preparation script for each task.
Here I create on repo basing on Jiatong's version since his version has much more modification from the official release including the RTTM label files or SD. So there are two data preparation scripts:
I also make some changes to decrease the data preparation time, including ignoring train-clean-360 for both tasks and ignoring WHAM noise augmentation for SS. Furthermore, since now SD and SS all have a specific setting in terms of
min/maxcondition ormix_clean/mix_bothcondition. Hence I think the data preparation script can now just prepare the specific setting we use for benchmarking, I believe these changes can save user some time from waiting the data to be ready.Could you please take a look if my change fit your need?
Thanks!!