Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saving catalog with overwrite should recreate the output directory #326

Closed
3 tasks done
camposandro opened this issue May 15, 2024 · 0 comments · Fixed by #327
Closed
3 tasks done

Saving catalog with overwrite should recreate the output directory #326

camposandro opened this issue May 15, 2024 · 0 comments · Fixed by #327
Assignees
Labels
bug Something isn't working

Comments

@camposandro
Copy link
Collaborator

Bug report

When saving a catalog to disk using to_hipscat the provided output path may have content (e.g. if previously a catalog was saved to that same directory). In this case we need to specify the overwrite flag to True.

However, the current behavior does not ensure that the output folder is erased and recreated. It only ensures that the previous catalog files that coincide with those of the new catalog are overwritten (parquet leaf files, as well as metadata). The parquet leaf files that do not directly match will the ones on the catalog currently being saved remain in that directory. The catalog saved to disk in this case is not correct.

We should throw an error alerting the user for the usage of the overwrite flag if it overwrite is False and the output directory has contents. If overwrite is True, the directory should be recreated from scratch.

Before submitting
Please check the following:

  • I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
  • I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
  • If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant