You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When saving a catalog to disk using to_hipscat the provided output path may have content (e.g. if previously a catalog was saved to that same directory). In this case we need to specify the overwrite flag to True.
However, the current behavior does not ensure that the output folder is erased and recreated. It only ensures that the previous catalog files that coincide with those of the new catalog are overwritten (parquet leaf files, as well as metadata). The parquet leaf files that do not directly match will the ones on the catalog currently being saved remain in that directory. The catalog saved to disk in this case is not correct.
We should throw an error alerting the user for the usage of the overwrite flag if it overwrite is False and the output directory has contents. If overwrite is True, the directory should be recreated from scratch.
Before submitting
Please check the following:
I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
The text was updated successfully, but these errors were encountered:
Bug report
When saving a catalog to disk using
to_hipscat
the provided output path may have content (e.g. if previously a catalog was saved to that same directory). In this case we need to specify the overwrite flag to True.However, the current behavior does not ensure that the output folder is erased and recreated. It only ensures that the previous catalog files that coincide with those of the new catalog are overwritten (parquet leaf files, as well as metadata). The parquet leaf files that do not directly match will the ones on the catalog currently being saved remain in that directory. The catalog saved to disk in this case is not correct.
We should throw an error alerting the user for the usage of the
overwrite
flag if itoverwrite
is False and the output directory has contents. Ifoverwrite
is True, the directory should be recreated from scratch.Before submitting
Please check the following:
The text was updated successfully, but these errors were encountered: