Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic metadata validation run after import #419

Open
1 of 3 tasks
hombit opened this issue Oct 23, 2024 · 0 comments
Open
1 of 3 tasks

Add basic metadata validation run after import #419

hombit opened this issue Oct 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@hombit
Copy link
Contributor

hombit commented Oct 23, 2024

Feature request

This proposal is related to #379 and suggests performing checks to validate metadata after the import. These checks should be done by default.

We can do this analysis with row-group stats from _metadata and check:

  • _heapix_29 is within [0, 12 * 4**29) range
  • Norder, Dir, Npix values match partitions
  • dec is within [-90, 90]

Also all these columns should have no nulls

Before submitting
Please check the following:

  • I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
  • I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
  • If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.
@hombit hombit added the enhancement New feature or request label Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: No status
Development

No branches or pull requests

1 participant