Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect overlapping data #59

Merged
merged 2 commits into from
Oct 3, 2024
Merged

Conversation

jeremyestein
Copy link
Collaborator

Fixes the rest of #50 by generating more realistic timestamps. Ie. more up to the moment, which is important because very fresh data is treated differently in collation.

Detect overlapping data, which would suggest a mismatch in actual vs expected sampling rate.

Simplify gap detection since the comparison vs rounded timestamps wasn't adding anything useful.

…e up to the current

moment so that we will wait for gaps to be filled.
between gaps and overlapping data (the latter suggesting a more serious
error)
Copy link

PR checklist

Default guide for a PR (if multiple PRs for the work, only keep one version of it and link to it on the other PRs)

  • From the UCLH data science desktop, a validation run has been set off
  • load times
    in UCL teams has been populated with the run information
  • During the run, glowroot has been checked for any queries which are taking a substantial proportion of the
    total processing time. This can be useful to identify indexes that are required.
  • After the run, look for any unexpected errors in the etl_per_message_logging table, the error_search.sql file
    on the shared drive can be used for this \\sharefs6\UCLH6\EMAP\Shared\EmapSqlScripts\devops\error_search.sql.
    Create an issue if you find an unexpected exception and is not related to the changes you've made, otherwise
    fix them!
  • After the run, populate the end time in
    load times
  • Let Aasiyah know about the completed validation and give her information on the changes and where to start
    with the validation
  • Check validation report and give any feedback to Aasiyah if there are any changes needed on her side,
    iterate on getting the validation to match at least 99% (validation and emap code).

@jeremyestein jeremyestein marked this pull request as ready for review September 25, 2024 17:00
Copy link
Contributor

@skeating skeating left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love the warp factor

Base automatically changed from jeremy/hf-data-location to sk/waveform-dev October 3, 2024 11:30
@jeremyestein jeremyestein merged commit a7f769f into sk/waveform-dev Oct 3, 2024
1 check passed
@jeremyestein jeremyestein deleted the jeremy/hf-data-gaps branch October 3, 2024 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants