Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add normalization filters to aggregate MDS adapter filters #122

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

craigrbarnes
Copy link
Contributor

Link to JIRA ticket if there is one:

New Features

Adds two new filters to the aggregate MDS ETL adapters.

normalize_value
Normalized a value by mapping from one value to another. This uses a mapping object passed as a filter parameter.
For example:

 "subject_gender": {
          "path": "gender",
          "filters": [
            "normalize_value"
          ],
          "filterParams": {
            "normalize_value": {
              "M": "Male",
              "F": "Female"
            }
          },
          "default": ""
        },

Will map values of 'M" to 'Male' and 'F' to 'Female. This can be used to normalize data values from different data sources.

normalize_tags

Normalizes tag values by defining a mapping of category names to new name values.
Example:

"tags": {
  "path": "tags",
  "filters": [
    "normalize_tags"
  ],
  "filterParams": {
    "normalize_tags": {
      "gender": {
        "M": "Male",
        "F": "Female"
      }
    }
  }
}, 

Will normalize the category 'gender' to Male and Female.

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

Copy link

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

@craigrbarnes craigrbarnes changed the title Add normalization filters to aggregate MDS adapter Filter Add normalization filters to aggregate MDS adapter filters Jan 10, 2025
Copy link

Please find the ci env pod logs here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant