Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add yaml support for hysds-io and job_specs #56

Open
DustinKLo opened this issue Jun 16, 2023 · 0 comments
Open

add yaml support for hysds-io and job_specs #56

DustinKLo opened this issue Jun 16, 2023 · 0 comments
Assignees

Comments

@DustinKLo
Copy link
Contributor

job_spec example:

{
  "command":"/home/ops/ariamh/ariaml/extractFeatures.sh",
  "disk_usage":"1GB",
  "imported_worker_files": {
    "/home/ops/.netrc": "/home/ops/.netrc",
    "/home/ops/.aws": "/home/ops/.aws",
    "/home/ops/ariamh/conf/settings.conf": "/home/ops/ariamh/conf/settings.conf"
  },
  "disable_pre_builtins": true,
  "params" : [
    {
      "name": "uri",
      "destination": "positional"
    },
    {
      "name": "uri",
      "destination": "localize"
    }
  ]
}

hysds-io example:

{
  "params" : [
    {
      "name": "localize_product_trigger",
      "from": "dataset_jpath:urls.0"
    },
    {
      "name": "scene_list",
      "from": "value",
      "value":"http://landsat-pds.s3.amazonaws.com/scene_list.gz"
    },
    {
      "name": "id",
      "from": "dataset_jpath:objectid"
    },
    {
      "name": "acq_time",
      "from": "dataset_jpath:metadata.acq_time"
    }
  ]
}

as of right now HySDS only supports json formatting for hysds-ios and job_spec. however JSON has script formatting:

  • no comments
  • no trailing commas

in the future it would be helpful to add YAML support, as YAML supports comments and such, ex:

---
command: "/home/ops/ariamh/ariaml/extractFeatures.sh"
disk_usage: 1GB
imported_worker_files:
  "/home/ops/.netrc": "/home/ops/.netrc"
  "/home/ops/.aws": "/home/ops/.aws"
  "/home/ops/ariamh/conf/settings.conf": "/home/ops/ariamh/conf/settings.conf"
disable_pre_builtins: true
params:
  - name: uri
    destination: positional
  - name: uri
    destination: localize
@DustinKLo DustinKLo self-assigned this Jun 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant