From d1d042682ad28fbf7e6956b614acba292f1f888c Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Fri, 31 May 2024 14:37:03 -0600 Subject: [PATCH 01/12] ApiDefinitionGuidelines: Add README section for API Definitions Signed-off-by: Gabe Goodhart --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 0ecfe2b6..b03af186 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ The rules for merging depend on the type of change in question and its scope of ## Formatting Guidelines -Design documents should be placed in `docs/`. +Design documents should be placed in [docs/](./docs). API Specifications should be placed in [api-definitions/](./api-definitions). ### Text @@ -43,3 +43,7 @@ easily updated in the future as needed. Some options include: * [Mermaid](https://github.com/mermaid-js/mermaid#readme) * [Excalidraw](https://excalidraw.com/) ** Be sure to leave "Embed Scene" turned on when exporting the PNG. + +### API Specifications + +API definitions use [OpenAPI](https://www.openapis.org/) format in [yaml](https://yaml.org/). From 224e8c99da0e1aa600f70865d28a78e05694b2f1 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Fri, 31 May 2024 14:53:25 -0600 Subject: [PATCH 02/12] ApiDefinitionGuidelines: Add stub of api-definitions directory Signed-off-by: Gabe Goodhart --- api-definitions/README.md | 17 +++++++++++++++++ api-definitions/instructlab.yaml | 6 ++++++ api-definitions/platform.yaml | 6 ++++++ 3 files changed, 29 insertions(+) create mode 100644 api-definitions/README.md create mode 100644 api-definitions/instructlab.yaml create mode 100644 api-definitions/platform.yaml diff --git a/api-definitions/README.md b/api-definitions/README.md new file mode 100644 index 00000000..4c9913d0 --- /dev/null +++ b/api-definitions/README.md @@ -0,0 +1,17 @@ +# API Definitions + +The common abstract interface for the service APIs is defined by [REST APIs](https://www.redhat.com/en/topics/api/what-is-a-rest-api) using [OpenAPI](https://www.openapis.org/). This project is the central home for these API definitions. + +## Style Guidelines + +* Use `kebab-case` for path elements +* Use `snake_case` for properties +* Use `UpperCamelCase` for internal reusable schema names + +## API Layout + +* There are two main portions of the APIs: + * [instructlab.yaml](./instructlab.yaml): This defines the user-facing `InstructLab` REST API + * [platform.yaml](./platform.yaml): This defines the platform-level APIs used by the `InstructLab` workflow. +* Each platform `Capability` should own its own fully-functional sub-API file that can be used by individual capability service implementations +* Any schema object that is reused between endpoints should be housed in a schema file under [common](./common) diff --git a/api-definitions/instructlab.yaml b/api-definitions/instructlab.yaml new file mode 100644 index 00000000..096ea0e5 --- /dev/null +++ b/api-definitions/instructlab.yaml @@ -0,0 +1,6 @@ +openapi: '3.0.2' +info: + title: InstructLab + version: '0.1.0' +# TODO +paths: {} diff --git a/api-definitions/platform.yaml b/api-definitions/platform.yaml new file mode 100644 index 00000000..4f69e7f0 --- /dev/null +++ b/api-definitions/platform.yaml @@ -0,0 +1,6 @@ +openapi: '3.0.2' +info: + title: AI Platform (We need a name for the IL-agnostic platform!) + version: '0.1.0' +# TODO +paths: {} From 70a15469afaab5830b9c8c2d8431aed8ca6a9166 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Fri, 31 May 2024 14:54:38 -0600 Subject: [PATCH 03/12] ApiDefinitionGuidelines: Add api-definitions-guidelines doc Signed-off-by: Gabe Goodhart --- docs/api-definitions-guidelines.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 docs/api-definitions-guidelines.md diff --git a/docs/api-definitions-guidelines.md b/docs/api-definitions-guidelines.md new file mode 100644 index 00000000..af8ecf7c --- /dev/null +++ b/docs/api-definitions-guidelines.md @@ -0,0 +1,20 @@ +# API Definitions Guidelines + +This document describes how service APIs will be managed for `InstructLab` and the sub-components of the `InstructLab` backend. + +## What parts of InstructLab need service APIs? + +There are two primary classes of service APIs needed to support InstructLab: + +* `Platform APIs`: These are APIs that are ignorant of `InstructLab` and provide generic AI platform capabilities (e.g., Fine Tuning, SDG, Eval) +* `InstructLab APIs`: These are the APIs that reflect the user-facing functionality of `InstructLab` itself. They are aware of the end-to-end `InstructLab` workflow. + +The `InstructLab APIs` are essential for hosting `InstructLab` as a service in a repeatable way. The `Platform APIs` are critical for component reuse and extensibility (e.g., new SDG algorithms for new taxonomy data types), but they are not strictly required for hosting `InstructLab` as a service. + +## How will service APIs be defined? + +Service APIs will be defined using [OpenAPI](https://www.openapis.org/) format in [yaml](https://yaml.org/). For structural and style guidelines, see [api-definitions](../api-definitions/README.md). + +## Where will service API definitions live? + +Service API definitions will live in [api-definitions](../api-definitions) within this repo. \ No newline at end of file From 01561355b34d44049bf40f04b24e5945289815c1 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Fri, 31 May 2024 15:12:22 -0600 Subject: [PATCH 04/12] ApiDefinitionGuidelines: Fix spellcheck errors API and 'extensibilitiy' should be in the dictionary, but yaml -> YAML is reasonable. Signed-off-by: Gabe Goodhart --- .spellcheck-en-custom.txt | 4 ++++ README.md | 2 +- docs/api-definitions-guidelines.md | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.spellcheck-en-custom.txt b/.spellcheck-en-custom.txt index 22db77fb..bb9ae6f0 100644 --- a/.spellcheck-en-custom.txt +++ b/.spellcheck-en-custom.txt @@ -3,6 +3,8 @@ Abhishek Akash AMDGPU +API +api arge arXiv backend @@ -37,6 +39,7 @@ Eval Excalidraw exfiltrate exfiltrating +extensibility Finetuning formedness GFX @@ -135,3 +138,4 @@ XT XTX Xu YAML +yaml \ No newline at end of file diff --git a/README.md b/README.md index b03af186..f998a2a1 100644 --- a/README.md +++ b/README.md @@ -46,4 +46,4 @@ easily updated in the future as needed. Some options include: ### API Specifications -API definitions use [OpenAPI](https://www.openapis.org/) format in [yaml](https://yaml.org/). +API definitions use [OpenAPI](https://www.openapis.org/) format in [YAML](https://yaml.org/). diff --git a/docs/api-definitions-guidelines.md b/docs/api-definitions-guidelines.md index af8ecf7c..0dae2825 100644 --- a/docs/api-definitions-guidelines.md +++ b/docs/api-definitions-guidelines.md @@ -13,7 +13,7 @@ The `InstructLab APIs` are essential for hosting `InstructLab` as a service in a ## How will service APIs be defined? -Service APIs will be defined using [OpenAPI](https://www.openapis.org/) format in [yaml](https://yaml.org/). For structural and style guidelines, see [api-definitions](../api-definitions/README.md). +Service APIs will be defined using [OpenAPI](https://www.openapis.org/) format in [YAML](https://yaml.org/). For structural and style guidelines, see [api-definitions](../api-definitions/README.md). ## Where will service API definitions live? From 83b627c9bfba8a9bb0b47d04aa1f746390957a68 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Tue, 4 Jun 2024 14:04:49 -0600 Subject: [PATCH 05/12] ApiDefinitionGuidelines: Add section on API versioning Signed-off-by: Gabe Goodhart --- .spellcheck-en-custom.txt | 1 + api-definitions/README.md | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/.spellcheck-en-custom.txt b/.spellcheck-en-custom.txt index bb9ae6f0..a6e9fae2 100644 --- a/.spellcheck-en-custom.txt +++ b/.spellcheck-en-custom.txt @@ -4,6 +4,7 @@ Abhishek Akash AMDGPU API +API's api arge arXiv diff --git a/api-definitions/README.md b/api-definitions/README.md index 4c9913d0..aa9eb5f6 100644 --- a/api-definitions/README.md +++ b/api-definitions/README.md @@ -15,3 +15,9 @@ The common abstract interface for the service APIs is defined by [REST APIs](htt * [platform.yaml](./platform.yaml): This defines the platform-level APIs used by the `InstructLab` workflow. * Each platform `Capability` should own its own fully-functional sub-API file that can be used by individual capability service implementations * Any schema object that is reused between endpoints should be housed in a schema file under [common](./common) + +## Versioning and Stability + +**WARNING** At this stage in development, we make no guarantees about stability and support for APIs! + +**FUTURE**: Once stabilized, the APIs will follow an agreed-upon form of [semantic versioning](https://semver.org/) so that users can rely on the API's stability. The decision of how to version the API and at what granularity to do so is still under discussion. \ No newline at end of file From e1d394dc7b07f17aac697275447c14a42aef7468 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Thu, 6 Jun 2024 14:11:33 -0600 Subject: [PATCH 06/12] ApiDefinitionGuidelines: Move API definitions under docs/backend Signed-off-by: Gabe Goodhart --- docs/{ => backend}/api-definitions-guidelines.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename docs/{ => backend}/api-definitions-guidelines.md (86%) diff --git a/docs/api-definitions-guidelines.md b/docs/backend/api-definitions-guidelines.md similarity index 86% rename from docs/api-definitions-guidelines.md rename to docs/backend/api-definitions-guidelines.md index 0dae2825..b78c1685 100644 --- a/docs/api-definitions-guidelines.md +++ b/docs/backend/api-definitions-guidelines.md @@ -13,8 +13,8 @@ The `InstructLab APIs` are essential for hosting `InstructLab` as a service in a ## How will service APIs be defined? -Service APIs will be defined using [OpenAPI](https://www.openapis.org/) format in [YAML](https://yaml.org/). For structural and style guidelines, see [api-definitions](../api-definitions/README.md). +Service APIs will be defined using [OpenAPI](https://www.openapis.org/) format in [YAML](https://yaml.org/). For structural and style guidelines, see [api-definitions](../../api-definitions/README.md). ## Where will service API definitions live? -Service API definitions will live in [api-definitions](../api-definitions) within this repo. \ No newline at end of file +Service API definitions will live in [api-definitions](../../api-definitions) within this repo. \ No newline at end of file From 8a1f9b9233f6c0e28e8938260f184bdf63302528 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Fri, 7 Jun 2024 07:24:23 -0600 Subject: [PATCH 07/12] ApiDefinitionGuidelines: Expand on the casing guidelines Signed-off-by: Gabe Goodhart --- .spellcheck-en-custom.txt | 2 ++ api-definitions/README.md | 7 +++++++ 2 files changed, 9 insertions(+) diff --git a/.spellcheck-en-custom.txt b/.spellcheck-en-custom.txt index a6e9fae2..fa143d40 100644 --- a/.spellcheck-en-custom.txt +++ b/.spellcheck-en-custom.txt @@ -8,6 +8,7 @@ API's api arge arXiv +ascii backend backends benchmarking @@ -127,6 +128,7 @@ triager's triagers unquantized USM +utf UX venv watsonx diff --git a/api-definitions/README.md b/api-definitions/README.md index aa9eb5f6..9b857ecc 100644 --- a/api-definitions/README.md +++ b/api-definitions/README.md @@ -5,8 +5,15 @@ The common abstract interface for the service APIs is defined by [REST APIs](htt ## Style Guidelines * Use `kebab-case` for path elements + * All characters must be in the [ascii](https://www.ascii-code.com/) character set to avoid percent encoding in URIs + * All letters must be lowercase + * Words are separated by the `-` (dash) character * Use `snake_case` for properties + * All characters must be in the [utf-8](https://www.w3schools.com/charsets/ref_html_utf8.asp) character set for simple `json` encoding + * Words are separated by the `_` (underscore) character * Use `UpperCamelCase` for internal reusable schema names + * These are internal names, so the character set is not limited + * Words are capitalized and concatenated with no separator ## API Layout From 7109cb742433fbdc69acd26eb1f849fce617e40b Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Mon, 10 Jun 2024 12:22:11 -0600 Subject: [PATCH 08/12] ApiDefinitionGuidelines: Adjust proposal to place API specs in a new repo Signed-off-by: Gabe Goodhart --- .spellcheck-en-custom.txt | 2 ++ api-definitions/README.md | 30 ---------------- api-definitions/instructlab.yaml | 6 ---- api-definitions/platform.yaml | 6 ---- docs/backend/api-definitions-guidelines.md | 40 +++++++++++++++++++++- 5 files changed, 41 insertions(+), 43 deletions(-) delete mode 100644 api-definitions/README.md delete mode 100644 api-definitions/instructlab.yaml delete mode 100644 api-definitions/platform.yaml diff --git a/.spellcheck-en-custom.txt b/.spellcheck-en-custom.txt index fa143d40..a944e0a8 100644 --- a/.spellcheck-en-custom.txt +++ b/.spellcheck-en-custom.txt @@ -114,7 +114,9 @@ Shivchander Signoff Srivastava subdirectory +submodule Sudalairaj +sync'ed Taj tatsu TBD diff --git a/api-definitions/README.md b/api-definitions/README.md deleted file mode 100644 index 9b857ecc..00000000 --- a/api-definitions/README.md +++ /dev/null @@ -1,30 +0,0 @@ -# API Definitions - -The common abstract interface for the service APIs is defined by [REST APIs](https://www.redhat.com/en/topics/api/what-is-a-rest-api) using [OpenAPI](https://www.openapis.org/). This project is the central home for these API definitions. - -## Style Guidelines - -* Use `kebab-case` for path elements - * All characters must be in the [ascii](https://www.ascii-code.com/) character set to avoid percent encoding in URIs - * All letters must be lowercase - * Words are separated by the `-` (dash) character -* Use `snake_case` for properties - * All characters must be in the [utf-8](https://www.w3schools.com/charsets/ref_html_utf8.asp) character set for simple `json` encoding - * Words are separated by the `_` (underscore) character -* Use `UpperCamelCase` for internal reusable schema names - * These are internal names, so the character set is not limited - * Words are capitalized and concatenated with no separator - -## API Layout - -* There are two main portions of the APIs: - * [instructlab.yaml](./instructlab.yaml): This defines the user-facing `InstructLab` REST API - * [platform.yaml](./platform.yaml): This defines the platform-level APIs used by the `InstructLab` workflow. -* Each platform `Capability` should own its own fully-functional sub-API file that can be used by individual capability service implementations -* Any schema object that is reused between endpoints should be housed in a schema file under [common](./common) - -## Versioning and Stability - -**WARNING** At this stage in development, we make no guarantees about stability and support for APIs! - -**FUTURE**: Once stabilized, the APIs will follow an agreed-upon form of [semantic versioning](https://semver.org/) so that users can rely on the API's stability. The decision of how to version the API and at what granularity to do so is still under discussion. \ No newline at end of file diff --git a/api-definitions/instructlab.yaml b/api-definitions/instructlab.yaml deleted file mode 100644 index 096ea0e5..00000000 --- a/api-definitions/instructlab.yaml +++ /dev/null @@ -1,6 +0,0 @@ -openapi: '3.0.2' -info: - title: InstructLab - version: '0.1.0' -# TODO -paths: {} diff --git a/api-definitions/platform.yaml b/api-definitions/platform.yaml deleted file mode 100644 index 4f69e7f0..00000000 --- a/api-definitions/platform.yaml +++ /dev/null @@ -1,6 +0,0 @@ -openapi: '3.0.2' -info: - title: AI Platform (We need a name for the IL-agnostic platform!) - version: '0.1.0' -# TODO -paths: {} diff --git a/docs/backend/api-definitions-guidelines.md b/docs/backend/api-definitions-guidelines.md index b78c1685..bb783e6b 100644 --- a/docs/backend/api-definitions-guidelines.md +++ b/docs/backend/api-definitions-guidelines.md @@ -17,4 +17,42 @@ Service APIs will be defined using [OpenAPI](https://www.openapis.org/) format i ## Where will service API definitions live? -Service API definitions will live in [api-definitions](../../api-definitions) within this repo. \ No newline at end of file +Service API definitions will live a new repository github.com/instructlab/service-api-definitions. This repo will have two primary responsibilities: + +1. House the static service API definitions +2. Build and publish any language-specific generated packages for consumption by service implementation projects (see below) + +## How will service implementations reference shared APIs? + +When a project chooses to implement one or more service APIs, there are three acceptable methods for doing so, listed in order of preference: + +1. Consume a supported language-specific package. The `service-api-definitions` repo will build consumable packages with generated code for supported languages. This is the preferred method of consumption as it avoids repository references and code duplication. +2. For languages without a supported package, the `service-api-definitions` repo may be held as a [git submodule](https://www.git-scm.com/book/en/v2/Git-Tools-Submodules). +3. It is also acceptable for an implementation to copy the relevant API definitions to the local project repository. Any changes made in the central repository will need to be sync'ed by the project owners, and any new APIs added in the project will not be considered usable until they have been integrated into the central API definitions. + +## Style Guidelines + +* Use `kebab-case` for path elements + * All characters must be in the [ascii](https://www.ascii-code.com/) character set to avoid percent encoding in URIs + * All letters must be lowercase + * Words are separated by the `-` (dash) character +* Use `snake_case` for properties + * All characters must be in the [utf-8](https://www.w3schools.com/charsets/ref_html_utf8.asp) character set for simple `json` encoding + * Words are separated by the `_` (underscore) character +* Use `UpperCamelCase` for internal reusable schema names + * These are internal names, so the character set is not limited + * Words are capitalized and concatenated with no separator + +## API Layout + +* There will be two main portions of the APIs: + * `instructlab.yaml`: This defines the user-facing `InstructLab` REST API + * `platform.yaml`: This defines the platform-level APIs used by the `InstructLab` workflow. +* Each platform `Capability` should own its own fully-functional sub-API file that can be used by individual capability service implementations +* Any schema object that is reused between endpoints should be housed in a schema file under the central `common` directory. + +## Versioning and Stability + +**WARNING** At this stage in development, we make no guarantees about stability and support for APIs! + +**FUTURE**: Once stabilized, the APIs will follow an agreed-upon form of [semantic versioning](https://semver.org/) so that users can rely on the API's stability. The decision of how to version the API and at what granularity to do so is still under discussion. \ No newline at end of file From 1a34e8c6500cc64606bb9db65af0c62c8b2cb771 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Thu, 6 Jun 2024 15:41:16 -0600 Subject: [PATCH 09/12] InitialApiDefinitions: Add initial common API data structures Signed-off-by: Gabe Goodhart --- api-definitions/common/README.md | 3 ++ api-definitions/common/file-pointer.yaml | 35 ++++++++++++++++ api-definitions/common/job-status.yaml | 29 +++++++++++++ api-definitions/common/s3-connection.yaml | 51 +++++++++++++++++++++++ 4 files changed, 118 insertions(+) create mode 100644 api-definitions/common/README.md create mode 100644 api-definitions/common/file-pointer.yaml create mode 100644 api-definitions/common/job-status.yaml create mode 100644 api-definitions/common/s3-connection.yaml diff --git a/api-definitions/common/README.md b/api-definitions/common/README.md new file mode 100644 index 00000000..4a876c33 --- /dev/null +++ b/api-definitions/common/README.md @@ -0,0 +1,3 @@ +# Common + +This section of the API definitions holds common structures that are shared across multiple service definitions. \ No newline at end of file diff --git a/api-definitions/common/file-pointer.yaml b/api-definitions/common/file-pointer.yaml new file mode 100644 index 00000000..7540a710 --- /dev/null +++ b/api-definitions/common/file-pointer.yaml @@ -0,0 +1,35 @@ +################################################################################ +# This schema defines the common ways to reference files and directories held in +# the various supported storage media. +################################################################################ + +schemas: + LocalPath: + required: ['path'] + properties: + path: + type: string + description: The name of the file on disk + + S3Path: + allOf: + - $ref: './s3-connection.yaml#/schemas/Bucket' + - type: object + description: 'Pointer to a path within an s3 bucket' + required: ['path'] + properties: + path: + type: string + description: The path within the bucket + + FilePointer: + description: Pointer to an individual file + oneOf: + - $ref: '#/schemas/LocalPath' + - $ref: '#/schemas/S3Path' + + DirectoryPointer: + description: Pointer to a directory + oneOf: + - $ref: '#/schemas/LocalPath' + - $ref: '#/schemas/S3Path' diff --git a/api-definitions/common/job-status.yaml b/api-definitions/common/job-status.yaml new file mode 100644 index 00000000..19cf8124 --- /dev/null +++ b/api-definitions/common/job-status.yaml @@ -0,0 +1,29 @@ +################################################################################ +# This schema defines the common elements of a Job Status response. Individual +# job types may extend this model with task-specific properties, but these +# common properties must be present in the response to all job status queries. +################################################################################ + +schemas: + JobStatus: + type: object + description: The status of a job in the system + properties: + job_id: + type: string + description: Unique identifier for a single job + status: + type: string + enum: + - QUEUED + - RUNNING + - COMPLETED + - CANCELED + - ERRORED + description: > + Status of the job in the system: + * QUEUED: The job has not started and is waiting to be scheduled + * RUNNING: The job is actively running as expected + * COMPLETED: The job has completed successfully + * CANCELED: The job was canceled by user action + * ERRORED: The job terminated in an error state and is not running diff --git a/api-definitions/common/s3-connection.yaml b/api-definitions/common/s3-connection.yaml new file mode 100644 index 00000000..602e5e1f --- /dev/null +++ b/api-definitions/common/s3-connection.yaml @@ -0,0 +1,51 @@ +################################################################################ +# This schema defines the common components used to reference content held in a +# cloud object store using an S3 interface. +################################################################################ + +schemas: + HMACCredentials: + type: object + properties: + access_key_id: + type: string + description: The public Access Key ID + secret_key: + type: string + description: The private Secret Key + + IAMCredentials: + type: object + properties: + # TODO: What else goes here? + apikey: + type: string + description: The IAM apikey + + + Service: + type: object + description: Pointer to an s3 service + required: ['endpoint', 'credentials'] + properties: + endpoint: + type: string + description: The qualified endpoint of the s3 service (http://, https://) + credentials: + oneOf: + - $ref: '#/schemas/HMACCredentials' + - $ref: '#/schemas/IAMCredentials' + region: + type: string + description: The region qualifier for this service + + Bucket: + allOf: + - $ref: '#/schemas/Service' + - type: object + description: Pointer to an s3 bucket + required: ['bucket'] + properties: + bucket: + type: string + description: The name of the bucket From 05073011144f7fa54f0d961670132b49c3140cde Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Thu, 6 Jun 2024 15:41:33 -0600 Subject: [PATCH 10/12] InitialApiDefinitions: Add initial SDG API to platform Signed-off-by: Gabe Goodhart --- api-definitions/platform.yaml | 21 ++++ .../platform/synthetic-data-generations.yaml | 104 ++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 api-definitions/platform.yaml create mode 100644 api-definitions/platform/synthetic-data-generations.yaml diff --git a/api-definitions/platform.yaml b/api-definitions/platform.yaml new file mode 100644 index 00000000..4d866521 --- /dev/null +++ b/api-definitions/platform.yaml @@ -0,0 +1,21 @@ +openapi: '3.0.2' +info: + title: InstructLab Backend Platform + version: '0.1.0' + +paths: + ## Inference ################################################################# + + ## Customization ############################################################# + + ## Data Jobs ################################################################# + + ######### + ## SDG ## + ######### + /synthetic-data-generations: + $ref: './platform/synthetic-data-generations.yaml#/paths/~1synthetic-data-generations' + /synthetic-data-generations/{job_id}: + $ref: './platform/synthetic-data-generations.yaml#/paths/~1synthetic-data-generations~1{job_id}' + /synthetic-data-generations/tasks: + $ref: './platform/synthetic-data-generations.yaml#/paths/~1synthetic-data-generations~1tasks' diff --git a/api-definitions/platform/synthetic-data-generations.yaml b/api-definitions/platform/synthetic-data-generations.yaml new file mode 100644 index 00000000..81464b26 --- /dev/null +++ b/api-definitions/platform/synthetic-data-generations.yaml @@ -0,0 +1,104 @@ +openapi: '3.0.2' +info: + title: Synthetic Data Generation + version: '0.1.0' + +paths: + /synthetic-data-generations: + post: + summary: Initialize a Synthetic Data Generation job + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/SDGJobBody' + responses: + "200": + description: A successfully submitted job + content: + application/json: + schema: + $ref: '../common/job-status.yaml#/schemas/JobStatus' + /synthetic-data-generations/{job_id}: + get: + summary: Retrieve the status of a Synthetic Data Generation job + responses: + "200": + description: The status for the job + content: + application/json: + schema: + $ref: '../common/job-status.yaml#/schemas/JobStatus' + + delete: + summary: Cancel a running Synthetic Data Generation job + responses: + "200": + description: The status for the job after cancellation + content: + application/json: + schema: + $ref: '../common/job-status.yaml#/schemas/JobStatus' + + /synthetic-data-generations/tasks: + get: + summary: List the currently supported SDG tasks + responses: + "200": + description: The set of currently supported SDG tasks + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/SDGTaskDefinition' + +components: + schemas: + SDGJobBody: + type: object + required: ['output_directory', 'tasks', 'seed_data'] + properties: + output_directory: + description: Location to place the output in + $ref: '../common/file-pointer.yaml#/schemas/DirectoryPointer' + + tasks: + description: Mapping from task name to task config. The config will be validated against the task's config schema. + type: object + additionalProperties: + type: object + description: Config for the given task. This will be deep-merged over the default values. + + seed_data: + description: The file or directory containing the seed data + oneOf: + - $ref: '../common/file-pointer.yaml#/schemas/FilePointer' + - $ref: '../common/file-pointer.yaml#/schemas/DirectoryPointer' + + SDGTaskDefinition: + type: object + required: ['name', 'data_json_schema', 'config_json_schema'] + properties: + name: + type: string + description: The name of the task + data_json_schema: + type: object + description: The json schema for input data files for this task + # TODO: This doesn't render cleanly for some reason, but the body here + # must be a valid JSON Schema + # $ref: 'https://json-schema.org/draft-04/schema#' + data_example: + type: string + description: Example of an input data file for this task + config_json_schema: + type: object + description: The json schema for the config of this task + # TODO: This doesn't render cleanly for some reason, but the body here + # must be a valid JSON Schema + # $ref: 'https://json-schema.org/draft-04/schema#' + config_defaults: + type: object + description: Default values for all config values From e9bd599b9d4e63e024017c34b540dd496718898f Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Mon, 10 Jun 2024 16:33:05 -0600 Subject: [PATCH 11/12] InitialApiDefinitions: Renames and status code updates after review Signed-off-by: Gabe Goodhart --- .../{file-pointer.yaml => file-path.yaml} | 18 +++++++++--------- ...ion.yaml => object-storage-connection.yaml} | 6 +++--- .../platform/synthetic-data-generations.yaml | 8 ++++---- 3 files changed, 16 insertions(+), 16 deletions(-) rename api-definitions/common/{file-pointer.yaml => file-path.yaml} (66%) rename api-definitions/common/{s3-connection.yaml => object-storage-connection.yaml} (85%) diff --git a/api-definitions/common/file-pointer.yaml b/api-definitions/common/file-path.yaml similarity index 66% rename from api-definitions/common/file-pointer.yaml rename to api-definitions/common/file-path.yaml index 7540a710..1cb5577f 100644 --- a/api-definitions/common/file-pointer.yaml +++ b/api-definitions/common/file-path.yaml @@ -11,25 +11,25 @@ schemas: type: string description: The name of the file on disk - S3Path: + ObjectStoragePath: allOf: - - $ref: './s3-connection.yaml#/schemas/Bucket' + - $ref: './object-storage-connection.yaml#/schemas/Bucket' - type: object - description: 'Pointer to a path within an s3 bucket' + description: 'Path within an object storage bucket' required: ['path'] properties: path: type: string description: The path within the bucket - FilePointer: - description: Pointer to an individual file + FilePath: + description: Path to an individual file oneOf: - $ref: '#/schemas/LocalPath' - - $ref: '#/schemas/S3Path' + - $ref: '#/schemas/ObjectStoragePath' - DirectoryPointer: - description: Pointer to a directory + DirectoryPath: + description: Path to a directory oneOf: - $ref: '#/schemas/LocalPath' - - $ref: '#/schemas/S3Path' + - $ref: '#/schemas/ObjectStoragePath' diff --git a/api-definitions/common/s3-connection.yaml b/api-definitions/common/object-storage-connection.yaml similarity index 85% rename from api-definitions/common/s3-connection.yaml rename to api-definitions/common/object-storage-connection.yaml index 602e5e1f..1584c6ae 100644 --- a/api-definitions/common/s3-connection.yaml +++ b/api-definitions/common/object-storage-connection.yaml @@ -25,12 +25,12 @@ schemas: Service: type: object - description: Pointer to an s3 service + description: Pointer to an object storage service required: ['endpoint', 'credentials'] properties: endpoint: type: string - description: The qualified endpoint of the s3 service (http://, https://) + description: The qualified endpoint of the object storage service (http://, https://) credentials: oneOf: - $ref: '#/schemas/HMACCredentials' @@ -43,7 +43,7 @@ schemas: allOf: - $ref: '#/schemas/Service' - type: object - description: Pointer to an s3 bucket + description: Pointer to an object storage bucket required: ['bucket'] properties: bucket: diff --git a/api-definitions/platform/synthetic-data-generations.yaml b/api-definitions/platform/synthetic-data-generations.yaml index 81464b26..4e735ea2 100644 --- a/api-definitions/platform/synthetic-data-generations.yaml +++ b/api-definitions/platform/synthetic-data-generations.yaml @@ -14,7 +14,7 @@ paths: schema: $ref: '#/components/schemas/SDGJobBody' responses: - "200": + "201": description: A successfully submitted job content: application/json: @@ -62,7 +62,7 @@ components: properties: output_directory: description: Location to place the output in - $ref: '../common/file-pointer.yaml#/schemas/DirectoryPointer' + $ref: '../common/file-path.yaml#/schemas/DirectoryPath' tasks: description: Mapping from task name to task config. The config will be validated against the task's config schema. @@ -74,8 +74,8 @@ components: seed_data: description: The file or directory containing the seed data oneOf: - - $ref: '../common/file-pointer.yaml#/schemas/FilePointer' - - $ref: '../common/file-pointer.yaml#/schemas/DirectoryPointer' + - $ref: '../common/file-path.yaml#/schemas/FilePath' + - $ref: '../common/file-path.yaml#/schemas/DirectoryPath' SDGTaskDefinition: type: object From 3747c8f7fea289c1f611ca82b4bbc11bfadaf205 Mon Sep 17 00:00:00 2001 From: Gabe Goodhart Date: Tue, 11 Jun 2024 11:27:07 -0600 Subject: [PATCH 12/12] InitialApiDefinitions: Remove top-level readme API guidelines Signed-off-by: Gabe Goodhart --- README.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/README.md b/README.md index f998a2a1..0ecfe2b6 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ The rules for merging depend on the type of change in question and its scope of ## Formatting Guidelines -Design documents should be placed in [docs/](./docs). API Specifications should be placed in [api-definitions/](./api-definitions). +Design documents should be placed in `docs/`. ### Text @@ -43,7 +43,3 @@ easily updated in the future as needed. Some options include: * [Mermaid](https://github.com/mermaid-js/mermaid#readme) * [Excalidraw](https://excalidraw.com/) ** Be sure to leave "Embed Scene" turned on when exporting the PNG. - -### API Specifications - -API definitions use [OpenAPI](https://www.openapis.org/) format in [YAML](https://yaml.org/).