Consider generating types from the OpenAPI spec

[Arnavion]

I've made a tool k8s-openapi-codegen that generates Rust types from the Kubernetes OpenAPI spec. For example this is what the generated PodSpec looks like:

/// PodSpec is a description of a pod.
#[derive(Debug, Default, Deserialize, Serialize)]
pub struct PodSpec {
    /// Optional duration in seconds the pod may be active on the node relative to StartTime before the system will actively try to mark it failed and kill associated containers. Value must be a positive integer.
    #[serde(rename = "activeDeadlineSeconds")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub active_deadline_seconds: Option<i64>,

    /// If specified, the pod's scheduling constraints
    #[serde(skip_serializing_if = "Option::is_none")]
    pub affinity: Option<::io::k8s::api::core::v1::Affinity>,

    /// AutomountServiceAccountToken indicates whether a service account token should be automatically mounted.
    #[serde(rename = "automountServiceAccountToken")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub automount_service_account_token: Option<bool>,

    /// List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated.
    pub containers: Vec<::io::k8s::api::core::v1::Container>,

    /// Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. This is an alpha feature introduced in v1.9 and CustomPodDNS feature gate must be enabled to use it.
    #[serde(rename = "dnsConfig")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub dns_config: Option<::io::k8s::api::core::v1::PodDNSConfig>,

    /// Set DNS policy for the pod. Defaults to "ClusterFirst". Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to 'ClusterFirstWithHostNet'. Note that 'None' policy is an alpha feature introduced in v1.9 and CustomPodDNS feature gate must be enabled to use it.
    #[serde(rename = "dnsPolicy")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub dns_policy: Option<String>,

    /// HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts file if specified. This is only valid for non-hostNetwork pods.
    #[serde(rename = "hostAliases")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub host_aliases: Option<Vec<::io::k8s::api::core::v1::HostAlias>>,

    /// Use the host's ipc namespace. Optional: Default to false.
    #[serde(rename = "hostIPC")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub host_ipc: Option<bool>,

    /// Host networking requested for this pod. Use the host's network namespace. If this option is set, the ports that will be used must be specified. Default to false.
    #[serde(rename = "hostNetwork")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub host_network: Option<bool>,

    /// Use the host's pid namespace. Optional: Default to false.
    #[serde(rename = "hostPID")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub host_pid: Option<bool>,

    /// Specifies the hostname of the Pod If not specified, the pod's hostname will be set to a system-defined value.
    #[serde(skip_serializing_if = "Option::is_none")]
    pub hostname: Option<String>,

    /// ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod
    #[serde(rename = "imagePullSecrets")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub image_pull_secrets: Option<Vec<::io::k8s::api::core::v1::LocalObjectReference>>,

    /// List of initialization containers belonging to the pod. Init containers are executed in order prior to containers being started. If any init container fails, the pod is considered to have failed and is handled according to its restartPolicy. The name for an init container or normal container must be unique among all containers. Init containers may not have Lifecycle actions, Readiness probes, or Liveness probes. The resourceRequirements of an init container are taken into account during scheduling by finding the highest request/limit for each resource type, and then using the max of of that value or the sum of the normal containers. Limits are applied to init containers in a similar fashion. Init containers cannot currently be added or removed. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
    #[serde(rename = "initContainers")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub init_containers: Option<Vec<::io::k8s::api::core::v1::Container>>,

    /// NodeName is a request to schedule this pod onto a specific node. If it is non-empty, the scheduler simply schedules this pod onto that node, assuming that it fits resource requirements.
    #[serde(rename = "nodeName")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub node_name: Option<String>,

    /// NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
    #[serde(rename = "nodeSelector")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub node_selector: Option<::std::collections::BTreeMap<String, String>>,

    /// The priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority.
    #[serde(skip_serializing_if = "Option::is_none")]
    pub priority: Option<i32>,

    /// If specified, indicates the pod's priority. "SYSTEM" is a special keyword which indicates the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default.
    #[serde(rename = "priorityClassName")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub priority_class_name: Option<String>,

    /// Restart policy for all containers within the pod. One of Always, OnFailure, Never. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy
    #[serde(rename = "restartPolicy")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub restart_policy: Option<String>,

    /// If specified, the pod will be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.
    #[serde(rename = "schedulerName")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub scheduler_name: Option<String>,

    /// SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty.  See type description for default values of each field.
    #[serde(rename = "securityContext")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub security_context: Option<::io::k8s::api::core::v1::PodSecurityContext>,

    /// DeprecatedServiceAccount is a depreciated alias for ServiceAccountName. Deprecated: Use serviceAccountName instead.
    #[serde(rename = "serviceAccount")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub service_account: Option<String>,

    /// ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
    #[serde(rename = "serviceAccountName")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub service_account_name: Option<String>,

    /// If specified, the fully qualified Pod hostname will be "<hostname>.<subdomain>.<pod namespace>.svc.<cluster domain>". If not specified, the pod will not have a domainname at all.
    #[serde(skip_serializing_if = "Option::is_none")]
    pub subdomain: Option<String>,

    /// Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds.
    #[serde(rename = "terminationGracePeriodSeconds")]
    #[serde(skip_serializing_if = "Option::is_none")]
    pub termination_grace_period_seconds: Option<i64>,

    /// If specified, the pod's tolerations.
    #[serde(skip_serializing_if = "Option::is_none")]
    pub tolerations: Option<Vec<::io::k8s::api::core::v1::Toleration>>,

    /// List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes
    #[serde(skip_serializing_if = "Option::is_none")]
    pub volumes: Option<Vec<::io::k8s::api::core::v1::Volume>>,
}

I've tested that the generated code compiles but I haven't used it against actual apiserver responses. It might make sense to also generate clients using the paths part of the OpenAPI spec.

You can run it yourself to check out all the generated types. Are you interested in incorporating the generated types into kubeclient?

[anowell]

Hey. Apologies for the delay. I've been very slow at getting back into my OSS stuff post-holiday.

YES! I'd absolutely love to see all the resources generated via codegen. Skimming your repo, it looks like a pretty straight-forward drop-in replacement. We could incorporate the codegen into this crate, but I think I'd actually lean toward making this crate depend on and re-export your generated resource types if you intend to manage it as a crate (which alternate clients or other tools could also use). I'd happily accept a PR for this, or I might try and carve out some time in the next couple weeks to make it happen.

I haven't dug in enough to grok client generation using the paths. I'd love to better grasp what that would look like. But in general, I'm very in favor of implementing as much as possible via codegen as long as it doesn't unnecessarily hinder the external ergonomics of using the client.

[Arnavion]

My initial feeling was that the codegen would be manually fixed up a bit first instead of being used directly. You can see it in the example in the OP as well - it converts the dotted definition names into Rust modules so the generated types are very nested. For example your crate has ::resources::PodSpec in resources/pod.rs, whereas the tool's codegen has ::resources::io::k8s::api::core::v1::PodSpec in resources/io/k8s/api/core/v1/pod_spec.rs

So I initially had this process in mind:

1. You decide to add PodSpec from the codegen to your crate.
2. You copy $codegen_out_dir/resources/io/k8s/api/core/v1/pod_spec.rs to $kubeclient-rs/resources/pod_spec.rs
3. You decide to include the typed definition of the pub affinity: Option<::io::k8s::api::core::v1::Affinity> field, so you copy $codegen_out_dir/resources/io/k8s/api/core/v1/affinity.rs to $kubeclient-rs/resources/affinity.rs, and change the field to pub affinity: Option<Affinity>
4. Repeat step 3 for any other fields you want to import, recursively for the fields of the new types.

Pros:

• The API is nicer - resources::PodSpec compared to resources::io::k8s::api::core::v1::PodSpec
• Compared to writing PodSpec by hand from scratch, the basic work of field names and types, serde annotations, comments, etc is already done.

Cons:

• The fixups are tedious manual work.
• Completely removing namespacing will prevent things like apps::v1beta1::StatefulSet / apps::v1beta2::StatefulSet / apps::v1::StatefulSet from coexisting.

But now that I think about it, all the definitions do start with io.k8s., so I could have the tool strip that part of the namespace atleast. So you would end up with ::resources::api::core::v1::PodSpec in resources/api/core/v1/pod_spec.rs, which is more palatable and doesn't need manual fixup. And if it's going to be in a separate crate, it can be shortened even more to just ::api::core::v1::PodSpec

What do you think? If you're okay with <some_crate_name>::api::core::v1::PodSpec, etc (both your crate as well as users of your crate would have to type it, even if just in a use statement), I can change the codegen to output that and publish it as a crate.

---

Re: generating the client using paths, I've been experimenting with it (not published to github) and it doesn't seem very useful.

• The generated code is very un-Rust-like, eg

  impl Client {
      fn list_core_v1_namespaced_pod(...) -> ListCoreV1NamespacedPodResult {
      }
  }
  
  enum ListCoreV1NamespacedPodResult {
      Ok(::api::core::v1::PodList),
      Unauthorized,
      Other(::http::StatusCode),
  }

  instead of

  impl Pod {
      fn list(client: Client, ...) -> Result<::api::core::v1::PodList, ::http::StatusCode> {
      }
  }

• The definitions are also not very good, eg one definition has two parameters both named path, so the codegen has to rename one of them to path_.

I think it's best to keep manually-generated clients for now.

[anowell]

<some_crate_name>::api::core::v1::PodSpec seems pretty reasonable. Is everything in the api namespace? If so, I can imagine stripping that too, but definitely not necessary.

Perhaps the biggest downfall I see of using codegen resources from an external crate is exemplified by the Secret resource

Being focused on ergonomics, Secret handles the encode on insert and decode on get, and I intentionally didn't make the data attribute pub. I'm chewing through the possibilities:

1. Codegen within this crate, and be able to attach extra implementations for stuff like this and perhaps manually tweak field visibility in select cases
2. Move all this functionality into traits exposed in kubeclient::prelude and accept that fields like data are public (secret.insert and secret.data.insert do slightly different things)
3. Continue to manually create some of the basic top-level resource types like Pod and Secret with the API I want, but their inner fields (like PodSpec) would come from your codegen types. The downside is that that resource types come from different crates (e.g. kubeclient::resources::Pod contains <your_crate>::api::core::v1::PodSpec, but I suspect most uses of kubeclient wouldn't need to specify inner types too often).

I'm not 100% decided, so definitely open to opinions here, but I lean slightly to starting with the 3rd approach (which is probably the quickest to implement), and consider the other approaches once I have a better feel for integrating the codegen types. In all cases, this will help fill out a lot of incomplete definitions!

[Arnavion]

Is everything in the api namespace?

• io.k8s.apiextensions-apiserver and io.k8s.apimachinery have custom resource- and schema validation-related definitions.
• io.k8s.kube-aggregator has aggregated API server-related definitions.
• io.k8s.kubernetes.pkg has definitions that were deprecated and moved to io.k8s.api, so they're just refs now. Currently the codegen compiles them to type aliases but I'm planning to ignore them completely.

I'm chewing through the possibilities:

I think option 2 will make it error-prone to use. Options 1 and 3 can both work. I agree that option 3 is probably better.

I'll drop the codegen into its own GH repo over the next few days so you can pull it as a git crate dependency and play around with it.

[anowell]

awesome and thanks!

[Arnavion]

Done.

# Cargo.toml
[dependencies]
k8s-openapi = { git = "https://github.com/Arnavion/k8s-openapi-codegen", branch = "master" }

// main.rs
extern crate k8s_openapi;

fn main() {
	let pod_spec: k8s_openapi::api::core::v1::PodSpec = Default::default();
	println!("{:#?}", pod_spec);
}

PodSpec {
    active_deadline_seconds: None,
    affinity: None,
    automount_service_account_token: None,
    containers: [],
    dns_config: None,
    dns_policy: None,
    host_aliases: None,
    host_ipc: None,
    host_network: None,
    host_pid: None,
    hostname: None,
    image_pull_secrets: None,
    init_containers: None,
    node_name: None,
    node_selector: None,
    priority: None,
    priority_class_name: None,
    restart_policy: None,
    scheduler_name: None,
    security_context: None,
    service_account: None,
    service_account_name: None,
    subdomain: None,
    termination_grace_period_seconds: None,
    tolerations: None,
    volumes: None
}

[ayosec]

@Arnavion In your example, the default value for containers is [], but other arrays (like volumes or init_containers) are None. Why is containers different?

EDIT: Sorry, I just saw that containers is defined as required somewhere else in the spec.

[anowell]

Master now uses several generated types from your repo. I'm definitely open to further ideas around codegen, and eventually, we should sync up about publishing these crates, but for now, I have a project that should give me a chance to get some more experience with this client and its codegen'd internals.

[mhristache]

Any plans to progress on this or it's an abandoned track?
Thanks

[anowell]

@maximih

It's not abandoned [yet?], but definitely sidelined at the moment. I started an ncurses-based tool for using this (think htop for kubernetes) to get a feel for the API and verify much of the functionality of this client, but a work project has my attention for at least a few more weeks. I do hope to get back to this soon - but really can't promise/commit to anything yet. I'd happily commit to discussing design changes and reviewing PRs in the meantime.

[Arnavion]

https://crates.io/crates/k8s-openapi/ is now published, with multiple supported versions of Kubernetes (version-specific modules gated by features) and a client API based on the http crate that works with both sync and async HTTP clients.