Use optional fields for ClusterConfig

[neoaggelos]

Summary

Update all cluster configurations to use *<type> fields, such that they can be set to empty values. The changes are extensive but are mostly to introduce structure to the code, and ensure that all checks happen in the right place

Notes

• This PR introduces huge breaking changes in the cluster configuration. These were left out during the initial development to get a v0 working implementation, but are required going forward. After merging this PR, any running clusters that migrate to this will be effectively bricked
• There is no point in working towards a migration path for now, as there has not been any stable releases yet. For the same reason, it is best to do the changes before there is a stable release.

Changes

• apiv1.UserFacingClusterConfig does not have nil structs, all individual fields can be nil instead.
• Add Getter() functions for all fields, to avoid having to check for pointers all around the code. To facilitate this, all zero values and nil values of fields should behave identically.
• types.ClusterConfig rework, same as above. All fields can be nil, and all structs are never nil.
• Move convert functions from API <-> internal types in the types package
• Move merge config validations into the types package
• Add extensive unit tests for all cluster config field updates, and also test for specific scenarios that match our validations.
• database.SetClusterConfig() now also returns the merged cluster config, simplifying code in the caller sites.
• Use json for storing the cluster configuration.
• Ensure types.ClusterConfig.SetDefaults() sets all fields such that they are not nil.
• Update all call sites (there are a lot of these, since the internal types are used throughout the API and hooks). Most are about using the designated Getters instead of the struct fields.

Previous issues

• Validating the merged cluster configuration happened in two steps, in types.MergeClusterConfig and then in api.validateConfig. This is now handled properly in types.MergeClusterConfig instead. Extensive tests have been added to validate the expected behaviours

• Default values for the cluster config were split between (*types.ClusterConfig).SetDefaults() and utils.GetUserFacingClusterConfig, making it hard to track around the code if a particular value is nil, or if the user will be presented with slightly misleading configs.

• Converting to and from userfacing config has been moved to the types package, it does not belong in utils, especially not coupled with retrieving the cluster config from the database

• It was impossible to "unset" configurations after they had been configured (e.g. reset the load-balancer.cidrs) field. We now use optional fields for everything (*type). We now consider nil values instead of zero values as "don't update". Further, we ensure that for each field the zero and the nil value behave in the same way, such that we can have straightforward Get$Field() and simplify the code.

This now works:

root@t1:~# k8s get load-balancer.cidrs 
[]
root@t1:~# k8s set load-balancer.cidrs=10.0.100.0/24
Configuration updated.
root@t1:~# k8s get load-balancer.cidrs 
[10.0.100.0/24]
root@t1:~# k8s set load-balancer.cidrs=
Configuration updated.
root@t1:~# k8s get load-balancer.cidrs 
[]

• Fixes a bug where setting dns.enabled=false and a dns.service-ip did not work, as it was overwritten to an empty config.

[bschimke95]

Great work as always @neoaggelos
Just a few nits.

[neoaggelos]

Waiting for #279 to rebase

[bschimke95]

LGTM

[github-actions]

Package	Line Rate
github.com/canonical/k8s/api/v1	42%
github.com/canonical/k8s/cmd/k8s	30%
github.com/canonical/k8s/cmd/util	14%
github.com/canonical/k8s/pkg/client/dqlite	44%
github.com/canonical/k8s/pkg/component	10%
github.com/canonical/k8s/pkg/k8sd/controllers	71%
github.com/canonical/k8s/pkg/k8sd/database	48%
github.com/canonical/k8s/pkg/k8sd/pki	55%
github.com/canonical/k8s/pkg/k8sd/setup	70%
github.com/canonical/k8s/pkg/k8sd/types	69%
github.com/canonical/k8s/pkg/proxy	6%
github.com/canonical/k8s/pkg/snap	15%
github.com/canonical/k8s/pkg/snap/util	90%
github.com/canonical/k8s/pkg/utils	40%
github.com/canonical/k8s/pkg/utils/control	61%
github.com/canonical/k8s/pkg/utils/errors	100%
github.com/canonical/k8s/pkg/utils/k8s	71%
Summary	39% (1867 / 4738)