Painpoints: What have you faced? What have you done so far to improve those? What's your plan for the future?

[fidencio]

Dear candidates,

I'd like to ask each and every one of you about the painpoints you, particularly, have faced with the project so far; what are the actions you've taken so far as a community member to improve those; and what is your plan for the future related to the painpoints you've faced.

/cc @sameo, @bergwolf, @amshinde, @fengwang666, @jepio, @stevenhorsman.

[bergwolf]

Let me start with the top one thing in my mind: vulnerability handling.

We have a vulnerability management team, which did handle several urgent vulnerabilities in the past. But as we can see in the KCSA repository, the latest update there was almost three years ago. The VMT did analyze a few more vulnerability issues in private in the past, but our exposure to possible vulnerabilities is decreasing.

As proof, we just saw (today) a to-be-published paper on how to attach Kata Containers in several ways (https://www.usenix.org/conference/usenixsecurity23/presentation/xiaojietao). It is a pity that the paper authors chose not to contact the community to disclose these issues and get confirmation or fixes.

In the past, I have served on the vulnerability management team (and still am). But we mostly only handle security issues reported to us directly. In the future, I think we should try to improve the situation in several ways:

1. Make it more clear for users, developers, and security researchers, that we have a vulnerability handling process, and make it easier for them to report security issues.
2. Enroll more folks who really care about the project's health (like active developers, and downstream vendors/users) in the VMT
3. Handle security issues more quickly (we did have a few occasions that we almost missed the disclosure deadline)

IMO, we really need more effort and focus on the security part of the project.

Cheers,
Tao

PS, I have created a private issue https://bugs.launchpad.net/katacontainers.io/+bug/2016086 to track and discuss the possible vulnerabilities as disclosed in the paper.

[fengwang666]

To be fair, some of issues included in the paper are actually disclosed to us a while ago (kata-containers/kata-containers#4935, kata-containers/kata-containers#4936). Totally agree we can do better on vulnerability handling.

[fengwang666]

@bergwolf Can you grant me access to the launchpad issue?

[bergwolf]

@fengwang666 sure, what is your lanchpad id?

[stevenhorsman]

The major pain point that I've face in the Kata Containers CCv0 work has been around CI/CD.

To start with we didn't have any CI/CD, or even an integration branch, so I help to set that up and create the initial test jobs. Over time these have grown a lot as we've added more CC tests and lots more platforms to test - some of which (for TEEs) are behind company firewalls, or on company owned machines, which adds to the difficulties in identifying and fixing issues.

So far my main contribution to this, (apart from be part of the problem in adding the new integration tests and platforms) has been to monitor the CC PRs, ensure that tests are running and if there are problem to a) try and re-run them and b) contact the owners to help with problem determination, or fixes on those boxes.

Going forward I see this problem from being much improved if the vPTG presentation from Fabiano can be implemented - where we build the deploy payload once (ideally, but maybe once for each arch) and then just test on the wide range of platforms we have. This should cut down the time and chance of failures a lot, so I plan to continue to support this effort as best as I can.

We've also had a challenge of keeping the CCv0 branch up-to-date with main. For the last 18 months me, or someone on my team has had the weekly task of merging the main codebase in, resolved the conflicts, getting it tested and ensuring that we stay up to date without breaking the commit history and causing developers issues. In the short-term we'll carry on with this, but we should be able to start executing on the plan we've been talking about for a long time to get the CCv0 specific code merged into main, in a controlled way, so I'm planning to be involved in helping with this.

[jepio]

I'd like to ask each and every one of you about the painpoints you, particularly, have faced with the project so far

My pain points so far have involved CI, CD, and stale code/docs. I'll go front to the back.

My first attempts at deploying kata containers a while back failed: as a developer I tried to deploy main and hit issues with cgroups compatibility between host and guest. The debugging experience was also difficult. I immediately started posting PRs to fix the issues I faced, both to the CCv0 and main branches. One of the fixes I posted was about making kata-agent output more visible in the host. I have also been working with people on slack, helping them out when they faced similar issues as myself.

I tend to follow the principles of "leave a place cleaner than how you found it" or "if you see trash, don't step over it - pick it up".

While working through my PRs I noticed flakiness in some of the tests running in CI, or it being hard to tell why a test failed, or some really old references in tests/workflows (travis anyone?). I started fixing those, so that others have a better contributor experience. Fixing those issues also helped me figure out better how the whole codebase works, how the infrastructure is set up etc.

I wanted to learn more about releases, so I volunteered to help out with the CoCo release process. This has been eye opening and showed me how things interact across repositories. What I saw scared me a bit. Together with the other volunteers, we managed to put together a checklist that is now the reference for how to prepare a release, but more needs to be done. I definitely want to focus either on automating the whole release pipeline or start the discussion on why repositories should be brought together to reduce the number of moving pieces.

There's more to be done on all 3 fronts and I intend to continue pushing those areas forward and help anyone else who wants to join forces.

[fengwang666]

Since I first started interacting with the project, I have been trying to adopt Kata Containers for running production workloads. However, in my opinion, there are two main pain points that currently hinder its widespread adoption:

Firstly, the project still lacks production maturity. The code quality and test coverage still have plenty of room for improvement. Given the complexity of the Kata Containers project and the fact that it supports many different use cases, there are many code branches that lack test coverage, especially edge cases. I've come across tons of critical bugs that can cause serious problems in production. Additionally, operational features such as metrics and logging are lagging behind. For example, at present, we still cannot obtain agent error logs without enabling debug mode. While I understand that this may be a chicken-and-egg problem, if we as a community are truly committed to production adoption, we need to focus more on less glamorous tasks such as testing, bug-fixing, and improving logging and metrics.

Secondly, there is a steep learning curve to running production workloads on Kata Containers. While it is easy to get a prototype up and running, there are many more considerations to be made when moving into production. These include measuring and tuning workload performance, evaluating the security risks of the entire stack, and assessing the cost of operating the whole stack. For small companies without the necessary expertise or resources, the thought of operating and maintaining an end-to-end virtualization stack can be daunting. It makes me wonder if it's even possible for regular folks to use this kind of technology, or if it's just for the big dogs like operating system vendors, hyperscale cloud service providers, and chip makers. As an OSS community, if our goal is to make Kata Containers available to everyone, we need to lower the entry barrier by adding more documentation, sharing performance tuning experiences, and building stronger security support for the whole stack, including vulnerability management and embargo notification.