Export compatibility config.sandboxImage in CRI#11114
Export compatibility config.sandboxImage in CRI#11114afbjorklund wants to merge 2 commits intocontainerd:mainfrom
Conversation
|
Hi @afbjorklund. Thanks for your PR. I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
a115c7a to
f5a7a93
Compare
|
/assign @mikebrow |
mikebrow
left a comment
There was a problem hiding this comment.
see comments.. does the crio team not have a list of pinned images?
Can you add a test case with the config entry in pinned set to nil, and a variation with the config set to something different than the default sandbox image.. Do you want "" in info or the actual that will be used when it's not set?
I don't follow the issue we are trying to fix, why would kubeadm provide a warning if the version number in the config or container runtime const default is not the same as for kubeadm?
| return snapshotstore.Snapshot{}, errors.New("not implemented") | ||
| } | ||
|
|
||
| func (s *fakeImageService) PinnedImage(name string) string { return "" } |
There was a problem hiding this comment.
There is no error, so it can't return NotImplemented
There was a problem hiding this comment.
Not following the add a fakeimageservce pinnedimage api here that only returns ""
Is that useful for the tests?
There was a problem hiding this comment.
goes with the why are we promoting a pinned image api in our private image service
There was a problem hiding this comment.
I added a map now, for the testing. i.e. so that you can leave it at default (""), or override
There was a problem hiding this comment.
Didn't bother with a fake ImageConfig, just setting the pinned image directly in the test
| GetImage(id string) (imagestore.Image, error) | ||
| GetSnapshot(key, snapshotter string) (snapshotstore.Snapshot, error) | ||
|
|
||
| PinnedImage(name string) string |
There was a problem hiding this comment.
? not sure if we need / want to promote this func or not
There was a problem hiding this comment.
internal/cri/server/status.go:70:34: c.ImageService.PinnedImage undefined (type ImageService has no field or method PinnedImage) was the error if not promoting it
There was a problem hiding this comment.
get image for the sandbox .labels["pinned"] I think?
There was a problem hiding this comment.
It was my impression that it was where the information could be found now, after 3baf5ed
| criconfig.Config | ||
| // kubeadm hardcodes config.sandboxImage | ||
| SandboxImage string `json:"sandboxImage"` | ||
| } |
There was a problem hiding this comment.
this is unfortunate.. as we already had a map field for pinned images in the config
There was a problem hiding this comment.
The map won't get exported. The bigger problem here is that our config structure should not become an unofficial external interface. This really needs to be documented and fields should be copied into a set of supported fields rather than dumping our config struct.
There was a problem hiding this comment.
I think the kubeadm implementation followed https://www.hyrumslaw.com/, when it was added back then
There was a problem hiding this comment.
The bigger problem here is that our config structure should not become an unofficial external interface.
In this thread, it was a known shortcoming:
kubernetes/kubeadm#2020 (comment)
But it merged anyway, and others retrofitted.
kubernetes/kubernetes#115610 (review)
https://github.com/kubernetes/kubernetes/blob/v1.27.0/cmd/kubeadm/app/util/runtime/runtime.go#L180
https://github.com/kubernetes/kubernetes/blob/v1.32.0/cmd/kubeadm/app/util/runtime/runtime.go#L287
There was a problem hiding this comment.
my point wasn't that we should use the containerd config as an exported structure types, though yes that was done for info fields, my point was this is but a small subset of the interesting image labels for images... A new field, for one of many that would be needed for various images with certain labels...
An array of name value pairs could be provided as kube has interest in the list of pinned images, though they already get that information from image list with the pinned label and sandbox label.
containerd/internal/cri/labels/labels.go
Lines 26 to 33 in 9b89fb9
There was a problem hiding this comment.
it should be noted that after an upgrade, service or config change we will have some running pods on different pinned sandbox images. We don't force pod restart due to a change in which version we are running. We've never had an issue to my knowledge, wrt having different pause versions at the same time.
There was a problem hiding this comment.
The only place we would run into a "problem" is if the pause image desired is not local and we are unable to pull it, due to being disconnected or having a connection issue with the registry.
There was a problem hiding this comment.
adding config verification, arguably makes us more brittle for connected environments or if some client is requiring a particular pause image for no particular reason
There was a problem hiding this comment.
The status of the images (if they are present or pinned) shouldn't matter, kubeadm only checks the config
f5a7a93 to
cf793ac
Compare
It was a part of moving the sandbox configuration from the kubelet, to the container runtime: Some users miss the fine print on the installation page, and "forget" to configure their runtime: https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
The pause image is pinned automatically:
The "config" in {
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
},
"runtimeHandlers": [
{
"name": "runc",
"features": {}
},
{
"name": "crun",
"features": {
"user_namespaces": true
}
},
{
"features": {
"user_namespaces": true
}
}
],
"config": {
"sandboxImage": "registry.k8s.io/pause:3.9"
}
}
So far I only validated it manually, using It should return the value that is actually being used, for pause. |
|
There is a separate issue of updating the Kubernetes documentation, to cover both v2 and v3 containerd configuration. But it it still being upgraded with 2.0, the only problem is when the user tries v1 (...by overwriting the default contents) The lima template uses: But there is no information in the Kubernetes documentation, on how you find out what image to actually use? For now, it has been updated to the latest version (3.10) - so it won't be outdated until pause 3.11 is out... |
|
This PR is stale because it has been open 90 days with no activity. This PR will be closed in 7 days unless new comments are made or the stale label is removed. |
|
This is still failing with kubeadm 1.34.1 and containerd 2.1.3, just that it does so much more silently now: Kubernetes is using So the users are still required to read, and update their configuration according to the documentation. Note: the documentation has not been updated. Need to check with |
|
As before, this doesn't really impact anything because all the changes made to |
The kubeadm preflight checks for config.sandboxImage in CRI, make sure that this value is present in the exported "config". Signed-off-by: Anders F Björklund <[email protected]>
2fddf90 to
c565741
Compare
Signed-off-by: Anders F Björklund <[email protected]>
c565741 to
149c352
Compare
The kubeadm preflight checks for config.sandboxImage in CRI,
make sure that this value is present in the exported "config".
Fixes the issue that was previously always shown by kubeadm:
detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadmkubernetes/kubernetes@d12b4d4
Note that this key (
config.sandboxImage) is also being used by other runtimes, such as CRI-O and Docker.Even though they don't have such a config, it still has to be added to the CRI status because compatibility.
cri-o/cri-o@875c3a2
It is part of the CRI API from Kubernetes 1.27
But it only works with raw maps and strings...
cmd/kubeadm/app/util/runtime/runtime.go