Skip to content

Export compatibility config.sandboxImage in CRI#11114

Open
afbjorklund wants to merge 2 commits intocontainerd:mainfrom
afbjorklund:config-sandbox
Open

Export compatibility config.sandboxImage in CRI#11114
afbjorklund wants to merge 2 commits intocontainerd:mainfrom
afbjorklund:config-sandbox

Conversation

@afbjorklund
Copy link

@afbjorklund afbjorklund commented Dec 8, 2024

The kubeadm preflight checks for config.sandboxImage in CRI,
make sure that this value is present in the exported "config".

Fixes the issue that was previously always shown by kubeadm:
detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm

kubernetes/kubernetes@d12b4d4

Note that this key (config.sandboxImage) is also being used by other runtimes, such as CRI-O and Docker.
Even though they don't have such a config, it still has to be added to the CRI status because compatibility.

cri-o/cri-o@875c3a2


It is part of the CRI API from Kubernetes 1.27

But it only works with raw maps and strings...

        type config struct {
                SandboxImage string `json:"sandboxImage,omitempty"`
        }

cmd/kubeadm/app/util/runtime/runtime.go

@k8s-ci-robot
Copy link

Hi @afbjorklund. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@dims
Copy link
Member

dims commented Dec 9, 2024

/assign @mikebrow

@thaJeztah thaJeztah added the cherry-pick/2.0.x Change to be cherry picked to release/2.0 branch label Dec 9, 2024
Copy link
Member

@mikebrow mikebrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comments.. does the crio team not have a list of pinned images?

Can you add a test case with the config entry in pinned set to nil, and a variation with the config set to something different than the default sandbox image.. Do you want "" in info or the actual that will be used when it's not set?

I don't follow the issue we are trying to fix, why would kubeadm provide a warning if the version number in the config or container runtime const default is not the same as for kubeadm?

return snapshotstore.Snapshot{}, errors.New("not implemented")
}

func (s *fakeImageService) PinnedImage(name string) string { return "" }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no error, so it can't return NotImplemented

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not following the add a fakeimageservce pinnedimage api here that only returns ""

Is that useful for the tests?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goes with the why are we promoting a pinned image api in our private image service

Copy link
Author

@afbjorklund afbjorklund Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a map now, for the testing. i.e. so that you can leave it at default (""), or override

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't bother with a fake ImageConfig, just setting the pinned image directly in the test

GetImage(id string) (imagestore.Image, error)
GetSnapshot(key, snapshotter string) (snapshotstore.Snapshot, error)

PinnedImage(name string) string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

? not sure if we need / want to promote this func or not

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internal/cri/server/status.go:70:34: c.ImageService.PinnedImage undefined (type ImageService has no field or method PinnedImage) was the error if not promoting it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get image for the sandbox .labels["pinned"] I think?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was my impression that it was where the information could be found now, after 3baf5ed

criconfig.Config
// kubeadm hardcodes config.sandboxImage
SandboxImage string `json:"sandboxImage"`
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is unfortunate.. as we already had a map field for pinned images in the config

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The map won't get exported. The bigger problem here is that our config structure should not become an unofficial external interface. This really needs to be documented and fields should be copied into a set of supported fields rather than dumping our config struct.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the kubeadm implementation followed https://www.hyrumslaw.com/, when it was added back then

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bigger problem here is that our config structure should not become an unofficial external interface.

In this thread, it was a known shortcoming:
kubernetes/kubeadm#2020 (comment)

But it merged anyway, and others retrofitted.
kubernetes/kubernetes#115610 (review)


https://github.com/kubernetes/kubernetes/blob/v1.27.0/cmd/kubeadm/app/util/runtime/runtime.go#L180
https://github.com/kubernetes/kubernetes/blob/v1.32.0/cmd/kubeadm/app/util/runtime/runtime.go#L287

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my point wasn't that we should use the containerd config as an exported structure types, though yes that was done for info fields, my point was this is but a small subset of the interesting image labels for images... A new field, for one of many that would be needed for various images with certain labels...

An array of name value pairs could be provided as kube has interest in the list of pinned images, though they already get that information from image list with the pinned label and sandbox label.

// PinnedImageLabelKey is the label value indicating the image is pinned.
PinnedImageLabelKey = criContainerdPrefix + ".pinned"
// PinnedImageLabelValue is the label value indicating the image is pinned.
PinnedImageLabelValue = "pinned"
// ContainerKindLabel is a label key indicating container is sandbox container or application container
ContainerKindLabel = criContainerdPrefix + ".kind"
// ContainerKindSandbox is a label value indicating container is sandbox container
ContainerKindSandbox = "sandbox"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be noted that after an upgrade, service or config change we will have some running pods on different pinned sandbox images. We don't force pod restart due to a change in which version we are running. We've never had an issue to my knowledge, wrt having different pause versions at the same time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only place we would run into a "problem" is if the pause image desired is not local and we are unable to pull it, due to being disconnected or having a connection issue with the registry.

Copy link
Member

@mikebrow mikebrow Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding config verification, arguably makes us more brittle for connected environments or if some client is requiring a particular pause image for no particular reason

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The status of the images (if they are present or pinned) shouldn't matter, kubeadm only checks the config

@afbjorklund
Copy link
Author

I don't follow the issue we are trying to fix, why would kubeadm provide a warning if the version number in the config or container runtime const default is not the same as for kubeadm?

It was a part of moving the sandbox configuration from the kubelet, to the container runtime:

Some users miss the fine print on the installation page, and "forget" to configure their runtime:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

@afbjorklund
Copy link
Author

afbjorklund commented Dec 10, 2024

see comments.. does the crio team not have a list of pinned images?

The pause image is pinned automatically:

s.StorageImageServer().UpdatePinnedImagesList(append(s.config.PinnedImages, s.config.PauseImage))

The "config" in crictl info is only for information:

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "runtimeHandlers": [
    {
      "name": "runc",
      "features": {}
    },
    {
      "name": "crun",
      "features": {
        "user_namespaces": true
      }
    },
    {
      "features": {
        "user_namespaces": true
      }
    }
  ],
  "config": {
    "sandboxImage": "registry.k8s.io/pause:3.9"
  }
}

Can you add a test case with the config entry in pinned set to nil, and a variation with the config set to something different than the default sandbox image.. Do you want "" in info or the actual that will be used when it's not set?

So far I only validated it manually, using crictl info (verbose)

It should return the value that is actually being used, for pause.

@afbjorklund
Copy link
Author

afbjorklund commented Dec 13, 2024

There is a separate issue of updating the Kubernetes documentation, to cover both v2 and v3 containerd configuration.

Configuration migrated from version 2, use `containerd config migrate` to avoid migration

But it it still being upgraded with 2.0, the only problem is when the user tries v1 (...by overwriting the default contents)

error="strict mode: fields in the document are missing in the target struct" key=sandbox_image plugin=io.containerd.grpc.v1.cri

The lima template uses: sandbox_image = "$(kubeadm config images list | grep pause | sort -r | head -n1)"

But there is no information in the Kubernetes documentation, on how you find out what image to actually use?

For now, it has been updated to the latest version (3.10) - so it won't be outdated until pause 3.11 is out...

@github-actions
Copy link

github-actions bot commented Oct 1, 2025

This PR is stale because it has been open 90 days with no activity. This PR will be closed in 7 days unless new comments are made or the stale label is removed.

@github-actions github-actions bot added the Stale label Oct 1, 2025
@afbjorklund
Copy link
Author

afbjorklund commented Oct 1, 2025

This is still failing with kubeadm 1.34.1 and containerd 2.1.3, just that it does so much more silently now:

failed to detect the sandbox image for local container runtime, no 'sandboxImage' field in CRI info config

Kubernetes is using registry.k8s.io/pause:3.10.1, while containerd defaults to registry.k8s.io/pause:3.10 sandbox.

So the users are still required to read, and update their configuration according to the documentation.


Note: the documentation has not been updated.

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"

Need to check with kubeadm config images list

app/constants/constants.go:     // PauseVersion indicates the default pause image version for kubeadm
app/constants/constants.go:     PauseVersion = "3.10.1"

@afbjorklund
Copy link
Author

afbjorklund commented Oct 1, 2025

As before, this doesn't really impact anything because all the changes made to pause are Windows-only (for now)...

# 3.10.1

* Updating base image for Windows pause container

# 3.10

* Add support for the -v flag on Windows. It prints the version similarly to Linux.

# 3.9

* Unsupported Windows Semi-Annual Channel container images removed

# 3.8

* Updating base image for Windows container images 

# 3.7

* Unsupported Windows Semi-Annual Channel container images removed

The kubeadm preflight checks for config.sandboxImage in CRI,
make sure that this value is present in the exported "config".

Signed-off-by: Anders F Björklund <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cri Container Runtime Interface (CRI) cherry-pick/2.0.x Change to be cherry picked to release/2.0 branch needs-ok-to-test size/M Stale

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

containerd 2.0.0 kubernetes bug

6 participants