Add a timeout when initializing the Podman client (broken Podman should not affect `odo dev` on cluster) by rm3l · Pull Request #6808 · redhat-developer/odo

rm3l · 2023-05-10T10:17:56Z

What type of PR is this:
/kind bug
/area dev

What does this PR do / why we need it:
This introduces a timeout for the Podman client initialization (e.g., a timeout for the podman version command to finish). The default value is 1s.

For some reason, this made the Podman tests flaky on GitHub (sometimes, podman version would take more than 1s to return). The timeout has therefore been made configurable (via the PODMAN_CMD_INIT_TIMEOUT environment variable, defaulting to 1s), and set to a slightly higher value in the tests.

Which issue(s) this PR fixes:
Fixes #6575

PR acceptance criteria:

Unit test
Integration test
Documentation

How to test changes / Special notes to the reviewer:
This can be tested by adding a script/program to your PATH that would delay all calls to Podman, e.g.:

mkdir -p ~/.local/bin

cat <<EOF > ~/.local/bin/podman-with-delay
#!/bin/bash

sleep 300 && podman "$@"
EOF

export PODMAN_CMD=~/.local/bin/podman-with-delay

# If ~/.local/bin is not already in your PATH
export PATH="~/.local/bin:$PATH"

At this point, odo dev on cluster should no longer hang.

odo dev --platform=podman would error out, but this is expected.

netlify · 2023-05-10T10:18:06Z

✅ Deploy Preview for odo-docusaurus-preview canceled.

Name	Link
🔨 Latest commit	`2d0a448`
🔍 Latest deploy log	https://app.netlify.com/sites/odo-docusaurus-preview/deploys/645ce681d6ea6900081e51df

odo-robot · 2023-05-10T10:24:34Z

NoCluster Tests on commit 5a0e0ac finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-10T10:24:47Z

OpenShift Unauthenticated Tests on commit 5a0e0ac finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-10T10:27:19Z

Unit Tests on commit 5a0e0ac finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-10T10:32:18Z

Validate Tests on commit 5a0e0ac finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-10T10:56:04Z

Windows Tests (OCP) on commit 5a0e0ac finished with errors.
View logs: TXT HTML

odo-robot · 2023-05-10T12:36:22Z

Kubernetes Tests on commit 5a0e0ac finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-10T12:47:16Z

Kubernetes Docs Tests on commit 7ff460b finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-10T12:49:58Z

OpenShift Tests on commit 5a0e0ac finished with errors.
View logs: TXT HTML

This command is called at dependency injection time to initialize a (nil-able) Podman client, even if users won't use Podman at all. As discussed, this command is supposed to be quite fast to return, hence this timeout of 1 second. Initially, we were using cmd.Output to get the command output, but as reported in [1], cmd.Output does not respect the context timeout. This explains the workaround of reading from both stdout and stderr pipes, *and* relying on cmd.Wait() to close those pipes properly when the program exits (either as expected or when the timeout is reached). [1] golang/go#57129 Co-authored-by: Philippe Martin <[email protected]>

…l Kubernetes/Podman clients could not be initialized This helps debug such potential issues instead of swallowing the errors.

This will allow setting a different value for environments like in GitHub where the Podman client would take slightly more time to return (I guess because of we are running a lot of Podman commands in parallel?).

Some tests did not pass because the Podman client did not initialize in 1s; I guess because we are running a lot of Podman commands in parallel? This should hopefully improve this situation.

pkg/podman/version.go

sonarqubecloud · 2023-05-11T15:39:32Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

rm3l · 2023-05-11T19:26:42Z

Running oc.exe with args [oc create configmap config-map-for-cleanup --from-literal type=testing --from-literal team=odo -n cmd-devfile-deploy-test161cji] and odo env: []
  [oc] error: failed to create configmap: Post "[https://c114-e.eu-de.containers.cloud.ibm.com:30329/api/v1/namespaces/cmd-devfile-deploy-test161cji/configmaps?fieldManager=kubectl-create&quot;:](https://c114-e.eu-de.containers.cloud.ibm.com:30329/api/v1/namespaces/cmd-devfile-deploy-test161cji/configmaps?fieldManager=kubectl-create%22:) dial tcp 149.81.180.114:30329: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
  [FAILED] in [BeforeEach] - C:/Users/Administrator.ANSIBLE-TEST-VS/3936/tests/helper/helper_cmd_wrapper.go:101 @ 05/11/23 11:03:24.24
  Deleting project: cmd-devfile-deploy-test47osp
  Running oc.exe with args [oc delete project cmd-devfile-deploy-test47osp --wait=false] and odo env: []
  [oc] Error from server (NotFound): namespaces "cmd-devfile-deploy-test47osp" not found
  [FAILED] in [AfterEach] - C:/Users/Administrator.ANSIBLE-TEST-VS/3936/tests/helper/helper_cmd_wrapper.go:101 @ 05/11/23 11:03:24.523
  << Timeline

Network errors.

/override windows-integration-test/Windows-test

openshift-ci · 2023-05-11T19:26:46Z

@rm3l: Overrode contexts on behalf of rm3l: windows-integration-test/Windows-test

Details

In response to this:

Running oc.exe with args [oc create configmap config-map-for-cleanup --from-literal type=testing --from-literal team=odo -n cmd-devfile-deploy-test161cji] and odo env: []
 [oc] error: failed to create configmap: Post "[https://c114-e.eu-de.containers.cloud.ibm.com:30329/api/v1/namespaces/cmd-devfile-deploy-test161cji/configmaps?fieldManager=kubectl-create&quot;:](https://c114-e.eu-de.containers.cloud.ibm.com:30329/api/v1/namespaces/cmd-devfile-deploy-test161cji/configmaps?fieldManager=kubectl-create%22:) dial tcp 149.81.180.114:30329: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
 [FAILED] in [BeforeEach] - C:/Users/Administrator.ANSIBLE-TEST-VS/3936/tests/helper/helper_cmd_wrapper.go:101 @ 05/11/23 11:03:24.24
 Deleting project: cmd-devfile-deploy-test47osp
 Running oc.exe with args [oc delete project cmd-devfile-deploy-test47osp --wait=false] and odo env: []
 [oc] Error from server (NotFound): namespaces "cmd-devfile-deploy-test47osp" not found
 [FAILED] in [AfterEach] - C:/Users/Administrator.ANSIBLE-TEST-VS/3936/tests/helper/helper_cmd_wrapper.go:101 @ 05/11/23 11:03:24.523
 << Timeline

Network errors.

/override windows-integration-test/Windows-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rm3l · 2023-05-11T19:29:57Z

dial tcp: lookup c100-e.eu-de.containers.cloud.ibm.com

...
Summarizing 1 Failure:
  [FAIL] odo remove binding command tests when the component with binding is bootstrapped (bindingName=my-nodejs-app-cluster-sample-ocp) when odo dev is running when binding is removed [It] should have led odo dev to delete ServiceBinding from the cluster
  /go/odo_1/tests/integration/cmd_remove_binding_test.go:73

Ran 486 of 886 Specs in 1788.480 seconds
FAIL! -- 485 Passed | 1 Failed | 0 Pending | 400 Skipped

Previous run passed.

/override OpenShift-Integration-tests/OpenShift-Integration-tests

openshift-ci · 2023-05-11T19:30:01Z

@rm3l: Overrode contexts on behalf of rm3l: OpenShift-Integration-tests/OpenShift-Integration-tests

Details

In response to this:

dial tcp: lookup c100-e.eu-de.containers.cloud.ibm.com

...
Summarizing 1 Failure:
 [FAIL] odo remove binding command tests when the component with binding is bootstrapped (bindingName=my-nodejs-app-cluster-sample-ocp) when odo dev is running when binding is removed [It] should have led odo dev to delete ServiceBinding from the cluster
 /go/odo_1/tests/integration/cmd_remove_binding_test.go:73

Ran 486 of 886 Specs in 1788.480 seconds
FAIL! -- 485 Passed | 1 Failed | 0 Pending | 400 Skipped

Previous run passed.

/override OpenShift-Integration-tests/OpenShift-Integration-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rm3l · 2023-05-11T19:31:35Z

/override kubernetes-infra-stage-test

Not related.

openshift-ci · 2023-05-11T19:31:39Z

@rm3l: Overrode contexts on behalf of rm3l: kubernetes-infra-stage-test

Details

In response to this:

/override kubernetes-infra-stage-test

Not related.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added kind/bug Categorizes issue or PR as related to a bug. area/dev Issues or PRs related to `odo dev` labels May 10, 2023

openshift-ci bot requested review from kadel and rnapoles-rh May 10, 2023 10:18

rm3l requested review from feloy and valaparthvi and removed request for rnapoles-rh May 10, 2023 10:18

rm3l changed the title ~~Add a timeout of 1s when initializing the Podman client (broken Podman should not affect odo dev on cluster)~~ Add a timeout when initializing the Podman client (broken Podman should not affect odo dev on cluster) May 10, 2023

rm3l and others added 4 commits May 10, 2023 17:48

Add test highlighting the issue and setting the expectations

397745f

Log the errors returned at dependency injection time when the optiona…

c81610d

…l Kubernetes/Podman clients could not be initialized This helps debug such potential issues instead of swallowing the errors.

Make the timeout configurable via the 'PODMAN_CMD_INIT_TIMEOUT' env var

4cabec2

This will allow setting a different value for environments like in GitHub where the Podman client would take slightly more time to return (I guess because of we are running a lot of Podman commands in parallel?).

rm3l force-pushed the 6575-broken-podman-affects-odo-dev-run-against-cluster branch from 1e4370e to c3a5b23 Compare May 10, 2023 15:48

Increase the timeout for Podman tests to an arbitrary value of 10s

f985ae7

Some tests did not pass because the Podman client did not initialize in 1s; I guess because we are running a lot of Podman commands in parallel? This should hopefully improve this situation.

rm3l force-pushed the 6575-broken-podman-affects-odo-dev-run-against-cluster branch from c3a5b23 to f985ae7 Compare May 10, 2023 15:52

feloy reviewed May 10, 2023

View reviewed changes

pkg/podman/version.go Show resolved Hide resolved

fixup! Add a timeout of 1s to the 'podman version' command

2d0a448

rm3l requested a review from feloy May 11, 2023 13:00

feloy approved these changes May 11, 2023

View reviewed changes

pkg/podman/version.go Show resolved Hide resolved

feloy approved these changes May 11, 2023

View reviewed changes

openshift-ci bot assigned feloy May 11, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. Required by Prow. label May 11, 2023

rm3l closed this May 11, 2023

rm3l reopened this May 11, 2023

openshift-merge-robot merged commit c217741 into redhat-developer:main May 11, 2023

rm3l deleted the 6575-broken-podman-affects-odo-dev-run-against-cluster branch May 11, 2023 19:35

Conversation

rm3l commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for odo-docusaurus-preview canceled.

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

odo-robot bot commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented May 11, 2023

Uh oh!

rm3l commented May 11, 2023

Uh oh!

openshift-ci bot commented May 11, 2023

Uh oh!

rm3l commented May 11, 2023

Uh oh!

openshift-ci bot commented May 11, 2023

Uh oh!

rm3l commented May 11, 2023

Uh oh!

openshift-ci bot commented May 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rm3l commented May 10, 2023 •

edited

Loading

netlify bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading

odo-robot bot commented May 10, 2023 •

edited

Loading