Troubleshooting
This page documents possible solutions for the most common issues reported by users.
If your issue is not listed here, feel free to reach out to the team in the #odo channel on the Kubernetes Slack. Or you can also file an issue or start a new discussion.
Authentication issues
odo deploy is failing to push container image components due to 401 errors
Description
odo is failing to push the Image component from the Devfile. It looks like it is trying to push images to the docker.io registry.
Example output
This most commonly happens with odo deploy, but might also happen with odo dev if an Image component is included in the Devfile as part of the inner-loop workflow.
Possible Causes
The most common cause for this issue is that the Image component from the local Devfile is using a relative name in its imageName field.
For example:
[...]
components:
- name: image-build
image:
imageName: go-image:latest
dockerfile:
uri: docker/Dockerfile
buildContext: .
rootRequired: false
Note that we recommend using relative image names to keep the Devfile portable.
Recommended Solution
As of odo v3.11.0, we are handling relative image names as selectors, provided that you locally indicate to odo a registry where to push those relative images.
See How odo handles image names for more details.
To fix this issue, use the command below to set your ImageRegistry preference before calling odo deploy.
odo preference set ImageRegistry $registry
Example output
Dev Sessions issues
The Image Pull Policy of the dev container is Always and I cannot change it
Description
When starting odo dev, the image pull policy of the dev container is currently hardcoded to Always,
which may not be ideal for all platforms.
Recommended Solution
The image pull policy can be changed by declaring a container-overrides attribute in the container component in the Devfile, like so:
components:
- name: runtime
attributes:
container-overrides:
imagePullPolicy: IfNotPresent
container:
command: ['tail']
args: ['-f', '/dev/null']
endpoints:
- name: http-node
targetPort: 3000
image: registry.access.redhat.com/ubi8/nodejs-16:latest
mountSources: true
I'm getting "Permission denied" errors when odo dev is syncing files
Description
When running odo dev against certain clusters, the Syncing files into the container stage fails due to "Permission denied" errors.
Example output
Possible Causes
Various factors are responsible for this:
- Storage Provisioner used for the cluster
- User set by the container image
- Location on the container where the files are to be synced
- Using Ephemeral vs Non-Ephemeral Volumes
Recommended Solution
Please refer to Troubleshoot Storage Permission issues on managed cloud providers clusters for possible solutions to fix this.
Orphan Devstate files
An odo dev session creates a .odo/devstate.<PID>.json file when the session starts, and deletes it at the end of the session.
If the session terminates abrupty, the state file won't be deleted, and will remain in the .odo directory.
You can delete such orphan devstate files using the command odo delete component.
Example output
Podman Issues
odo says it cannot access Podman
Description
odo dev --platform podman fails to start with the following error:
✗ unable to access podman.Do you have podman client installed and configured correctly?
cause: executable "podman" not recognized as podman client
This seems flaky however, because sometimes it works.
Possible Causes
When initializing our connector to Podman, we have a default timeout of 1 second for the Podman executable to respond.
For some reason, odo sometimes might not be able to get a response from Podman during this short period of time.
Recommended Solution
First make sure the podman version -f json command returns in a timely manner on your system.
If the podman command cannot be found, please make sure Podman is installed correctly and available on your system path.
Or you can set the PODMAN_CMD environment variable to indicate where odo can find the podman executable, for example:
export PODMAN_CMD=/absolute/path/to/podman
If the podman executable is available, you can increase the timeout using the PODMAN_CMD_INIT_TIMEOUT environment variable.
You can increase this timeout before running odo and this should hopefully fix the issue, for example:
export PODMAN_CMD_INIT_TIMEOUT=10s
Please file an issue if the problem persists.
I am using the remote Podman client and I am unable to reach the ports forwarded by odo
Description
Your local Podman is configured as a remote client, interacting with another Podman Backend node via SSH.
For example, you are using the podman-remote binary, or are calling Podman with the --remote option.
The Podman remote client works fine, and you can start an odo Dev session leveraging it, e.g.:
$ odo dev --platform podman
[...]
✓ Waiting for the application to be ready [1s]
- Forwarding from 127.0.0.1:20001 -> 3000
↪ Dev mode
Status:
Watching for changes in the current directory /path/to/project
Web console accessible at http://localhost:20000/
Keyboard Commands:
[Ctrl+c] - Exit and delete resources from podman
[p] - Manually apply local changes to the application on podman
However, the port forwarded by odo dev seems unreachable, e.g.:
$ curl http://127.0.0.1:20001
curl: (7) Failed to connect to localhost port 20001 after 0 ms: Connection refused
Possible causes
Podman provides the ability to use a local client interacting with a Podman backend node through an SSH connection. However, as explained in this discussion, all resources are only created on the backend node. This includes any ports that might need to be forwarded.
Recommended Solution
Since you have already configured the SSH connection between the Podman remote client and the backend node, one possible workaround
could be to manually open up an SSH tunnel (using the same credentials) right after starting the odo dev session.
This way, SSH will do the work of forwarding ports between the machine running odo dev (along with the Podman remote client) and the Podman backend node.
Example:
$ ssh -v -i /path/to/ssh/key -NL 20001:127.0.0.1:20001 -l $user $podman_backend_host
Right after creating the SSH Tunnel in a separate terminal , you will be able to reach the port displayed by odo for port forwarding.
More details on SSH Tunneling.
I have an image component in my Devfile and odo dev on Podman errors out with requested access to the resource is denied
Description
You are iterating locally with Podman and have a Devfile more or less similar to the one below, in which you are building a local container image and starting a container component based on the image built.
components:
- image:
autoBuild: true
dockerfile:
buildContext: .
rootRequired: false
uri: Containerfile-web
imageName: web
name: webimage
- container:
image: web
name: web
metadata:
name: helloworld
schemaVersion: 2.2.0
Running odo dev --platform=podman would error out with the following error message:
$ odo dev --platform=podman
[...]
↪ Building & Pushing Image: webimage
• Building image locally ...
[...]
✓ Building image locally [23s]
• Pushing image to container registry ...
[...]
Error: writing blob: initiating layer upload to /v2/library/webimage/blobs/uploads/
in registry-1.docker.io: requested access to the resource is denied
[...]
Possible causes
In the example above, the webimage image component has autoBuild set to true, which means that it will be built automatically at start time.
And when handling an image component, odo dev will try to build it and push it.
In this case, because it is a relative image name (imageName: web), by default, the container engine tries to push it to docker.io. Therefore, we will need to:
- either instruct
odo devnot to push images built, - or set an
ImageRegistrypreference indicating where to push the images.
If you instruct odo dev not to push images, odo might still fail when trying to create the pod on Podman, with an error similar to the following:
✗ Deploying pod [462ms]
Error occurred on Push - exit status 125:
Complete Podman output:
Trying to pull localhost/web:latest...
Error: initializing source docker://localhost/web:latest:
pinging container registry localhost:
Get "https://localhost/v2/":
tls: failed to verify certificate:
x509: certificate is valid for *.apps-crc.testing, not localhost
This is because the imagePullPolicy is Always by default; Podman will try to pull the image, and uses localhost as the default search registry for relative image names.
Recommended solution
- If the Container component in the Devfile uses the image built from an Image component, you'll need to override the Image Pull Policy, as depicted in The Image Pull Policy of the dev container is
Alwaysand I cannot change it. From the example Devfile above, this would mean adding the followingcontainer-overridesattributes to the Container component:
components:
- image:
autoBuild: true
dockerfile:
buildContext: .
rootRequired: false
uri: Containerfile-web
imageName: web
name: webimage
- container:
image: web
name: web
attributes:
container-overrides:
imagePullPolicy: IfNotPresent
metadata:
name: helloworld
schemaVersion: 2.2.0
- The second step depends on whether you want to push the images built or not.
If you do not want to push images or do not have access to a registry
You need to instruct odo dev not to push image components since you don't have access to a registry. This can be done by setting the ODO_PUSH_IMAGES environment variable to false. See Environment variables controlling odo behavior.
ODO_PUSH_IMAGES=false odo dev --platform podman
Example Output
If you want to push images to a registry
You can set the ImageRegistry preference to a registry where you have access. See How odo handles image names for further details.
Note that Podman will need to be able to pull images from that location.
odo preference set ImageRegistry quay.io/$USER
Now running odo dev --platform=podman should work by pushing and pulling the images built from that location.