Submissions (in "[TUTO] The INGInious tutorial" course) timing out, CPU at 0%? #696

dabrowskiw · 2021-08-15T08:08:35Z

dabrowskiw
Aug 15, 2021

Hi everyone!

I have a curious problem that I wasn't able to find anything on, so I'm trying my luck here. I have just installed Inginious on a debian 10.10 machine with apache2 following the instructions (roughly - docker and mongodb installations are slightly different on debian than on ubuntu, and I had to add www-data to the docker group, otherwise I got "Service not available" errors when submitting tasks because the web server process could not start docker images). However, I still am not able to successfully submit even the first "Getting started" task: I always get "Your submission timed out. Your score is 0.0%". /var/log/apache2/error.log only days that the job was started, and then after the timeout it says that it's killing the job. I tried increasing the timeout to 30 seconds, no dice. When I watch top after submitting the job, I can see dockerd flash up for a moment, but then it just sits there at 0% CPU until it gets killed due to timeout. I unfortunately have no experience with docker, so I have no idea where to look to find out what might be causing that.

By the way, This is what happens in /var/log/daemon.log (the dockerd log on debian) when I do the submission:

Aug 15 08:02:48 inginious-ikg dockerd[427]: time="2021-08-15T08:02:48.332563322Z" level=warning msg="Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap."
Aug 15 08:02:48 inginious-ikg systemd[1]: var-lib-docker-overlay2-8b8ad0557504b708968c89d99e8535e684d45b134cdeb90c6fef804e0dc65e1c\x2dinit-merged.mount: Succeeded.
Aug 15 08:02:48 inginious-ikg containerd[347]: time="2021-08-15T08:02:48.446994151Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/0c3c6eeb557ee9f3656a80ee2cc29aab3afde1fe65b4330161be3d5e47aba622 pid=1488
Aug 15 08:03:18 inginious-ikg dockerd[427]: time="2021-08-15T08:03:18.807622852Z" level=info msg="ignoring event" container=0c3c6eeb557ee9f3656a80ee2cc29aab3afde1fe65b4330161be3d5e47aba622 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Aug 15 08:03:18 inginious-ikg containerd[347]: time="2021-08-15T08:03:18.807807045Z" level=info msg="shim disconnected" id=0c3c6eeb557ee9f3656a80ee2cc29aab3afde1fe65b4330161be3d5e47aba622
Aug 15 08:03:18 inginious-ikg containerd[347]: time="2021-08-15T08:03:18.807867082Z" level=error msg="copy shim log" error="read /proc/self/fd/12: file already closed"
Aug 15 08:03:18 inginious-ikg systemd[1]: run-docker-netns-3808aacb17ba.mount: Succeeded.
Aug 15 08:03:18 inginious-ikg systemd[1]: var-lib-docker-overlay2-8b8ad0557504b708968c89d99e8535e684d45b134cdeb90c6fef804e0dc65e1c-merged.mount: Succeeded.

And this is what happens in /var/log/apache2/error.log:

[Sun Aug 15 08:02:48.276131 2021] [wsgi:error] [pid 365:tid 139642783385344] [client 141.45.247.245:36182] 2021-08-15 08:02:48,276 - inginious.course_factory - INFO - Caching course tutorial, referer: http://inginious-ikg.f4.htw-berlin.de/course/tutorial/01_getting_started
[Sun Aug 15 08:02:48.308092 2021] [wsgi:error] [pid 365:tid 139642196190976] 2021-08-15 08:02:48,308 - inginious.backend - INFO - Adding a new job b'\\x005G(\\t' 7fbf4d45-82c5-4131-95e1-e0f42a828541 to the queue
[Sun Aug 15 08:02:48.308369 2021] [wsgi:error] [pid 365:tid 139642783385344] [client 141.45.247.245:36182] 2021-08-15 08:02:48,308 - inginious.webapp.submissions - INFO - New submission from superadmin - [email protected] - tutorial/01_getting_started - 141.45.247.245, referer: http://inginious-ikg.f4.htw-berlin.de/course/tutorial/01_getting_started
[Sun Aug 15 08:02:48.308777 2021] [wsgi:error] [pid 365:tid 139642196190976] 2021-08-15 08:02:48,308 - inginious.backend - INFO - Sending job b'\\x005G(\\t' 7fbf4d45-82c5-4131-95e1-e0f42a828541 to agent b'\\x00\\xd8O\\xccq'
[Sun Aug 15 08:02:48.309375 2021] [wsgi:error] [pid 365:tid 139642196190976] 2021-08-15 08:02:48,309 - inginious.agent.docker - INFO - Received request for jobid 7fbf4d45-82c5-4131-95e1-e0f42a828541
[Sun Aug 15 08:03:18.712833 2021] [wsgi:error] [pid 365:tid 139642196190976] 2021-08-15 08:03:18,712 - inginious.agent.docker - INFO - Killing container 0c3c6eeb557ee9f3656a80ee2cc29aab3afde1fe65b4330161be3d5e47aba622 as it used its 30 wall time seconds
[Sun Aug 15 08:03:18.869877 2021] [wsgi:error] [pid 365:tid 139642196190976] 2021-08-15 08:03:18,869 - inginious.agent.docker - WARNING - Container 0c3c6eeb557ee9f3656a80ee2cc29aab3afde1fe65b4330161be3d5e47aba622 has not given any result
[Sun Aug 15 08:03:18.881641 2021] [wsgi:error] [pid 365:tid 139642196190976] 2021-08-15 08:03:18,881 - inginious.backend - INFO - Job 7fbf4d45-82c5-4131-95e1-e0f42a828541 finished on agent b'\\x00\\xd8O\\xccq'

Any tips would be greatly appreciated!

anthonygego · 2021-08-16T13:28:52Z

anthonygego
Aug 16, 2021
Maintainer

Are you able to run any other container ? For instance a simple CentOS container using docker run -it centos /bin/bash

Is cgroup actually mounted ? Is there any folder like /sys/fs/cgroup in your filesystem ? Things probably won't work without cgroups.

As the timeout actually happens after the 30secs, you can see if it prints some logs by running docker logs on the running container that you can find by typing docker ps in your terminal.

2 replies

dabrowskiw Aug 16, 2021
Author

Thanks for the quick reply!

Are you able to run any other container ? For instance a simple CentOS container using docker run -it centos /bin/bash

Yes, this works:

root@inginious-ikg:/home/local# docker run -it centos /bin/bash
Unable to find image 'centos:latest' locally
latest: Pulling from library/centos
7a0437f04f83: Pull complete 
Digest: sha256:5528e8b1b1719d34604c87e11dcd1c0a20bedf46e83b5632cdeac91b8c04efc1
Status: Downloaded newer image for centos:latest
[root@054ddfc15b05 /]# cat /etc/centos-release
CentOS Linux release 8.3.2011

Is cgroup actually mounted ? Is there any folder like /sys/fs/cgroup in your filesystem ? Things probably won't work without cgroups.

I don't really know what cgroup is, but the folder exists - does this look correct?

root@inginious-ikg:/home/local# ls -lasih /sys/fs/cgroup/
total 0
11466 0 drwxr-xr-x 14 root root 360 Aug  15 07:05 .
    2 0 drwxr-xr-x  6 root root   0 Aug  15 07:05 ..
    1 0 dr-xr-xr-x  5 root root   0 Aug  15 07:05 blkio
11490 0 lrwxrwxrwx  1 root root  11 Aug  15 07:05 cpu -> cpu,cpuacct
    1 0 dr-xr-xr-x  5 root root   0 Aug  15 07:05 cpu,cpuacct
11489 0 lrwxrwxrwx  1 root root  11 Aug  15 07:05 cpuacct -> cpu,cpuacct
    1 0 dr-xr-xr-x  3 root root   0 Aug  15 07:05 cpuset
    1 0 dr-xr-xr-x  5 root root   0 Aug  15 07:05 devices
    1 0 dr-xr-xr-x  3 root root   0 Aug  15 07:05 freezer
    1 0 dr-xr-xr-x  5 root root   0 Aug  15 07:05 memory
11496 0 lrwxrwxrwx  1 root root  16 Aug  15 07:05 net_cls -> net_cls,net_prio
    1 0 dr-xr-xr-x  3 root root   0 Aug  15 07:05 net_cls,net_prio
11495 0 lrwxrwxrwx  1 root root  16 Aug  15 07:05 net_prio -> net_cls,net_prio
    1 0 dr-xr-xr-x  3 root root   0 Aug  15 07:05 perf_event
    1 0 dr-xr-xr-x  5 root root   0 Aug  15 07:05 pids
    1 0 dr-xr-xr-x  2 root root   0 Aug  15 07:05 rdma
    1 0 dr-xr-xr-x  6 root root   0 Aug  15 07:05 systemd
    1 0 dr-xr-xr-x  5 root root   0 Aug  15 07:05 unified

As the timeout actually happens after the 30secs, you can see if it prints some logs by running docker logs on the running container that you can find by typing docker ps in your terminal.

I got the logs, but they seem... moderately informative. Is this helpful in any way?

root@inginious-ikg:/home/local# docker ps                                                      
CONTAINER ID   IMAGE          COMMAND       CREATED         STATUS         PORTS     NAMES
c6793d028baa   862b05613c03   "INGInious"   3 seconds ago   Up 2 seconds             upbeat_lehmann
root@inginious-ikg:/home/local# docker logs c6793d028baa
2021-08-16 16:57:18,516 - inginious.container - INFO - Hello
2021-08-16 16:57:18,516 - inginious.container - INFO - Starting serve
2021-08-16 16:57:18,517 - inginious.container - INFO - Serving...
root@inginious-ikg:/home/local#

I looked at the logs every second, and the content doesn't change until the container gets killed (at which point it's just Error: No such container: c6793d028baa).

Is any of this helpful in any way?

anthonygego Aug 18, 2021
Maintainer

This is good. The communication probably doesn't work then. As @darkyne suggested, check that your containers are built using the files corresponding to the installed version. If this is OK and still doesn't work, also check if SELinux is actually disabled.

darkyne · 2021-08-17T09:56:22Z

darkyne
Aug 17, 2021

Hello,

What code have you entered as inputs for the task?
I suppose it's just
print("Hello World!")
but might as well ask.

Last time someone faced this kind of issue, his INGInious installation used old versions of the base-container: https://github.com/UCL-INGI/INGInious/tree/master/base-containers/base
Maybe you could check if these files are up to date on your version?
His logs were very similar to yours with the container's logs ending with "serving" and never going into the next step ( should be "received message start") and ending with timeout.

If these files are outdated on your version, you will have to fetch new version and rebuild images with some commands, some thing like:
cd base-containers/base && docker build -t ingi/inginious-c-base ./

I hope this may help.

10 replies

darkyne Sep 30, 2021

Ok what I suggest you to have a clean version:
Remove all the images:
docker rmi -f <image_id>
After that, I suggest you to pip3 uninstall inginious (installation via pip is temporarly broken because of a containers versions issue)
Then you can clone the repo in a nice derectory:
git clone https://github.com/UCL-INGI/INGInious.git
In the repo directory, run:
./inginious-install
And you should have everything up to date and running.
Please after that, show again your result for:
docker image ls

Sounds like a lot but it should take you only few minutes :)

dabrowskiw Sep 30, 2021
Author

Thanks, done!

However, could it be that there are some other docker-related config files I somehow need to regenerate? I now get "Cannot create container" in the web UI, and the error in /var/log/apache2/error.log is:

[Thu Sep 30 09:19:12.455578 2021] [wsgi:error] [pid 27942:tid 139643848734464] [client 141.45.246.190:59742] 2021-09-30 09:19:12,455 - inginious.webapp.submissions - INFO - New submission from superadmin - [email protected] - tutorial/01_getting_started - 141.45.246.190, referer: http://inginious-ikg.f4.htw-berlin.de/course/tutorial/01_getting_started
[Thu Sep 30 09:19:12.456611 2021] [wsgi:error] [pid 27942:tid 139642196190976] 2021-09-30 09:19:12,456 - inginious.backend - INFO - Adding a new job b'\\x00,\\xadi\\xa0' d71f2a98-bdff-4dd6-b0c4-4fae86796827 to the queue
[Thu Sep 30 09:19:12.457211 2021] [wsgi:error] [pid 27942:tid 139642196190976] 2021-09-30 09:19:12,457 - inginious.backend - INFO - Sending job b'\\x00,\\xadi\\xa0' d71f2a98-bdff-4dd6-b0c4-4fae86796827 to agent b'\\x00=UGy'
[Thu Sep 30 09:19:12.457760 2021] [wsgi:error] [pid 27942:tid 139642196190976] 2021-09-30 09:19:12,457 - inginious.agent.docker - INFO - Received request for jobid d71f2a98-bdff-4dd6-b0c4-4fae86796827
[Thu Sep 30 09:19:12.611179 2021] [wsgi:error] [pid 27942:tid 139641894184704] 2021-09-30 09:19:12,479 - inginious.agent.docker - WARNING - Cannot create container! 404 Client Error for http+docker://localhost/v1.41/containers/create: Not Found ("No such image: sha256:862b05613c035c926c8d0c738879a7d23d3b38ece4865f449111e33fc32e2383")
[Thu Sep 30 09:19:12.611203 2021] [wsgi:error] [pid 27942:tid 139641894184704] Traceback (most recent call last):
[Thu Sep 30 09:19:12.611208 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 268, in _raise_for_status
[Thu Sep 30 09:19:12.611214 2021] [wsgi:error] [pid 27942:tid 139641894184704]     response.raise_for_status()
[Thu Sep 30 09:19:12.611236 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/requests/models.py", line 953, in raise_for_status
[Thu Sep 30 09:19:12.611241 2021] [wsgi:error] [pid 27942:tid 139641894184704]     raise HTTPError(http_error_msg, response=self)
[Thu Sep 30 09:19:12.611246 2021] [wsgi:error] [pid 27942:tid 139641894184704] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/create
[Thu Sep 30 09:19:12.611251 2021] [wsgi:error] [pid 27942:tid 139641894184704] 
[Thu Sep 30 09:19:12.611255 2021] [wsgi:error] [pid 27942:tid 139641894184704] During handling of the above exception, another exception occurred:
[Thu Sep 30 09:19:12.611260 2021] [wsgi:error] [pid 27942:tid 139641894184704] 
[Thu Sep 30 09:19:12.611264 2021] [wsgi:error] [pid 27942:tid 139641894184704] Traceback (most recent call last):
[Thu Sep 30 09:19:12.611269 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/inginious/agent/docker_agent/__init__.py", line 362, in __new_job_sync
[Thu Sep 30 09:19:12.611274 2021] [wsgi:error] [pid 27942:tid 139641894184704]     course_common_student_path, runtime, ports)
[Thu Sep 30 09:19:12.611279 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/inginious/agent/docker_agent/_docker_interface.py", line 143, in create_container
[Thu Sep 30 09:19:12.611284 2021] [wsgi:error] [pid 27942:tid 139641894184704]     runtime=runtime
[Thu Sep 30 09:19:12.611288 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/docker/models/containers.py", line 870, in create
[Thu Sep 30 09:19:12.611299 2021] [wsgi:error] [pid 27942:tid 139641894184704]     resp = self.client.api.create_container(**create_kwargs)
[Thu Sep 30 09:19:12.611304 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/docker/api/container.py", line 430, in create_container
[Thu Sep 30 09:19:12.611308 2021] [wsgi:error] [pid 27942:tid 139641894184704]     return self.create_container_from_config(config, name)
[Thu Sep 30 09:19:12.611312 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/docker/api/container.py", line 441, in create_container_from_config
[Thu Sep 30 09:19:12.611317 2021] [wsgi:error] [pid 27942:tid 139641894184704]     return self._result(res, True)
[Thu Sep 30 09:19:12.611321 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 274, in _result
[Thu Sep 30 09:19:12.611325 2021] [wsgi:error] [pid 27942:tid 139641894184704]     self._raise_for_status(response)
[Thu Sep 30 09:19:12.611330 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 270, in _raise_for_status
[Thu Sep 30 09:19:12.611334 2021] [wsgi:error] [pid 27942:tid 139641894184704]     raise create_api_error_from_http_exception(e)
[Thu Sep 30 09:19:12.611338 2021] [wsgi:error] [pid 27942:tid 139641894184704]   File "/usr/local/lib/python3.7/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
[Thu Sep 30 09:19:12.611342 2021] [wsgi:error] [pid 27942:tid 139641894184704]     raise cls(e, response=response, explanation=explanation)
[Thu Sep 30 09:19:12.611347 2021] [wsgi:error] [pid 27942:tid 139641894184704] docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/containers/create: Not Found ("No such image: sha256:862b05613c035c926c8d0c738879a7d23d3b38ece4865f449111e33fc32e2383")
[Thu Sep 30 09:19:12.612878 2021] [wsgi:error] [pid 27942:tid 139642196190976] 2021-09-30 09:19:12,612 - inginious.backend - INFO - Job d71f2a98-bdff-4dd6-b0c4-4fae86796827 finished on agent b'\\x00=UGy'

Apparently, inginious is looking for the old container - the first part of the hash in the error message Not Found ("No such image: sha256:862b05613c035c926c8d0c738879a7d23d3b38ece4865f449111e33fc32e2383") exactly fits the ID 862b05613c03 of the old ingi/inginious-c-default image. However, now docker image ls says:

REPOSITORY                 TAG       IMAGE ID       CREATED          SIZE
ingi/inginious-c-java      latest    9c81f0cb43a8   13 minutes ago   1.3GB
ingi/inginious-c-python3   latest    102eb295374f   16 minutes ago   692MB
ingi/inginious-c-default   latest    fea743fbdace   18 minutes ago   692MB
ingi/inginious-c-base      latest    e4d03d4b307b   18 minutes ago   692MB
centos                     7         eeb6ee3f44bd   2 weeks ago      204MB

How do I tell inginious to use the images it created during the re-installation instead of the old ones?

darkyne Sep 30, 2021

Are you facing this issue when simply running ./inginious-webapp ?

dabrowskiw Sep 30, 2021
Author

Argh, thanks, that was the problem.

Just to be sure that there are no stale services running, I have now rebooted the whole system. However, I now get the "timed out" error again, with the following in /var/log/daemon.log:

Sep 30 09:42:43 inginious-ikg dockerd[452]: time="2021-09-30T09:42:43.552885332Z" level=warning msg="Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap."
Sep 30 09:42:43 inginious-ikg systemd[1]: var-lib-docker-overlay2-794285f1de5de63ee24bde48f3f43ea7e63b75b13b55caba39fe2786646c2ca5\x2dinit-merged.mount: Succeeded.
Sep 30 09:42:43 inginious-ikg containerd[339]: time="2021-09-30T09:42:43.659224287Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/e0489fbc100459e78f75f96a467ac86ccb647913abdaff0c481598a890632b95 pid=1166
Sep 30 09:42:53 inginious-ikg dockerd[452]: time="2021-09-30T09:42:53.033073709Z" level=info msg="ignoring event" container=e0489fbc100459e78f75f96a467ac86ccb647913abdaff0c481598a890632b95 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 30 09:42:53 inginious-ikg containerd[339]: time="2021-09-30T09:42:53.033168317Z" level=info msg="shim disconnected" id=e0489fbc100459e78f75f96a467ac86ccb647913abdaff0c481598a890632b95
Sep 30 09:42:53 inginious-ikg containerd[339]: time="2021-09-30T09:42:53.036158338Z" level=error msg="copy shim log" error="read /proc/self/fd/12: file already closed"
Sep 30 09:42:53 inginious-ikg systemd[1]: run-docker-netns-2ad610100432.mount: Succeeded.
Sep 30 09:42:53 inginious-ikg systemd[1]: var-lib-docker-overlay2-794285f1de5de63ee24bde48f3f43ea7e63b75b13b55caba39fe2786646c2ca5-merged.mount: Succeeded.

and the following in /var/log/apache2/error.log:

[Thu Sep 30 09:42:43.507952 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:42:43,507 - inginious.backend - INFO - Adding a new job b'\\x00\\x00\\xbe.\\x12' 435ffa05-450b-4893
-88c6-d2aefc91a684 to the queue                                                                                                                                                               [Thu Sep 30 09:42:43.508132 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:42:43,508 - inginious.backend - INFO - Sending job b'\\x00\\x00\\xbe.\\x12' 435ffa05-450b-4893-88c6
-d2aefc91a684 to agent b'\\x00@6\\x15\\xb0'                                                                                                                                                   [Thu Sep 30 09:42:43.508798 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:42:43,508 - inginious.agent.docker - INFO - Received request for jobid 435ffa05-450b-4893-88c6-d2ae
fc91a684                                                                                                                                                                                      [Thu Sep 30 09:42:43.529967 2021] [wsgi:error] [pid 375:tid 140253817984768] [client 141.45.246.190:59876] 2021-09-30 09:42:43,529 - inginious.webapp.submissions - INFO - New submission from
 superadmin - [email protected] - tutorial/01_getting_started - 141.45.246.190, referer: http://inginious-ikg.f4.htw-berlin.de/course/tutorial/01_getting_started                 [Thu Sep 30 09:42:52.935165 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:42:52,935 - inginious.agent.docker - INFO - Killing container e0489fbc100459e78f75f96a467ac86ccb647
913abdaff0c481598a890632b95 as it used its 9 wall time seconds                                                                                                                                [Thu Sep 30 09:42:53.099001 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:42:53,098 - inginious.agent.docker - WARNING - Container e0489fbc100459e78f75f96a467ac86ccb647913ab
daff0c481598a890632b95 has not given any result                                                                                                                                               [Thu Sep 30 09:42:53.108939 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:42:53,108 - inginious.backend - INFO - Job 435ffa05-450b-4893-88c6-d2aefc91a684 finished on agent b
'\\x00@6\\x15\\xb0'

I have then increased the timeout to 30 seconds from 3 seconds, and now the process just hangs (it doesn't give a result, but it doesn't terminate with a timeout error either), with the following in /var/log/daemon.log:

Sep 30 09:46:33 inginious-ikg dockerd[452]: time="2021-09-30T09:46:33.439872859Z" level=warning msg="Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap."
Sep 30 09:46:33 inginious-ikg systemd[1]: var-lib-docker-overlay2-c4ac2185b135f32e6012d19e540e83fdde7b13abcf1eb7f5ba1281a8e2aa319e\x2dinit-merged.mount: Succeeded.
Sep 30 09:46:33 inginious-ikg systemd-udevd[1459]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 30 09:46:33 inginious-ikg systemd-udevd[1459]: Using default interface naming scheme 'v240'.
Sep 30 09:46:33 inginious-ikg systemd-udevd[1459]: Could not generate persistent MAC address for veth56e6419: No such file or directory
Sep 30 09:46:33 inginious-ikg systemd-udevd[1460]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 30 09:46:33 inginious-ikg systemd-udevd[1460]: Using default interface naming scheme 'v240'.
Sep 30 09:46:33 inginious-ikg systemd-udevd[1460]: Could not generate persistent MAC address for veth09b752f: No such file or directory
Sep 30 09:46:33 inginious-ikg containerd[339]: time="2021-09-30T09:46:33.706649908Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/b8bd1566af53386f77cfa62f5b9929f6c23d64fe2bd2d8a3c8492ad82f9bd8cd pid=1493

and the following in /var/log/apache2/error.log:

[Thu Sep 30 09:46:33.415616 2021] [wsgi:error] [pid 375:tid 140253960595200] [client 141.45.246.190:59896] 2021-09-30 09:46:33,415 - inginious.webapp.submissions - INFO - New submission from superadmin - [email protected] - tutorial/01_getting_started - 141.45.246.190, referer: http://inginious-ikg.f4.htw-berlin.de/course/tutorial/01_getting_started
[Thu Sep 30 09:46:33.416235 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:46:33,415 - inginious.backend - INFO - Sending job b'\\x00\\x00\\xbe.\\x12' f2b939d5-c7f4-41e5-8e6d-92aa9ce1998a to agent b'\\x00@6\\x15\\xb0'
[Thu Sep 30 09:46:33.417339 2021] [wsgi:error] [pid 375:tid 140251754395392] 2021-09-30 09:46:33,417 - inginious.agent.docker - INFO - Received request for jobid f2b939d5-c7f4-41e5-8e6d-92aa9ce1998a

So... it's apparently back to square 1?

Sorry :$

darkyne Sep 30, 2021

Damn !
Well, at least now you should have the latest images...
Still not sure what is causing your issue, looking more into this soon.

Submissions (in "[TUTO] The INGInious tutorial" course) timing out, CPU at 0%? #696

Uh oh!

Uh oh!

dabrowskiw Aug 15, 2021

Replies: 2 comments · 12 replies

Uh oh!

anthonygego Aug 16, 2021 Maintainer

Uh oh!

dabrowskiw Aug 16, 2021 Author

Uh oh!

anthonygego Aug 18, 2021 Maintainer

Uh oh!

Uh oh!

darkyne Aug 17, 2021

Uh oh!

Uh oh!

darkyne Sep 30, 2021

Uh oh!

dabrowskiw Sep 30, 2021 Author

Uh oh!

Uh oh!

darkyne Sep 30, 2021

Uh oh!

Uh oh!

dabrowskiw Sep 30, 2021 Author

Uh oh!

darkyne Sep 30, 2021

dabrowskiw
Aug 15, 2021

Replies: 2 comments 12 replies

anthonygego
Aug 16, 2021
Maintainer

dabrowskiw Aug 16, 2021
Author

anthonygego Aug 18, 2021
Maintainer

darkyne
Aug 17, 2021

dabrowskiw Sep 30, 2021
Author

dabrowskiw Sep 30, 2021
Author