Without deep expertise in this area, I wanted to flag how Cloudflare built their Workflows product: https://developers.cloudflare.com/workflows/

I'm not suggesting we adopt Cloudflare Workflows directly, but the underlying architecture is interesting. It's built on V8 isolates/workers and Durable Objects. DOs as a primitive seem to allow you to solve a lot of problems flexibly.

@cam_swords This seems okay, but I'm wondering if there's an earlier entrypoint too? I guess between Serve() getting a job and us executing it? That way, if a panic occurs, it only affects that client. And that protects on anything between that and function execution.

@ash2k @avonbertoldi This has broken at least the Windows instance runners.

instance.NewProvider(runnerCommand)

Previously, I suspect the runner command here would have just been gitlab-runner, because it's up to the user to provide it in the remote environment's $PATH. The instance runner can target Windows and Linux environments from Windows or Linux, so hard-coding the path for the remote environment here, based on the runner manager, is an accidental breaking change (as confirmed by our pilot runners that deployed this today).

Similarly, docker-windows and docker used to be separate, but a design several years ago made them entirely equal, and we made any OS differences (other than those in the abstract shell) be driven by inspecting the Docker daemon to determine the OS. The change here suggests that steps isn't enabled when docker-windows is used, which might break anybody running steps for Linux, but from a Windows-based runner manager.

Commenting here because it means I can directly address the responsible code and in case this has been independently discovered and you're already working on it. If not, we should resolve this to an issue and fix 👍

@GitLabDuo Can you perform a whole review of this MR? With the persona of a private detective.

Odd that GitLab Duo fails to review the whole MR (presumably because of the tag / pilot runner issue?), but asking it to perform a review via a message doesn't.

@GitLabDuo Can you review just commands/steps/steps.go?

Arran Walker (6132ce70) at 17 Mar 14:47

... and 25 more commits

This is a copy of the existing Dockerfile. I would have just used Dockerfile, but I had to include an entrypoint for our prebuilt container tooling to work 🤷

@avonbertoldi @cam_swords Are you able to review this?

@avonbertoldi You're familiar with the underlying base image.

@cam_swords you're familiar with the bootstrapping process: It's now extended to copy more than just the helper binary.

HelperEnvs is used to wrap the environment variables of any command that would previously be executed in the helper container, ensuring we setup our bundled CA certs and our bundled version of Git (just as would have been the case in the helper container).

Arran Walker (91e13115) at 17 Mar 14:01

Arran Walker (b55597a3) at 17 Mar 13:58

... and 24 more commits

Arran Walker (ed674f5d) at 17 Mar 13:57

... and 23 more commits

Arran Walker (417d0ee1) at 17 Mar 12:00

... and 8 more commits

Arran Walker (4bb6aebb) at 17 Mar 12:00

... and 7 more commits

Arran Walker (1e9b8250) at 17 Mar 10:24

Arran Walker (dea99250) at 17 Mar 10:00

Arran Walker (7a2ee93a) at 17 Mar 09:04

Arran Walker (2b542ab7) at 17 Mar 08:31

Arran Walker (7f0c13b7) at 17 Mar 08:11

Arran Walker (7dab695e) at 17 Mar 08:08