@avonbertoldi I'll look into this. I wish there was a full gRPC interface. That way we could construct a gRPC request on KAS and pass it all the way to the step-runner without having the encode / decode. Currently, we have to do KAS -> proto -> runner -> YAML -> step-runner client -> proto (RunRequest) -> step-runner -> ... and back - which seems very inefficient.

Looking at the runner (arguably, the only step-runner user atm) I don't understand why to abstract proto at all. Why is that?

I've had Duo summarize what primitives other platforms are building and what we should focus on, let me know if this is useful.

@gabrielengel_gl I've looked at it and I'm not sure what exactly I should take away from it.

The three core primitives plus on_event and the replay engine cover both MVC use cases (K8s deployment and AI Agent flow). OPA is the key differentiator and should follow immediately after the core engine works.

These two statements are so overly simplified that I would argue that they are mostly noise at this point. How come the three primitives cover Kubernetes deployments and AI Agent flows? There is soooo much more to these two use cases from a product and also implementation perspective. The OPA part is also quite hyped in this statement - but why is that? Do you agree?

Why isn't OPA just another run? Should it be?

Same with this statement:

AutoFlow's OPA policy layer is uniquely built-in where every other platform requires you to hand-roll approval/compliance gates.

Point being, I think we need to deeply understand what the user journey for AutoFlow should look like. Lets take these questions as examples:

• How does a user write AutoFlow workflows?
• Where does a user persist and manage those workflows?
• Who can do it?
• How are events connected to AutoFlow workflows?
• Can an event trigger multiple AutoFlow workflows?
• Does AutoFlow have (configurable) hardcoded top-level AutoFlow workflows for events that itself spawn user-defined child workflows (you can see this here: https://excalidraw.com/#json=4plyrs1Ilb2gTY1rYR0dK,FW_G2Zrsl4NR-HXROBdjcQ)
• What triggers AutoFlow in the first place? Do we have eventing guarantees, e.g. exactly once delivery?
• How are AutoFlow workflows and activities retried?
• How do running AutoFlow workflows surface to the user?
  • For the CD system: how does it conceptionally (and UI/UX-wise) integrate with GitLab pipelines?
  • Can a user "observe" running workflows / activities?
• there are plenty like this 😄

I think we need to ask these questions to be able to determine the criteria for what primitives / concepts we need and how to combine them into a coherent system. And it'll be crucial to understand these concepts and what tradeoffs come with them.

I think we need to collect and formalize these questions and their answers so that we can work out an MVC. Would that help? WDYT?

@gabrielengel_gl I agree. We've used job routing because we have a much clearer picture for it then CD, additionally, since for CD we need want to run GitLab Functions on a Runner - we kinda need to go through "a" job router anyway - or at least end up on a runner without conflicting with the existing job scheduling in Rails. There is some natural interdependency there.

Timo Furrer (13e8be60) at 17 Mar 15:55

... and 12 more commits

Timo Furrer (4a89cf65) at 17 Mar 15:40

For posterity: @cam_swords and I had a sync chat and we discovered (or rather I got to learn about this from him) that we can delegate outputs. This is what I'm going to do from the runner and in here we simply expose the top-level outputs.

Timo Furrer (5f035a1c) at 17 Mar 15:36

Timo Furrer (52f49e00) at 17 Mar 15:35

@brendan777 I think it won't - at least not from a CLI runtime perspective. This really only affects the markdown files for the markdown docs. LGTM!

I don't know why anyone needs env (tell me otherwise!), and there is no guarantee that a function even has an exit code, as these only apply to functions that are executable - there are other function types.

@cam_swords fair enough - I just noticed that these are sometimes available and mentioned them. Really, this is about outputs primarily.

So with status returning Status (success/failure/running/cancelled), Outputs (top-level only - the function you ran), start/end time, and FollowLogs getting you the logs, is there anything missing that you need?

I think there is a misunderstanding here. The top-level function (that gets passed to the step-runner as YAML) is a nameless step dynamically created in the runner wrapping the actual step I want to run. That dynamically created step doesn't have outputs - because we don't know them. This code already exists here. That's a problem, because I want the output from that single wrapped step. Do I understand the current implementation wrongly? Or is it not used correctly from the runner?

@ash2k and I had a sync meeting today, here is a summary of what we've discussed:

• We need to have a closer look into how DBOS, river and Temporal work under the head - especially w.r.t the database. We've learnt (gitlab-org/cluster-integration/gitlab-agent#818 (comment 3157001919)) that the database plays a crucial role long term. We might want to use one of them to bootstrap AutoFlow, but eventually we might need a more integrated / custom solution (without overfitting!).
• How to deploy KAS: multiple deployments with per-deployment enabled modules? Keep a single deployment? How about cross-instance job router - e.g. sharing runners between GitLab.com and Dedicated (which is an idea that was floating around)
• Discussed how job routing could look like with AutoFlow. This is what we doodled during the meeting: https://excalidraw.com/#json=4plyrs1Ilb2gTY1rYR0dK,FW_G2Zrsl4NR-HXROBdjcQ. I'll take same time later to write this up properly. CD events (and other) would be processed in a similar way.
• What initial primitives / building blocks do we need? We already know about the "essentials" work durable workflow execution, like Workflows and Activities. What about queues? (mentioning those because they come up a lot)

I'm going to create some issues later to track these things more closely.

cc @adebayo_a

@cam_swords okay, interesting. Okay, so HOW can I access the top-level outputs (you called it a "nameless step") in that example without knowing what the steps outputs are?

Of course doing something forwarding like in your previous example:

spec:
  outputs:
    deployed_to:
      type: string
---
run:
  - name: release
    func: registry.gitlab.com/funcs/release
    inputs:
      artifact: ${{ steps.build.outputs.artifact }}
outputs:
  deployed_to: ${{ steps.release.outputs.environment }}

however, in a generic situation I don't know what outputs to forward.

I think the step-runner gRPC would benefit from an API to just run a function directly given it's "run spec" and it returns it's outputs, env is set, exit codes, logs etc. - but that's another topic.

What would you recommend how I capture the outputs of e.g. that "release" step in the previous example without the caller knowing that an output called environment even exists?

As it turns out, we can't really do this because it changes the type of the field.

What does this MR do?

Revert "Merge branch 'malvarez-consolidate-http-status-code-field' into 'main'"

This reverts merge request !6492

Why was this MR needed?

The previous changes changed the type of the logging field status from String to Integer. This was pointed out to have the risk of causing type conflicts: Slack (internal) & https://opster.com/guides/elasticsearch/glossary/elasticsearch-conflicting-field/

We are reverting the changes to first figure out the necessary type mappings.

What's the best way to test this MR?

What are the relevant issue numbers?

@cam_swords thanks for sharing this example, because I was missing the top-level outputs and was directly accessing the sub-steps outputs.

spec:
  outputs:
    deployed_to:
      type: string
---
run:
  - name: build
    func: registry.gitlab.com/funcs/build
  - name: release
    func: registry.gitlab.com/funcs/release
    inputs:
      artifact: ${{ steps.build.outputs.artifact }}
outputs:
  deployed_to: ${{ steps.release.outputs.environment }}

I basically created a step in the runner like this:

	name := "function"
	step := schema.Step{
		Name:   &name,
		Func:   req.FunctionRef,
		Env:    req.Env,
		Inputs: toStepInputs(req.Inputs),
	}

And I think the problem is that when I run this in the runner today, the runner wraps it in another Step, see https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/steps/steps.go#L68. And this "wrapping" causes the empty outputs on that step and therefore I had to access the outputs from the sub step (which is the one I actually want to run). Do you know why that "wrapping" step even exists? 🤔

Would it help if functions had control flow? e.g.

Unfortunately, not. Well, users might use that "function control flow" if they like. Yet, we still need to run individual functions from KAS - no way around that.

Timo Furrer (1d3ac641) at 17 Mar 10:38

Timo Furrer (be90bf32) at 17 Mar 10:38

... and 1 more commit

What does this MR do and why?

Implements DeleteWorkItem to support the workItemDelete GraphQL mutation. The method accepts fullPath and iid, internally queries for the Global ID, then executes the deletion mutation.

Also adds cleanup to gitlab_test/utils_test.go for TestCreateWorkItem() integration test.

Related Issues

#2220

This is blocked by huhtest, that only supports huh v1.

What does this MR do and why?

Implements DeleteWorkItem to support the workItemDelete GraphQL mutation. The method accepts fullPath and iid, internally queries for the Global ID, then executes the deletion mutation.

Also adds cleanup to gitlab_test/utils_test.go for TestCreateWorkItem() integration test.

Related Issues

#2220