Leaminn Ma activity https://gitlab.com/lma-git 2026-03-17T23:05:42Z tag:gitlab.com,2026-03-17:5215095549 Leaminn Ma commented on merge request !226892 at GitLab.org / GitLab 2026-03-17T23:04:08Z lma-git Leaminn Ma

@pedropombeiro BE & devopsverify LGTM. Just left a nit.

tag:gitlab.com,2026-03-17:5215095545 Leaminn Ma approved merge request !226892: Add organization_id filtering to Ci::RunnersFinder at GitLab.org / GitLab 2026-03-17T23:04:08Z lma-git Leaminn Ma

What does this MR do and why?

Scopes user-scoped runner listings by the user's organizations so that multi-org users only see runners from organizations they belong to. The finder additionally accepts an optional organization_id parameter to narrow results to a specific organization.

Problem: The user-scoped path (User#ci_available_runners via GET /runners, currentUser { runners }) unions runners from all groups/projects the user has access to across all organizations. Once a user has multi-org membership, this query returns runners from every organization where they hold the required role -- with no filtering by organization.

Paths not affected:

  • Admin paths (Ci::Runner.all): Return all runners on the cell -- correct by design (admins are cell-scoped)
  • Project/group-scoped paths: Safe by construction -- runners are derived through hierarchy joins scoped to a single group/project, and access is guarded by declarative policy checks (can?(@current_user, :read_runners, @group/@project)) which verify the user has the required role on that specific resource

Design: two-layer organization filtering

Organization filtering is applied in two layers as a defense-in-depth measure:

  1. Model layer (User#ci_available_runners): Always scopes results to runners belonging to the user's own organizations via .in_organization(organizations.pluck(:id)). The pluck resolves org IDs on the main DB first, avoiding cross-database subqueries. This ensures a user can never see runners from an organization they don't belong to, regardless of what the caller passes.

  2. Finder layer (Ci::RunnersFinder#user_runners): When an organization_id parameter is provided (e.g. from Current.organization), narrows results further to that specific organization. This is the presentation filter for "show me runners in the current org context."

The model layer acts as a security boundary, while the finder layer provides the UX-level scoping. Even if a caller passes a malicious organization_id, the model layer prevents data leakage.

Changes

  • app/models/ci/runner.rb -- Add .in_organization(organization_id) scope. Accepts a scalar ID or an array. Instance runners (which have NULL organization_id by design) are naturally excluded by the WHERE clause
  • app/models/user.rb -- ci_available_runners now chains .in_organization(organizations.pluck(:id)) to scope results to the user's own organizations. Uses pluck to resolve IDs on main DB, avoiding cross-database subqueries with the CI DB
  • app/finders/ci/runners_finder.rb -- Add user_runners method that optionally narrows results further by organization_id param when provided

Database queries

Since ci_runners lives on the CI database while organizations/organization_users live on the main database, organizations.pluck(:id) is resolved on the main DB first. The resulting array of org IDs is then passed as literals to the CI DB query. Normally, this will be a single value, since users normally belong to a single organization.

1. Original query -- no org filter (query plan):

SELECT COUNT(*) FROM ((SELECT "ci_runners".* FROM "ci_runners"
  INNER JOIN "ci_runner_projects" ON "ci_runner_projects"."runner_id" = "ci_runners"."id"
  WHERE "ci_runner_projects"."project_id" = ANY ('{...}'::integer[]))
UNION
(WITH "cte_namespace_ids" AS MATERIALIZED (
  SELECT "ci_namespace_mirrors"."namespace_id" FROM "ci_namespace_mirrors"
  WHERE ((traversal_ids[1]) = ANY ('{36425,36427,5268246}'::integer[])))
SELECT "ci_runners".* FROM "ci_runners"
  INNER JOIN "ci_runner_namespaces" ON "ci_runner_namespaces"."runner_id" = "ci_runners"."id"
  WHERE ci_runner_namespaces.namespace_id IN (SELECT namespace_id FROM cte_namespace_ids)))
ci_runners;

2. Model layer -- scoped to user's organizations (query plan):

SELECT COUNT(*) FROM (...same union...) ci_runners
WHERE "ci_runners"."organization_id" IN (<user's org IDs from main DB>);

3. Both layers -- model + finder organization_id param (query plan):

SELECT COUNT(*) FROM (...same union...) ci_runners
WHERE "ci_runners"."organization_id" IN (<user's org IDs>) AND "ci_runners"."organization_id" = 1;

References

Screenshots or screen recordings

Not applicable -- backend service change only.

How to set up and validate locally

  1. Open a Rails console and create test data:
    org_a = Organizations::Organization.first
    org_b = create(:organization)
    group_a = create(:group, organization: org_a)
    group_b = create(:group, organization: org_b)
    runner_a = create(:ci_runner, :group, groups: [group_a])
    runner_b = create(:ci_runner, :group, groups: [group_b])
    user = create(:user, owner_of: [group_a, group_b])
  2. Verify the model scopes to the user's organizations:
    user.ci_available_runners.to_sql
    # => includes WHERE organization_id IN (1, <org_b.id>)
  3. Verify the finder narrows by organization_id:
    Ci::RunnersFinder.new(current_user: user, params: { user: user, organization_id: org_a.id }).execute
    # => returns only runner_a
  4. Run the specs:
    bundle exec rspec spec/models/ci/runner_spec.rb -e 'in_organization' --format documentation
    bundle exec rspec spec/finders/ci/runners_finder_spec.rb -e 'organization' --format documentation

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Changelog: other

tag:gitlab.com,2026-03-17:5215095538 Leaminn Ma commented on merge request !226892 at GitLab.org / GitLab 2026-03-17T23:04:08Z lma-git Leaminn Ma

Nit / Non-blocking:

We could use other_organization defined in the outer context since it already exists.

tag:gitlab.com,2026-03-17:5214822520 Leaminn Ma commented on merge request !227714 at GitLab.org / GitLab 2026-03-17T21:30:38Z lma-git Leaminn Ma

Hi @narendran-kannan Could you please do the first backend review here? Please pass to @mbobin for maintainer review if it looks okay to you. Thanks!

tag:gitlab.com,2026-03-17:5214816391 Leaminn Ma commented on merge request !227714 at GitLab.org / GitLab 2026-03-17T21:28:51Z lma-git Leaminn Ma

FF is enabled by default in specs.

tag:gitlab.com,2026-03-17:5214815673 Leaminn Ma commented on merge request !227714 at GitLab.org / GitLab 2026-03-17T21:28:36Z lma-git Leaminn Ma

FF is enabled by default in specs.

tag:gitlab.com,2026-03-17:5214814242 Leaminn Ma commented on merge request !227714 at GitLab.org / GitLab 2026-03-17T21:28:11Z lma-git Leaminn Ma

This is the same error being done in existing code paths in Ci::PlayBuildService and Ci::PlayBridgeService.

tag:gitlab.com,2026-03-17:5214787039 Leaminn Ma commented on merge request !227714 at GitLab.org / GitLab 2026-03-17T21:20:20Z lma-git Leaminn Ma

We need to change this from a bridge into a build because otherwise :play_job permission fails due to the user not having access to the downstream project. But we're not testing permissions here; we're only testing invalid transition, so we can just change the job type.

tag:gitlab.com,2026-03-17:5214771761 Leaminn Ma pushed to project branch add-defense-in-enqueue-job-service-update-tests at GitLab.org / GitLab 2026-03-17T21:16:12Z lma-git Leaminn Ma

Leaminn Ma (ee92503f) at 17 Mar 21:16

Add access check in EnqueueJobService

tag:gitlab.com,2026-03-17:5214766403 Leaminn Ma commented on merge request !227706 at GitLab.org / GitLab 2026-03-17T21:14:57Z lma-git Leaminn Ma

Closed in favour of Add access check to Ci::EnqueueJobService & add... (!227714).

tag:gitlab.com,2026-03-17:5214765760 Leaminn Ma closed merge request !227706: Draft: test: improve EnqueueJobService tests at GitLab.org / GitLab 2026-03-17T21:14:45Z lma-git Leaminn Ma

Relates to #524123

Changes

This MR significantly improves test coverage for Ci::EnqueueJobService to address the production incident and prevent future regressions.

Files Modified/Created

1. spec/services/ci/enqueue_job_service_spec.rb (Enhanced)

  • Documentation: Added clear comments explaining that this service does NOT perform authorization checks, following the defense-in-depth principle
  • User assignment tests: Verifies reassignment when job already has a user and setting user when nil
  • Environment handling: Tests jobs with environment associations
  • Optimistic locking: Tests retry behavior when StaleObjectError occurs
  • ResetSkippedJobsService integration: Expanded tests to verify correct parameters and execution order
  • Edge cases: Special characters (quotes, newlines, unicode), long values (1000+ chars), duplicate variable keys
  • Bridge job support: Tests that the service works correctly with bridge jobs (not just builds)
  • Protected environment behavior: Documents expected behavior without authorization checks

2. spec/services/ci/enqueue_job_service_authorization_spec.rb (New File)

  • Authorization contract documentation: Explains the defense-in-depth principle and multi-layer authorization
  • Service hierarchy: Documents that EnqueueJobService is a low-level service that trusts its callers
  • Authorization tests: Demonstrates how PlayBuildService and PlayBridgeService enforce permissions
  • Variable permission checks: Shows enforcement of :set_pipeline_variables ability
  • Defense-in-depth demonstration: Illustrates multiple authorization layers
  • Caller responsibilities: Documents correct and incorrect usage patterns

Test Coverage Summary

Before: ~80 lines, basic coverage After: ~350+ lines in main spec + ~180 lines in authorization spec

Key Design Decisions

  1. No Authorization in EnqueueJobService - Following the existing pattern where authorization happens in PlayBuildService and PlayBridgeService
  2. Defense-in-Depth Documentation - Clearly documented the multi-layer authorization approach per GitLab guidelines
  3. Comprehensive Edge Cases - Added tests for special characters, long values, and duplicate keys to prevent regressions
  4. Authorization Contract Spec - Created a separate spec file to serve as living documentation of the authorization contract

Production Incident Context

These improvements address the production incident (gitlab-com/gl-infra/production#19377) by:

  • Documenting the authorization contract clearly
  • Ensuring the service behavior is well-tested for edge cases
  • Making it explicit that callers must handle authorization
  • Providing comprehensive test coverage to prevent future regressions
tag:gitlab.com,2026-03-17:5214765442 Leaminn Ma commented on issue #524123 at GitLab.org / GitLab 2026-03-17T21:14:38Z lma-git Leaminn Ma

Created [FF] `ci_enqueue_job_authorization` -- Add auth... (#593898) for the changes in the MR: Add access check to Ci::EnqueueJobService & add... (!227714)

tag:gitlab.com,2026-03-17:5214730697 Leaminn Ma approved merge request !227590: Prepare partitioning: swap PK for ci_build_needs at GitLab.org / GitLab 2026-03-17T21:01:35Z lma-git Leaminn Ma

What does this MR do and why?

Prepare partitioning: swap PK for ci_build_needs

Changelog: changed

References

Screenshots or screen recordings

Before After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #583634

tag:gitlab.com,2026-03-17:5214282935 Leaminn Ma opened issue #593898: [FF] `ci_enqueue_job_authorization` -- Add authorization check to Ci::EnqueueJobService layer at GitLab.org / GitLab 2026-03-17T18:34:02Z lma-git Leaminn Ma tag:gitlab.com,2026-03-17:5214020145 Leaminn Ma commented on issue #524123 at GitLab.org / GitLab 2026-03-17T17:18:45Z lma-git Leaminn Ma

Add defense in depth to this service

I agree with doing this. However, I'm concerned about making any change to an integral service in the CI. So I think this requires a feature flag just in case.

Bumping weight to 2.

tag:gitlab.com,2026-03-17:5214008679 Leaminn Ma opened merge request !227714: Add access check to Ci::EnqueueJobService &amp; add tests that cover protected env access at GitLab.org / GitLab 2026-03-17T17:16:18Z lma-git Leaminn Ma

What does this MR do and why?

Context

Ci::EnqueueJobService is a central point for starting manual jobs for features like protected environments, but it doesn't have good test coverage as it was shown in gitlab-com/gl-infra/production#19377.

Moreover, it does not have defense in depth and relies on the caller to enforce authorization checks. We check the ability :play_job in Ci::PlayBuildService and Ci::PlayBridgeService. But it's not enforced in Ci::Pipeline#reset_source_bridge!.

This MR

This MR adds an access check inside Ci::EnqueueJobService so that it's applied no matter where the service is called. This prevents jobs from being enqueued in protected environments where the user does not have access.

Test coverage has been added for various environment/access scenarios. They follow the permission chart below (generated by Claude Code):

Screenshot_2026-03-17_at_1.15.56_PM

This change is made behind a feature flag: ci_enqueue_job_authorization. Roll-out issue: #593898

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #524123

tag:gitlab.com,2026-03-17:5214005140 Leaminn Ma pushed new project branch add-defense-in-enqueue-job-service-update-tests at GitLab.org / GitLab 2026-03-17T17:15:25Z lma-git Leaminn Ma

Leaminn Ma (11338bf7) at 17 Mar 17:15

tag:gitlab.com,2026-03-17:5213960437 Leaminn Ma pushed to project branch duo/feature/524123-improve-enqueue-job-service-tests-3165832 at GitLab.org / GitLab 2026-03-17T17:03:50Z lma-git Leaminn Ma

Leaminn Ma (0bd3152b) at 17 Mar 17:03

Add defense in depth to EnqueueJobService

tag:gitlab.com,2026-03-17:5213861528 Leaminn Ma commented on issue #591196 at GitLab.org / GitLab 2026-03-17T16:41:18Z lma-git Leaminn Ma

Status Update

Progress this week:

Confidence for current milestone:

  • 🔴 At risk - may not make it
  • 🟡 Some concerns - watching closely
  • 🟢 On track - confident we'll deliver

/cc @golnazs