With the new Siphon config structure in the monolith we'll ensure that sensitive fields are detected and skipped: gitlab-org/gitlab!226902

Adam Hegyi (ae5e39ec) at 17 Mar 11:30

Adam Hegyi (06235a4d) at 17 Mar 11:30

... and 1 more commit

This MR contains the following updates:

View the Renovate pipeline for this MR

Note: The pre-commit manager in Renovate is not supported by the pre-commit maintainers or community. Please do not report any problems there, instead create a Discussion in the Renovate repository if you have any questions.

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻️ Rebasing: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this MR and you won't be reminded about these updates again.

• If you want to rebase/retry this MR, check this box

This MR has been generated by Renovate Bot.

This MR contains the following updates:

View the Renovate pipeline for this MR

Note: The pre-commit manager in Renovate is not supported by the pre-commit maintainers or community. Please do not report any problems there, instead create a Discussion in the Renovate repository if you have any questions.

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻️ Rebasing: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this MR and you won't be reminded about these updates again.

• If you want to rebase/retry this MR, check this box

This MR has been generated by Renovate Bot.

@stefanosxan, thanks! DB LGTM! 👍

I couldn't open the depesz links in the description, maybe they're expired?

What does this MR do and why?

This MR automatically attaches newly created custom work item types to existing custom lifecycles in their namespace. When a user creates a new custom work item type, the system now automatically creates the necessary TypeCustomLifecycle associations so the new type inherits the custom lifecycle configuration.

Background

!225548 implemented the ability for custom work item types to fetch and use custom lifecycles. However, it only handled the read side - custom types could find their lifecycle, but when creating a new custom type, no lifecycle association was created.

This MR completes the implementation by handling the write side - automatically creating lifecycle associations when custom types are created.

Implementation Details

For namespace-scoped custom types:

• When creating a custom type in a namespace with custom lifecycles, automatically attach it to the appropriate lifecycle
• Prefers the lifecycle attached to the Issue type (as the default base type)
• Falls back to the first available custom lifecycle if Issue type is not attached

For organization-scoped custom types:

• Batch processes all root groups in the organization that have custom lifecycles
• Creates lifecycle associations for each namespace with custom lifecycles
• Uses insert_all! for efficient bulk insertion

Converted types:

• Skips lifecycle attachment for types converted from system-defined types
• These types inherit lifecycle behavior from their source system-defined type

Changes

• Add type_custom_lifecycles association to Namespace model
• Add for_namespace scope to TypeCustomLifecycle model
• Update WorkItems::Types::CreateService to automatically attach custom types to lifecycles
• Add comprehensive test coverage for all scenarios

Database

That's all from my local host as in production we don't have custom types yet.

In case we a single namespace the

1. query to fetch the type_Custom_lifecycle for the namespace: (that's from production as the record exist ) https://console.postgres.ai/gitlab/gitlab-production-main/sessions/49519/commands/147654

2. the query to create the record: https://explain-depesz.postgres.ai/s/Kl

In case we have an organization.

As it is a bit hard to create 100 root groups with custom lifecycles, i used the ones i have in my localhost that are 11.

1. Fetch the first root group for the each_batch https://explain-depesz.postgres.ai/s/lE

2. Fetch the last root_group for the each_batch https://explain-depesz.postgres.ai/s/pm

3. Loop over the root groups: https://explain-depesz.postgres.ai/s/Za

4. create type_custom_lifecycle records in bulk:

https://explain-depesz.postgres.ai/s/nx

References

• Follow-up to: !225548
• Related work item: #581952

How to set up and validate locally

Prerequisites

1. Enable the feature flag: Feature.enable(:work_item_configurable_types)

Test scenario 1: Namespace with custom lifecycle

# In Rails console
group = Group.create!(name: 'Test Group', path: 'test-group')
lifecycle = WorkItems::Statuses::Custom::Lifecycle.create!(namespace: group, name: 'Custom Lifecycle')
issue_type = WorkItems::TypesFramework::SystemDefined::Type.default_issue_type

# Attach Issue type to the custom lifecycle
WorkItems::TypeCustomLifecycle.create!(
  work_item_type_id: issue_type.id,
  lifecycle: lifecycle,
  namespace: group
)

# Create a new custom work item type
service = WorkItems::Types::CreateService.new(
  container: group,
  current_user: User.first,
  params: { name: 'Feature', icon_name: 'work-item-feature' }
)
result = service.execute

# Verify the new type is attached to the custom lifecycle
custom_type = result.payload[:work_item_type]
type_lifecycle = WorkItems::TypeCustomLifecycle.find_by(work_item_type_id: custom_type.id)


type_lifecycle.lifecycle == lifecycle # Should be true

Test scenario 2: Organization with multiple groups

# Create organization with multiple groups having custom lifecycles
org = Organizations::Organization.create!(name: 'Test Org', path: 'test-org')
group1 = Group.create!(name: 'Group 1', path: 'group-1', organization: org)
group2 = Group.create!(name: 'Group 2', path: 'group-2', organization: org)

lifecycle1 = WorkItems::Statuses::Custom::Lifecycle.create!(namespace: group1, name: 'Lifecycle 1')
lifecycle2 = WorkItems::Statuses::Custom::Lifecycle.create!(namespace: group2, name: 'Lifecycle 2')

# Attach Issue type to both lifecycles
issue_type = WorkItems::TypesFramework::SystemDefined::Type.default_issue_type
WorkItems::TypeCustomLifecycle.create!(work_item_type_id: issue_type.id, lifecycle: lifecycle1, namespace: group1)
WorkItems::TypeCustomLifecycle.create!(work_item_type_id: issue_type.id, lifecycle: lifecycle2, namespace: group2)

# Create organization-level custom type
service = WorkItems::Types::CreateService.new(
  container: org,
  current_user: User.first,
  params: { name: 'Bug', icon_name: 'bug' }
)
result = service.execute

# Verify the type is attached to both groups' lifecycles
custom_type = result.payload[:work_item_type]
type_lifecycles = WorkItems::TypeCustomLifecycle.where(work_item_type_id: custom_type.id)

type_lifecycles.count == 2 # Should be true
type_lifecycles.pluck(:namespace_id).sort == [group1.id, group2.id].sort # Should be true

Just want to raise awareness that this might create a long running transaction (which might lock records longer than usual). Given that this code targets only self-managed customers with likely low number of top level groups, I think we're safe to keep the transaction.

Adam Hegyi (d7fb4122) at 17 Mar 10:13

... and 1 more commit

Adam Hegyi (23cf5be2) at 17 Mar 10:13

What does this MR do and why?

The db/click_house/main.sql schema file has the two PROJECTIONs in ci_finished_builds in a different order than what the migrations produce. This causes the clickhouse-schema-check CI job to fail because it detects uncommitted schema changes after running migrations.

This MR reorders the projections so that by_project_pipeline_finished_at_name_v2 comes before build_stats_by_project_pipeline_finished_at_name_stage_name, matching the order produced by running all ClickHouse migrations.

References

Failing job: https://gitlab.com/gitlab-org/gitlab/-/jobs/13503051419

Screenshots or screen recordings

N/A — no UI changes.

How to set up and validate locally

No local validation needed. The fix is verified by the clickhouse-schema-check CI job passing on this branch.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

@arun.sori, can you take a look?

@GitLabDuo, I only see it in one consumer config

What

This MR adds deployment-related tables to Siphon (main)

Adam Hegyi (971fffa7) at 17 Mar 09:31

Adam Hegyi (2941ea28) at 17 Mar 09:23

What does this MR do?

Fixes misleading log message in adaptive batch processing. Previously, the "Batch finished with reduced batch size" message was logged even when the batch succeeded on the first attempt without any retries. This change introduces a retried flag to ensure the log message only appears when the batch size was actually reduced due to an error.

How do I test it?

1. Run the existing tests to verify behavior
2. Observe logs during batch processing - the reduced batch size message should only appear after actual retries

Adam Hegyi (174be228) at 17 Mar 09:23

... and 1 more commit