@dmeshcharakou

Another half is that the current replicator doesn't support models with more than one mounted file uploader.

Yes, I got the issue of Debian that one model with 2 file uploaders.

I was mainly focus on how the replicator will consume update event since we move Helm to GA, I think this is a bug that we need address if it's possible.

I got a confirmation from Geo team and created this MR: Add EVENT_UPDATED support to Helm metadata cach... (!227771). This can solve the issue for Helm metadata cache.

Thanks @cwoolley-gitlab for the detailed explanation!

I'm curious about the effort needed to add Geo support for 2 file uploaders.

If it's a relatively small change on the Geo side, it's more preferable to keep the Debian domain model cohesive rather than splitting it. We'd avoid the complexity of managing 2 separate records (create/update/query/delete operations) and keep the application logic simpler.

@dbalexandre can you please do groupgeo review to this MR?

@fmccawley would you mind to init backend review here?

Once it's done, can you forward to @mkhalifa3 for backend maintainer review?

Thank you!

::Geo::Event.last is the established pattern used consistently across the Geo spec suite.

This is needed to assert that we are not calling geo_handle_after_update.

We didn't change the implementation here, just get the was_persisted in the middle of the process.

Sylvia Shen (0ef34cab) at 18 Mar 16:11

Sylvia Shen (b4ce8880) at 18 Mar 14:55

... and 152 more commits

I rebased the branch and updated the milestone of the migration.

@dmeshcharakou can you please take another look for backend?

@Quintasan can please you review database side again?

Sorry for another round-trip, thank you! 🙇🏻‍♀️

Sylvia Shen (23098e6b) at 17 Mar 22:13

🎯 What does this MR do and why?

Add EVENT_UPDATED support to Helm metadata cache replicator to fix stale data on Geo secondaries.

Helm metadata caches are mutable blobs that get updated in-place when charts are pushed or modified. The current Geo replicator only handles EVENT_CREATED and EVENT_DELETED, so updates don't trigger replication events. This MR adds EVENT_UPDATED support by:

1. Registering EVENT_UPDATED in PackagesHelmMetadataCacheReplicator
2. Implementing consume_event_updated to reuse existing download/sync logic
3. Overriding immutable? to return false for mutable blob handling
4. Calling geo_handle_after_update in CreateMetadataCacheService after updates

This follows the existing pattern used for mutable resources like Git repositories.

Design Decision: Option A vs Option B

Option A: Keep Helm mutable, add EVENT_UPDATED to Geo ✅ (chosen)

• Keep Helm metadata updating in-place as it currently does
• Add EVENT_UPDATED support to the Geo replicator to handle these updates
• Pros: Semantically correct, minimal changes, quick fix
• Cons: Creates an exception in Geo for mutable blobs

Option B: Make Helm immutable like Debian

• Refactor Helm to treat updates as delete + create operations (each update creates a new blob)
• Geo can continue using only EVENT_CREATED/EVENT_DELETED
• Pros: Aligns with long-term architecture, reduces Geo exceptions, cleaner logic
• Cons: Higher uncertainty (race condition and potential time gap with no data) and more effort, broader impact

We chose Option A because it's semantically correct (Helm metadata is genuinely updated in-place), requires minimal changes, and provides a quick fix without large-scale refactoring.

📋 References

Related to #593838

Slack discussion: https://gitlab.slack.com/archives/C32LCGC1H/p1773679654331229

📸 Screenshots or screen recordings

N/A

🧪 How to set up and validate locally

Prerequisites

• Set up a Geo primary and secondary instance
• Have a test Helm chart package ready (e.g., test-chart-0.123.0.tgz)

Steps

1. Upload Helm package to primary

   curl --request POST \
       --user "root:$TOKEN" \
       --form '[email protected]' \
       "${GITLAB_URL}/api/v4/projects/${PROJECT_ID}/packages/helm/api/stable/charts"

2. Verify metadata cache created on primary

   cache = Packages::Helm::MetadataCache.last
   puts cache.file.read

3. Verify metadata cache replicated to secondary

   • Check that the same cache record exists on the secondary
   • Verify the file content matches the primary

4. Upload another version of the Helm package to primary

   curl --request POST \
       --user "root:$TOKEN" \
       --form '[email protected]' \
       "${GITLAB_URL}/api/v4/projects/${PROJECT_ID}/packages/helm/api/stable/charts"

5. Verify metadata cache updated on primary

   cache = Packages::Helm::MetadataCache.last
   puts cache.file.read  # Should show updated content with new chart version

6. Verify metadata cache updated on secondary

   • Check that the secondary's cache record is updated (not stale)
   • Verify the file content matches the primary's updated content
   • Confirm EVENT_UPDATED was triggered and processed by Geo

✅ MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Changelog: fixed EE: true

Sylvia Shen (da77234c) at 17 Mar 21:38

... and 1 more commit

Sylvia Shen (2e916007) at 17 Mar 17:43

Sylvia Shen (2e8a1b2f) at 17 Mar 17:33

... and 1399 more commits

Sylvia Shen (fadc2039) at 17 Mar 14:54

... and 1 more commit

@dmeshcharakou after some investigation, I found that the failed pipeline is caused by the duplicated timestamp of the migration.

I fixed the issue in !226867 (b2c96087), the pipeline passed 🎉

Can you please take another look?

Thank you!

Thanks for the ping @dmeshcharakou!

You're right that after_update_commit doesn't trigger a Geo event for blob replicators. After checking, BlobReplicatorStrategy only supports EVENT_CREATED and EVENT_DELETED (no EVENT_UPDATED), and assumes blobs are immutable.

This means secondaries will have stale data until reverification detects a checksum mismatch - potentially up to 7 days with the default minimum_reverification_interval.

However, RepositoryReplicatorStrategy does support EVENT_UPDATED for mutable resources. A similar approach could work for Helm metadata cache: adding EVENT_UPDATED support to the replicator and calling geo_handle_after_update after updates. (We do have something similar with wiki_repository.))

I've posted in #g_geo to confirm this approach with the Geo team.

👉🏻 https://gitlab.slack.com/archives/C32LCGC1H/p1773679654331229

For Debian distributions, I think D-split still makes sense since it also solves the dual-blob problem (two files per record). That said, if the EVENT_UPDATED approach works for Helm, we could potentially simplify D-split by using in-place updates instead of the create-new-on-change pattern. We can revisit this after we get confirmation from the Geo team.

🗜️ Context

In !185280, we introduced a rate limit for virtual registries Client APIs.

See !185280

This MR bumps that limit to 4000 / 15 secs (up from 1000 / 15 secs). For the reasoning here, see #588754 (comment 3058747520) (internal)

🤔 What does this MR do and why?

• Update setting virtual_registries_endpoints_api_limit default value to 4000
• Update the related specs

Worth noting here that virtual registries are still behind feature flag s. Some artifact formats are enabled by default and others are still in work in progress

📖 References

• #588754

🪜 Screenshots or screen recordings

No UI changes

🧑‍🔬 How to set up and validate locally

This MR doesn't change how the setting is read, applied or updated. We are merely updating the default value.

If you still want to test these parts, you can follow !185280.

🏎️ MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

✅

Database review

⬆️ Migration up

$ rails db:migrate
main: == [advisory_lock_connection] object_id: 137680, pg_backend_pid: 6605
main: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: migrating ==
main: -- execute("UPDATE application_settings\nSET rate_limits = jsonb_set(rate_limits, '{virtual_registries_endpoints_api_limit}', '4000')\nWHERE rate_limits->>'virtual_registries_endpoints_api_limit' = '1000'\n")
main:    -> 0.0502s
main: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: migrated (0.0644s)

main: == [advisory_lock_connection] object_id: 137680, pg_backend_pid: 6605
ci: == [advisory_lock_connection] object_id: 137680, pg_backend_pid: 6609
ci: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: migrating ==
ci: -- The migration is skipped since it modifies the schemas: [:gitlab_main].
ci: -- This database can only apply migrations in one of the following schemas: [:gitlab_ci, :gitlab_ci_cell_local, :gitlab_internal, :gitlab_shared, :gitlab_shared_cell_local, :gitlab_shared_org].
ci: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: migrated (0.0102s)

ci: == [advisory_lock_connection] object_id: 137680, pg_backend_pid: 6609
sec: == [advisory_lock_connection] object_id: 137680, pg_backend_pid: 6614
sec: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: migrating ==
sec: -- The migration is skipped since it modifies the schemas: [:gitlab_main].
sec: -- This database can only apply migrations in one of the following schemas: [:gitlab_internal, :gitlab_sec, :gitlab_shared, :gitlab_shared_cell_local, :gitlab_shared_org].
sec: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: migrated (0.0095s)

sec: == [advisory_lock_connection] object_id: 137680, pg_backend_pid: 6614

⬇️ Migration down

$ rails db:migrate:down:main VERSION=20260317110531 && r db:migrate:down:ci VERSION=20260317110531 && r db:migrate:down:sec VERSION=20260317110531
main: == [advisory_lock_connection] object_id: 136840, pg_backend_pid: 5886
main: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: reverting ==
main: -- execute("UPDATE application_settings\nSET rate_limits = jsonb_set(rate_limits, '{virtual_registries_endpoints_api_limit}', '1000')\nWHERE rate_limits->>'virtual_registries_endpoints_api_limit' = '4000'\n")
main:    -> 0.0342s
main: == 20260317110531 UpdateVirtualRegistriesEndpointsApiLimitDefault: reverted (0.0448s)

main: == [advisory_lock_connection] object_id: 136840, pg_backend_pid: 5886
ci: == [advisory_lock_connection] object_id: 136860, pg_backend_pid: 6078
ci: == [advisory_lock_connection] object_id: 136860, pg_backend_pid: 6078
sec: == [advisory_lock_connection] object_id: 136860, pg_backend_pid: 6292
sec: == [advisory_lock_connection] object_id: 136860, pg_backend_pid: 6292