What does this MR do and why?

This code change adds support for replicating and verifying design management action uploads in GitLab's Geo feature (which keeps multiple GitLab instances synchronized).

The changes create a new database table to track the synchronization status of design files uploaded through GitLab's design management system. This includes adding verification states to ensure files are properly copied between different GitLab servers, along with checksums to verify file integrity.

The update also adds new monitoring metrics so administrators can track how many design uploads have been successfully synchronized, failed, or are pending verification across their GitLab instances. Additionally, it updates the API documentation to include these new design upload statistics in the Geo status reports.

This enhancement ensures that design files uploaded to GitLab projects are properly backed up and synchronized across multiple GitLab installations for disaster recovery and performance purposes.

References

Related to #589909

How to set up and validate locally

Prerequisites

• A working Geo GDK setup with both primary and secondary running. Follow the Geo development setup guide.

1. Run database migrations

rails db:migrate # on the primary
rails db:migrate:geo # on the secondary

2. Enable the feature flags on the primary

# In Rails console on the primary
Feature.enable(:geo_design_management_action_upload_replication)
Feature.enable(:geo_design_management_action_upload_force_primary_checksumming)

3. Create test data on the primary

Upload a design to a project issue (e.g., attach an image via the Design Management tab on an issue). Alternatively, use the Rails console:

# In Rails console on the primary
project = Project.first
issue = project.issues.first || Issues::CreateService.new(container: project, current_user: User.first, params: { title: 'Test issue for designs' }).execute[:issue]

file = CarrierWaveStringFile.new_file(
  file_content: "Seeded design upload file in project #{project.full_path}",
  filename: 'seeded_design_upload.txt',
  content_type: 'text/plain'
)

UploadService.new(issue.design_collection, file, DesignManagement::DesignUploader).execute

Verify the upload exists in the design_management_action_uploads partition:

Geo::DesignManagementActionUpload.count
# Should be > 0

4. Verify checksumming on the primary

Wait for the verification worker to process, or trigger it manually:

# In Rails console on the primary
Geo::DesignManagementActionUpload.first.replicator.verify
Geo::DesignManagementActionUpload.first.design_management_action_upload_state.reload
Geo::DesignManagementActionUpload.first.design_management_action_upload_state.verification_state
# Should be 2 (verification_succeeded)

5. Verify replication on the secondary

Once the upload is created on the primary, Geo will automatically replicate it to the secondary. Check the sync status in the secondary Rails console:

# In Rails console on the secondary
Geo::DesignManagementActionUploadRegistry.count
# Should be > 0

registry = Geo::DesignManagementActionUploadRegistry.last
registry.state
# Should be 2 (synced)

If the registry is empty or not yet synced, you can manually trigger sync:

# In Rails console on the secondary
Geo::DesignManagementActionUploadReplicator.new(model_record_id: Geo::DesignManagementActionUpload.first.id).sync

6. Verify verification on the secondary

# In Rails console on the secondary
registry = Geo::DesignManagementActionUploadRegistry.last
registry.reload
registry.verification_state
# Should be 2 (verification_succeeded)

7. Test GraphQL API on the secondary

Note: You must be logged in as an admin user. Non-admin users will get null for Geo-related queries.

Note: When querying from the secondary's GraphQL explorer, add a custom header REQUEST_PATH with the value `/api/v4/geo/node_proxy/{node_id}/graphql

Open the GraphQL explorer on the secondary instance (http://<secondary-url>/-/graphql-explorer) and run:

query {
  geoNode {
    name
    primary
    designManagementActionUploadRegistries {
      nodes {
        id
        state
        verificationState
        designManagementActionUploadId
        lastSyncedAt
        verifiedAt
      }
    }
  }
}

Expected result: you should see registry entries with state: "SYNCED" and verificationState: "VERIFIED".

8. Verify Geo Sites API

Check the Geo Sites API includes the new design management action upload statistics:

curl --header "PRIVATE-TOKEN: <your-token>" "http://<primary-url>/api/v4/geo_sites/status"

Look for the new fields in the response:

• design_management_action_uploads_count
• design_management_action_uploads_checksummed_count
• design_management_action_uploads_checksum_failed_count
• design_management_action_uploads_synced_count
• design_management_action_uploads_failed_count
• design_management_action_uploads_registry_count
• design_management_action_uploads_synced_in_percentage
• design_management_action_uploads_verified_in_percentage

9. Verify Geo admin page

Visit /admin/geo/sites on the secondary and confirm that "Design Management Action Uploads" appears as a new data type with replication and verification progress.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Douglas Barbosa Alexandre (96d30369) at 16 Mar 22:27

... and 2 more commits

What does this MR do and why?

This code change adds support for tracking and replicating user uploads in GitLab's Geo feature, which helps organizations maintain synchronized copies of their GitLab data across multiple locations.

The main additions include:

1. New database table: Creates a "user_upload_states" table to track the verification status of user uploads, including when they were verified, any retry attempts, and checksums to ensure data integrity.

2. Database structure: Adds proper indexing and foreign key relationships to efficiently query and maintain the verification states, with support for organization-based data sharding.

3. GraphQL API updates: Extends the API to expose user upload registry information, allowing administrators to monitor replication status through GitLab's interface.

4. Monitoring metrics: Adds Prometheus metrics to track user upload synchronization statistics like total uploads, successful verifications, and failed attempts across Geo sites.

5. Documentation updates: Updates API documentation to include the new user upload tracking fields in Geo site status responses.

This enhancement ensures that user-uploaded files (like avatars, attachments, etc.) are properly replicated and verified across Geo sites, improving data consistency and reliability for organizations using GitLab's geographic replication feature.

References

Related to #589918

How to set up and validate locally

Prerequisites

• A working Geo GDK setup with both primary and secondary running. Follow the Geo development setup guide.

1. Run database migrations

rails db:migrate # on the primary
rails db:migrate:geo # on the secondary

2. Enable the feature flags on the primary

# In Rails console on the primary
Feature.enable(:geo_user_upload_replication)
Feature.enable(:geo_user_upload_force_primary_checksumming)

3. Create test data on the primary

Upload a file to a user (e.g., update a user avatar). Alternatively, use the Rails console:

# In Rails console on the primary
user = User.first
file = CarrierWaveStringFile.new_file(
  file_content: "Seeded upload file for user #{user.username}",
  filename: 'seeded_upload.txt',
  content_type: 'text/plain'
)

UploadService.new(user, file, PersonalFileUploader).execute

Verify the upload exists in the user_uploads partition:

Geo::UserUpload.count
# Should be > 0

4. Verify checksumming on the primary

Wait for the verification worker to process, or trigger it manually:

# In Rails console on the primary
Geo::UserUpload.first.replicator.verify
Geo::UserUpload.first.user_upload_state.reload
Geo::UserUpload.first.user_upload_state.verification_state
# Should be 2 (verification_succeeded)

5. Verify replication on the secondary

Once the upload is created on the primary, Geo will automatically replicate it to the secondary. Check the sync status in the secondary Rails console:

# In Rails console on the secondary
Geo::UserUploadRegistry.count
# Should be > 0

registry = Geo::UserUploadRegistry.last
registry.state
# Should be 2 (synced)

If the registry is empty or not yet synced, you can manually trigger sync:

# In Rails console on the secondary
Geo::UserUploadReplicator.new(model_record_id: Geo::UserUpload.first.id).sync

6. Verify verification on the secondary

# In Rails console on the secondary
registry = Geo::UserUploadRegistry.last
registry.reload
registry.verification_state
# Should be 2 (verification_succeeded)

7. Test GraphQL API on the secondary

Note: You must be logged in as an admin user. Non-admin users will get null for Geo-related queries.

Note: When querying from the secondary's GraphQL explorer, add a custom header REQUEST_PATH with the value `/api/v4/geo/node_proxy/{node_id}/graphql

Open the GraphQL explorer on the secondary instance (http://<secondary-url>/-/graphql-explorer) and run:

query {
  geoNode {
    name
    primary
    userUploadRegistries {
      nodes {
        id
        state
        verificationState
        userUploadId
        lastSyncedAt
        verifiedAt
      }
    }
  }
}

Expected result: you should see registry entries with state: "SYNCED" and verificationState: "VERIFIED".

8. Verify Geo Sites API

Check the Geo Sites API includes the new user upload statistics:

curl --header "PRIVATE-TOKEN: <your-token>" "http://<primary-url>/api/v4/geo_sites/status"

Look for the new fields in the response:

• user_uploads_count
• user_uploads_checksummed_count
• user_uploads_checksum_failed_count
• user_uploads_synced_count
• user_uploads_failed_count
• user_uploads_registry_count
• user_uploads_synced_in_percentage
• user_uploads_verified_in_percentage

9. Verify Geo admin page

Visit /admin/geo/sites on the secondary and confirm that "User Uploads" appears as a new data type with replication and verification progress.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Douglas Barbosa Alexandre (3d802589) at 16 Mar 21:24

... and 2 more commits

Douglas Barbosa Alexandre (b56134f9) at 16 Mar 21:21

... and 1 more commit

@nbelokolodov Could you do the analytics instrumentation review, please?

@axil Could you do the technical writer review, please?

@ahegyi Could you do the database maintainer review, please?

@s_murray @mkozono will be on PTO from March 17 to 27. Do you mind taking over these MRS and doing the backend and groupgeo maintainer review, please?

Please note that there will be 20 more MRs like these. Ok to send them all to you?

To make Rubocop happy. See https://docs.rubocop.org/rubocop-rails/latest/cops_rails.html#railsredundantforeignkey.

I don't see anything wrong, but could you remind me what prompted this change?

To avoid an unnecessary join with the groups.

Does this def scope_method also need to change to namespace_id_in instead of group_id_in?

Good catch! I fixed it in !227188.

Douglas Barbosa Alexandre (64f58eb1) at 16 Mar 19:42

... and 34 more commits

Douglas Barbosa Alexandre (ffe6a663) at 16 Mar 16:35

... and 407 more commits

Douglas Barbosa Alexandre (1f6104d8) at 13 Mar 23:56

... and 1 more commit

Douglas Barbosa Alexandre (ee041655) at 13 Mar 23:51

Douglas Barbosa Alexandre (4b666aab) at 13 Mar 22:01

... and 2 more commits

Douglas Barbosa Alexandre (04e34334) at 13 Mar 21:58

... and 1 more commit