Bojan Marjanovic (c6cb1d80) at 17 Mar 15:55

Bojan Marjanovic (526f1262) at 17 Mar 15:54

... and 1 more commit

What does this MR do and why?

Implements the Cells::Claims::VerificationService to backfill Cells claims between Rails database and the Topology Service. The verification service iterates over all records for a given model in batches, compares them against what the Topology Service has, and commits the diff (creates for records missing in TS, destroys for orphaned TS records). This ensures claims remain in sync even when prior writes were missed.

This MR specifically implements the backfill portion as part of the whole Verification Loop.

Not included in this MR

• Checks the diff when a record is found in both local and TS, eg reconciling different bucket value for the same source_id. This will be done in a follow up MR.

References

• gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#468

How to set up and validate locally

1. With seeded GDK, check the users table has some data in it
2. Run Cells::Claims::VerificationService.new(User).execute
3. Check the claims table in gdk psql -d topology_service
4. It should have 2 * number of users record in Rails (because User has 2 claimable attributes, thus 2x the amount in TS)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist (https://docs.gitlab.com/ee/development/code_review.html#acceptance-checklist)

Thanks @mcavoj

The database side LGTM! 🚀

What does this MR do and why?

Persist per-project schedules for scan execution policies

Add the security_scan_execution_project_schedules table and ScanExecutionProjectSchedule model to persist per-project random offsets for scheduled scan execution policies.

The RuleScheduleService now creates a project schedule row on every execution (behind the scan_execution_policy_project_schedule_creation feature flag).

References

1. 👉 !227096
2. !227101
3. !227102

Database

.for_rule_schedule

SELECT "security_policies".* FROM "security_policies" WHERE "security_policies"."type" = 1 AND "security_policies"."security_orchestration_policy_configuration_id" = 309 AND "security_policies"."policy_index" = 0

Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/49791/commands/148219

How to set up and validate locally

1. In rails console enable the feature flag

   Feature.enable(:scan_execution_policy_project_schedule_creation)

2. Create a project with a scan execution policy that runs on schedule

   scan_execution_policy:
     - name: Secrets on schedule
       description: ''
       enabled: true
       actions:
         - scan: secret_detection
           template: latest
           variables:
             SECURE_ENABLE_LOCAL_CONFIGURATION: 'false'
       rules:
         - type: schedule
           cadence: 0 0 1 * *
           branch_type: default
           timezone: Europe/Zurich
       skip_ci:
         allowed: true
   

3. Trigger the schedule pipeline manually via rails console

   schedule = Security::OrchestrationPolicyRuleSchedule.last
   schedule.update_columns(next_run_at: 1.day.ago) && Security::OrchestrationPolicyRuleScheduleWorker.new.perform    

4. Verify that project schedule record was created with schedule.project_schedules
5. Run the schedule again and verify that it doesn't get re-created again

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #592731

What does this MR do and why?

Implements the Cells::Claims::VerificationService to backfill Cells claims between Rails database and the Topology Service. The verification service iterates over all records for a given model in batches, compares them against what the Topology Service has, and commits the diff (creates for records missing in TS, destroys for orphaned TS records). This ensures claims remain in sync even when prior writes were missed.

This MR specifically implements the backfill portion as part of the whole Verification Loop.

Not included in this MR

• Checks the diff when a record is found in both local and TS, eg reconciling different bucket value for the same source_id. This will be done in a follow up MR.

References

• gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#468

How to set up and validate locally

1. With seeded GDK, check the users table has some data in it
2. Run Cells::Claims::VerificationService.new(User).execute
3. Check the claims table in gdk psql -d topology_service
4. It should have 2 * number of users record in Rails (because User has 2 claimable attributes, thus 2x the amount in TS)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist (https://docs.gitlab.com/ee/development/code_review.html#acceptance-checklist)

Thanks @bdenkovych 🙇

The MR was already in great shape, the ordering column is something you can pick-up quickly. 💯

Thanks @mdangelo6

The database side LGTM! 🚀

What does this MR do and why?

• This MR backfills the description_versions table, setting issue_id to the value of the corresponding epic's issue_id, and setting epic_id to null.

References

• #454439

Screenshots or screen recordings

Not provided

Database

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/49881/commands/148334

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #454439

Bojan Marjanovic (3848a081) at 17 Mar 12:24

Hey @ajaythomasinc 👋

could you please review this tiny change?

Why is this change being made?

As per https://gitlab.com/gitlab-com/team-member-epics/employment/-/work_items/9015, I'm updating my team's page and team's tag.

💡 Provide a detailed answer to the question on why this change is being proposed, in accordance with our value of Transparency.

Please add the details saying why, not just what in this section. Example: We have discussed the topic in Slack - (copy of Slack conversation). The current process is not efficient, this MR makes the description of X more clear, and helps move Y forward.

CHANGEME

Author and Reviewer Checklist

Please verify the check list and ensure to tick them off before the MR is merged.

• Provided a concise title for this Merge Request (MR)
• Added a description to this MR explaining the reasons for the proposed change, per say why, not just what
  • Copy/paste the Slack conversation to document it for later, or upload screenshots. Verify that no confidential data is added, and the content is SAFE
• Assign reviewers for this MR to the correct Directly Responsible Individual/s (DRI)
  • If the DRI for the page/s being updated isn’t immediately clear, then assign it to one of the people listed in the Maintained by section on the page being edited
  • If your manager does not have merge rights, please ask someone to merge it AFTER it has been approved by your manager in #mr-buddies
  • The when to get approval handbook section explains the workflow in more detail
• For transparency, share this MR with the audience that will be impacted.
  • Team: For changes that affect your direct team, share in your group Slack channel
  • Department: If the update affects your department, share the MR in your department Slack channel
  • Company: If the update affects all (or the majority of) GitLab team members, post an update in #whats-happening-at-gitlab linking to this MR
    • For high-priority company-wide announcements work with the internal communications team to post the update in #company-fyi and align on a plan to circulate in additional channels like the "While You Were Iterating" Newsletter

Commits

• Update bmarjanovic team tag

Bojan Marjanovic (768a1190) at 17 Mar 12:20

What does this MR do and why?

Implements the Cells::Claims::VerificationService to backfill Cells claims between Rails database and the Topology Service. The verification service iterates over all records for a given model in batches, compares them against what the Topology Service has, and commits the diff (creates for records missing in TS, destroys for orphaned TS records). This ensures claims remain in sync even when prior writes were missed.

This MR specifically implements the backfill portion as part of the whole Verification Loop.

Not included in this MR

• Checks the diff when a record is found in both local and TS, eg reconciling different bucket value for the same source_id. This will be done in a follow up MR.

References

• gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#468

How to set up and validate locally

1. With seeded GDK, check the users table has some data in it
2. Run Cells::Claims::VerificationService.new(User).execute
3. Check the claims table in gdk psql -d topology_service
4. It should have 2 * number of users record in Rails (because User has 2 claimable attributes, thus 2x the amount in TS)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist (https://docs.gitlab.com/ee/development/code_review.html#acceptance-checklist)