John Cai (a9ede422) at 17 Mar 21:34

... and 5 more commits

cc: @stanhu

Description

Praefect provides high availability for Gitaly clusters by coordinating replication, routing, and failover. To ensure production reliability, we need to systematically test Praefect's behavior under various failure conditions.

Background

Praefect is a gRPC proxy that sits between GitLab Rails and Gitaly nodes, managing:

• Request routing (accessor vs mutator RPCs)
• Primary election via SQL elector with quorum-based consensus
• Replication queue processing
• Reference transaction voting for strong consistency

Proposed Chaos Test Scenarios

1. Single Gitaly Node Failures

2. Quorum Loss Scenarios

3. Database Failures

4. Network Partition Scenarios

5. Transaction/Consistency Failures

6. Praefect Instance Failures

7. Resource Exhaustion

Acceptance Criteria

• Chaos test framework integrated with existing test infrastructure
• Each scenario has automated test coverage
• Tests validate both failure behavior AND recovery
• Metrics captured during chaos events (latency, error rates, recovery time)
• Documentation of expected vs actual behavior for each scenario
• CI/CD integration for periodic chaos testing

Suggested Implementation Approach

1. Use existing chaos patterns - Praefect already has a Chaos service (see internal/praefect/service/server/info.go)
2. Leverage testhelper infrastructure - Use testhelper.MustCreateGitalyClient() patterns
3. Database chaos - Use testdb package with connection manipulation
4. Network chaos - Consider tools like toxiproxy or iptables rules for network simulation

Key Metrics to Monitor

• gitaly_praefect_transactions_total (success/failure breakdown)
• gitaly_praefect_replication_jobs (queue depth)
• gitaly_praefect_node_latency_bucket (health check latency)
• Recovery time to healthy state
• Data consistency verification post-recovery

So at a high level, this is essentially a message broker between system resource conditions being met, and whatever needs to respond to those events?

With our current adaptive limiter, would this sit between the watchers and the adaptive limiters?

John Cai (11a5ad5e) at 17 Mar 19:46

... and 5 more commits

@leeeee-gtlb per our conversation around Gitaly on k8s support, what if we introduced this idea of "early support" which essentially means you have to register in order to receive support if you're running production workloads on Gitaly on k8s?

This would still allow us to "GA" while gating the support load. Thoughts?

What does this MR do and why?

We are ready to declare Gitaly on K8s GA but with early support. This means customers must register to receive support on Gitaly on K8s.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

John Cai (d7e3d125) at 17 Mar 19:21

Also, the diagram has a step Git writes refs. Should there be a separate step where Git is writing the objects? Does that come before or after the reference-transaction hook?

Yeah that could clarify things. I think objects get written before the reference-transaction hook.

Makes sense! Do you know what would happen if the secondaries were lagging behind in generation number?

Only secondaries that are up to date get included in the set of voters

@GitLabDuo can you give me an example in the codebase where it's formatted like this?

John Cai (d1d47aa7) at 17 Mar 14:27

... and 591 more commits

@GitLabDuo i can't find anywhere in the codebase where it's formatted like that.

i believe we have a plan but we are waiting for the upstream contribution, and then we can go ahead with @eric.p.ju's design

great work @oli.campeau and @gl-gitaly!

UPDATE: The issues described above represent edge cases in Praefect's consistency model. They do not constitute data loss—Git objects remain durable, and replicas self-heal through reconciliation. Several of these issues are being addressed directly (see #7059+, PostReceiveHook invoked before write has been r... (gitaly#5406)).

While RAFT does remain the long term vision of Gitaly, Praefect remains a viable HA solution.

John Cai (030021bb) at 16 Mar 20:24

@pks-gitlab will gitaly#7059 unblock this effort?

One way to address inconsistencies on crash is to ensure clean startup. Here's a possibility

Pending Transaction Tracking for Praefect Crash Consistency

TL;DR

Record transaction intent in PostgreSQL before telling Gitalys to commit. On Praefect startup, check for any pending transactions and verify the actual state of replicas to reconcile the database.

New Table

CREATE TABLE pending_transactions (
    transaction_id   BIGINT PRIMARY KEY,
    repository_id    BIGINT NOT NULL REFERENCES repositories(repository_id) ON DELETE CASCADE,
    primary_storage  TEXT NOT NULL,
    expected_voters  TEXT[] NOT NULL,
    change_type      TEXT NOT NULL,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Flow

Normal operation:

1. Transaction quorum reached
2. INSERT pending transaction record
3. Send COMMIT to Gitalys (they write to disk)
4. Request finalizer runs IncrementGeneration()
5. DELETE pending transaction record

On Praefect startup:

1. Query all pending transactions
2. For each: verify actual replica states (e.g., compare HEAD refs across storages)
3. Update database generation to reflect which replicas actually committed
4. Queue replication jobs for any that didn't
5. Delete the pending record

Why This Works

• If Praefect crashes before INSERT: No record exists, no inconsistency
• If Praefect crashes after INSERT but before COMMIT sent: Recovery finds record, verifies no Gitaly committed, deletes record
• If Praefect crashes after COMMIT: Recovery finds record, detects which Gitalys committed, updates generations accordingly The database always ends up reflecting the true on-disk state.