• kubectl configured and pointing at the target cluster

• kubectl config current-context
• Cluster connectivity verified: kubectl get nodes

• kubectl config get-contexts

• gcloud container clusters get-credentials CLUSTER_NAME --region REGION --project PROJECT_ID

kubectl create ns argocd
kubectl apply -n argocd -k ./kubernetes/bootstrap/argocd/
kubectl wait --for=condition=available deployment/argocd-server -n argocd --timeout=300s

Apply the root Application for your environment:

# dev
kubectl apply -f ./kubernetes/clusters/dev/base.yaml

# stg
kubectl apply -f ./kubernetes/clusters/stg/base.yaml

# prd
kubectl apply -f ./kubernetes/clusters/prd/base.yaml

After this, ArgoCD manages itself and all resources under clusters/<env>/ automatically via Git.

# Get admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d && echo

# Port-forward
kubectl port-forward svc/argocd-server -n argocd 8080:80

Open http://localhost:8080 — login with user admin and the password above.

kubernetes/
├── base/
│   ├── cnpg-system/               # CNPG namespace, AppProject, nested Argo CD Application
│   └── external-secrets/          # Kustomize base (shared templates with placeholders)
├── bootstrap/argocd/              # ArgoCD installation (Kustomize)
└── clusters/
    ├── dev/                       # Dev environment
    ├── stg/                       # Staging environment
    └── prd/                       # Production environment
        ├── base.yaml              # Root Application (entry point)
        ├── external-secrets.yaml  # ArgoCD Application for ESO (Kustomize overlay)
        ├── argocd/                # ArgoCD self-management + ingress
        ├── cert-manager/          # TLS certificates
        ├── external-dns/          # DNS automation
        ├── external-secrets/      # Kustomize overlay (env-specific values)
        ├── cnpg-operator.yaml     # Argo CD Application for CNPG stack (path + Git ref per env)
        └── rhesis/                # Application manifests

Any YAML added under clusters/<env>/ and pushed to main is automatically deployed by the root *-base Application (unless you change that Application’s sync policy).

• Dev (kubernetes/clusters/dev/cnpg-operator.yaml): Git ref main; the parent Application uses automated sync. The nested cnpg-operator Application auto-syncs the Helm chart (see clusters/dev/cnpg-system/cnpg-operator-automated-sync.yaml).
• Stg and prd (kubernetes/clusters/stg/cnpg-operator.yaml and kubernetes/clusters/prd/cnpg-operator.yaml): spec.source.targetRevision points at a release branch (for example release/v1.2.3). Create that branch from the commit you intend to ship before syncing. Automated sync is disabled on the nested cnpg-operator Application: use manual Sync in Argo CD after the Git ref is updated. Promotion: validate on stg, then bump targetRevision on prd to the same ref and sync prd.
• Chart version per environment: edit kubernetes/clusters/<env>/cnpg-system/cnpg-operator-helm-chart.yaml (e.g. upgrade stg first, then prd).
• AppProject: cnpg-system restricts sources and destinations for the CNPG Helm Application (kubernetes/base/cnpg-system/argocd-project.yaml).

Use this to validate ESO and the GCP Secret Manager integration on the dev cluster without applying Terraform, then optionally run a full e2e test.

Confirms the external-secrets module is valid and shows the 4 new resources (SA, IAM binding, Workload Identity binding, Secret Manager API).

cd terraform/infrastructure/envs/dev
terraform init
terraform plan

Ensure kubectl is pointed at the dev cluster (kubectl config current-context).

Step 2a — Preview the Kustomize output:

kubectl kustomize kubernetes/clusters/dev/external-secrets/

Need to change: need to change the dev, stg, prd project_id and service_account based on the environment.

This should output the Namespace, ClusterSecretStore, Application, and a local ConfigMap — all with the correct rhesis-dev-sandbox project ID.

Step 2b — Apply Namespace and ESO Application first:

The ClusterSecretStore depends on the ESO webhook, which is deployed by the ArgoCD Application. Apply without the ClusterSecretStore first:

kubectl apply -k kubernetes/clusters/dev/external-secrets/ --server-side --force-conflicts 2>&1 || true

The ClusterSecretStore will fail on the first apply — this is expected because the ESO webhook isn't running yet.

Step 2c — Wait for ESO to deploy:

kubectl -n argocd get applications external-secrets -w

Wait until the Application is Synced and Healthy, then press Ctrl+C.

Step 2d — Verify ESO is running:

kubectl -n external-secrets get pods

You should see the controller and webhook pods.

Step 2e — Verify CRDs:

kubectl get crd clustersecretstores.external-secrets.io

Step 2f — Apply again to create the ClusterSecretStore:

Now that the webhook is running, re-apply to create the ClusterSecretStore:

kubectl apply -k kubernetes/clusters/dev/external-secrets/

Step 2g — Verify ClusterSecretStore:

Inspect status (expect Ready: False until Terraform has been applied for the GCP SA and Workload Identity):

kubectl get clustersecretstore gcp-secret-manager -o yaml

To test Workload Identity and secret sync:

cd terraform/infrastructure/envs/dev
terraform apply

Verify the ESO Kubernetes SA is annotated with the GCP SA:

kubectl -n external-secrets get sa external-secrets -o yaml | grep gcp-service-account

Create a secret in GCP Secret Manager (if you don’t have one):

echo -n "my-secret-value" | gcloud secrets create test-secret --data-file=- --project=rhesis-dev-sandbox

Create a test ExternalSecret (replace the remoteRef.key if you use another secret name):

kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: test-secret
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: gcp-secret-manager
    kind: ClusterSecretStore
  target:
    name: test-secret
  data:
    - secretKey: value
      remoteRef:
        key: test-secret
EOF

Check that the Kubernetes Secret was created:

kubectl get secret test-secret -n default -o jsonpath='{.data.value}' | base64 -d && echo

Remove manually applied resources so ArgoCD can manage them from Git after merge:

kubectl delete clustersecretstore gcp-secret-manager
kubectl -n argocd delete application external-secrets
kubectl delete configmap eso-config
kubectl delete namespace external-secrets

If you created a test ExternalSecret and GCP secret, delete them as needed:

kubectl delete externalsecret test-secret -n default
kubectl delete secret test-secret -n default
gcloud secrets delete test-secret --project=rhesis-dev-sandbox --quiet