GitOps with ArgoCD: What Teams Wish They Knew Before Starting

GitOps sounds simple: Git is the single source of truth for your infrastructure. Push a change, and the system converges to match. In practice, adopting GitOps with ArgoCD reveals that the concept is simple but the execution has sharp edges.

What GitOps Actually Means

Traditional deployment: CI pipeline builds the image, then pushes it to the cluster. The pipeline has cluster credentials and executes kubectl apply.

GitOps deployment: CI pipeline builds the image, then updates a Git repo with the new image tag. ArgoCD watches that repo and applies the changes to the cluster.

Traditional:  Code Repo → CI Build → CI Deploys → Cluster
GitOps:       Code Repo → CI Build → Updates Config Repo → ArgoCD → Cluster

The critical difference: nothing outside the cluster pushes changes to it. ArgoCD pulls from Git. No CI pipeline needs cluster credentials.

The Repository Structure

After trying several approaches, this structure works best:

infrastructure/
├── apps/
│   ├── api-service/
│   │   ├── base/
│   │   │   ├── deployment.yml
│   │   │   ├── service.yml
│   │   │   └── kustomization.yml
│   │   └── overlays/
│   │       ├── staging/
│   │       │   ├── kustomization.yml
│   │       │   └── replicas-patch.yml
│   │       └── production/
│   │           ├── kustomization.yml
│   │           └── replicas-patch.yml
│   └── worker-service/
│       └── ...
├── argocd/
│   ├── api-service.yml
│   └── worker-service.yml
└── README.md

ArgoCD Application Definition

argocd/api-service.yml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/infrastructure.git
    targetRevision: main
    path: apps/api-service/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m

Key settings:

selfHeal: true — if someone kubectl edits a resource manually, ArgoCD reverts it to match Git. This is the whole point of GitOps.
prune: true — if you remove a manifest from Git, ArgoCD deletes the resource from the cluster. Without this, orphaned resources accumulate.
Retry with backoff — transient failures (API server hiccups, webhook timeouts) don't leave the app in a failed state.

The Image Update Problem

The biggest gotcha in GitOps: how do you update the image tag after a CI build?

Option 1: CI Updates the Config Repo

.github/workflows/build.yml

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Build and push image
        run: |
          docker build -t registry.example.com/api:${{ github.sha }} .
          docker push registry.example.com/api:${{ github.sha }}
 
      - name: Update config repo
        run: |
          git clone https://github.com/myorg/infrastructure.git
          cd infrastructure
          kustomize edit set image \
            api=registry.example.com/api:${{ github.sha }} \
            -C apps/api-service/overlays/production
          git add .
          git commit -m "deploy: api-service ${{ github.sha }}"
          git push

This works but creates a tight coupling between CI and the config repo.

Option 2: ArgoCD Image Updater (Preferred)

ArgoCD Image Updater watches your container registry and automatically updates image tags in Git:

api-service-with-image-updater.yml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=registry.example.com/api
    argocd-image-updater.argoproj.io/api.update-strategy: semver
    argocd-image-updater.argoproj.io/write-back-method: git

This decouples CI from GitOps entirely. CI pushes the image, Image Updater detects it, updates Git, ArgoCD syncs.

Handling Secrets

Secrets are the hardest part of GitOps. You can't commit plaintext secrets to Git, but Git is supposed to be the single source of truth.

Sealed Secrets is a solid solution here — encrypt secrets locally, commit the encrypted version to Git, and the Sealed Secrets controller decrypts them in the cluster:

seal-secret.sh

# Create a regular secret
kubectl create secret generic db-credentials \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml > secret.yml
 
# Seal it (encrypt for the cluster's public key)
kubeseal --format=yaml < secret.yml > sealed-secret.yml
 
# Commit the sealed version — safe to store in Git
rm secret.yml  # Never commit the plaintext version
git add sealed-secret.yml
git commit -m "chore: update db credentials"

What Can Go Wrong

Sync Waves and Dependencies

If Service B depends on Service A, you need sync waves to ensure A is deployed first:

service-a.yml

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

service-b.yml

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"

Without sync waves, ArgoCD applies everything in parallel, and Service B may crash because Service A isn't ready yet.

Drift Detection False Positives

Some Kubernetes controllers modify resources after creation (adding default fields, mutating webhooks). ArgoCD sees these as "drift" and tries to revert them, creating an endless sync loop.

Fix with ignore differences:

ignore-mutations.yml

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # If using HPA, ignore replica count
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP  # Assigned by K8s, changes on recreation

Key Takeaways

Separate config repos from code repos — keeps ArgoCD signal clean
Enable selfHeal and prune — without these, GitOps is just Git-triggered deploys
Use ArgoCD Image Updater — avoid coupling CI to your config repo
Solve secrets before adopting GitOps — Sealed Secrets or External Secrets Operator
Plan for drift detection noise — ignoreDifferences is not optional in production

#What GitOps Actually Means

#The Repository Structure

#ArgoCD Application Definition

#The Image Update Problem

#Option 1: CI Updates the Config Repo

#Option 2: ArgoCD Image Updater (Preferred)

#Handling Secrets

#What Can Go Wrong

#Sync Waves and Dependencies

#Drift Detection False Positives

#Key Takeaways