Skip to content

GitOps Workflows Guide: Advanced GitOps Practices

Overview

GitOps is a set of practices that uses Git as the single source of truth for declarative infrastructure and applications. This guide covers advanced GitOps workflows, tools, and best practices for implementing robust continuous delivery pipelines.

Core Principles

Declarative Configuration

  • Everything as Code: Infrastructure, applications, and configurations stored in Git
  • Version Control: Full history and audit trail of all changes
  • Reproducibility: Ability to recreate environments from Git state

Automated Synchronization

  • Continuous Reconciliation: Automatic synchronization between Git and runtime
  • Drift Detection: Identify and correct configuration drift
  • Self-Healing: Automatic recovery from failures

Observability and Audit

  • Complete Audit Trail: Every change tracked and attributable
  • Real-time Monitoring: Visibility into deployment status and health
  • Compliance: Automated compliance checking and reporting

GitOps Tools Ecosystem

GitOps Operators

ArgoCD

# ArgoCD Application definition
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-production
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/my-app
    targetRevision: HEAD
    path: overlays/production
    kustomize:
      version: v4.4.1
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
  revisionHistoryLimit: 10

Flux

# Flux GitRepository and Kustomization
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/my-app
  ref:
    branch: main
  secretRef:
    name: git-auth
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 5m
  path: ./kustomize
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-app
  targetNamespace: production
  validation: client

Comparison: ArgoCD vs Flux

Feature ArgoCD Flux
UI Rich web UI CLI-focused
GitOps Engine Application Controller GitOps Toolkit
Multi-tenancy Projects and RBAC Multi-tenant by design
Ecosystem Large, active Growing rapidly
Complexity Medium Low to Medium
Customization Extensive Modular

Advanced GitOps Patterns

Multi-Environment Deployments

Environment Promotion

# ArgoCD ApplicationSet for multi-environment
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app
spec:
  generators:
  - list:
      elements:
      - env: development
        namespace: dev
        wave: "1"
      - env: staging
        namespace: staging
        wave: "2"
      - env: production
        namespace: prod
        wave: "3"
  template:
    metadata:
      name: '{{env}}-my-app'
      annotations:
        argocd.argoproj.io/sync-wave: '{{wave}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/my-app
        targetRevision: HEAD
        path: overlays/{{env}}
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Progressive Delivery

# Flagger configuration for canary deployments
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: my-app
  namespace: production
spec:
  provider: kubernetes
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  progressDeadlineSeconds: 60
  service:
    port: 80
    targetPort: 8080
  analysis:
    interval: 30s
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m

GitOps with Infrastructure as Code

Crossplane for Cloud Resources

# Crossplane AWS provider configuration
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-aws:v0.33.0
---
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: aws-provider
spec:
  credentials:
    source: Secret
    secretRef:
      name: aws-creds
      key: creds
      namespace: crossplane-system

Terraform with GitOps

# Terraform Cloud configuration
terraform {
  cloud {
    organization = "my-org"
    workspaces {
      name = "infrastructure"
    }
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

# VPC resource
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name        = "main-vpc"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

Secrets Management in GitOps

Sealed Secrets

# SealedSecret for encrypted secrets
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-secret
  namespace: production
spec:
  encryptedData:
    password: AgA2...
    username: AgB3...
  template:
    metadata:
      name: my-secret
      namespace: production
    type: Opaque

External Secrets Operator

# ExternalSecret for cloud secret stores
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example-secret
spec:
  refreshInterval: 15s
  secretStoreRef:
    name: aws-secret-store
    kind: SecretStore
  target:
    name: example-secret
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: prod/database
      property: password

Branching Strategies

Git Flow for GitOps

main (production)
├── releases/v1.0
├── releases/v1.1
└── releases/v2.0

staging
├── feature/new-feature
├── feature/bug-fix
└── hotfix/critical-fix

Environment Branches

# Branch-based environment deployment
# .argocd/config.yaml
applications:
  - name: dev-my-app
    source:
      repoURL: https://github.com/myorg/my-app
      targetRevision: develop
      path: overlays/dev
  - name: staging-my-app
    source:
      repoURL: https://github.com/myorg/my-app
      targetRevision: staging
      path: overlays/staging
  - name: prod-my-app
    source:
      repoURL: https://github.com/myorg/my-app
      targetRevision: main
      path: overlays/prod

CI/CD Integration

GitHub Actions with ArgoCD

# .github/workflows/deploy.yml
name: Deploy to Kubernetes
on:
  push:
    branches: [ main, staging, develop ]
  pull_request:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Configure Git
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Update ArgoCD Application
        run: |
          # Update image tag in git
          sed -i "s|image: my-app:.*|image: my-app:${{ github.sha }}|g" kustomize/deployment.yaml
          git add kustomize/deployment.yaml
          git commit -m "Update image to ${{ github.sha }}"
          git push

Automated Testing in GitOps

Policy as Code

# OPA Gatekeeper constraint template
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-merge-conflict

  - repo: https://github.com/zricethezav/gitleaks
    rev: v8.15.0
    hooks:
      - id: gitleaks

  - repo: https://github.com/open-policy-agent/conftest
    rev: v0.35.0
    hooks:
      - id: conftest
        args: [test, --policy, policy/, kustomize/]

Monitoring and Observability

GitOps Metrics

# Prometheus metrics for ArgoCD
apiVersion: v1
kind: ServiceMonitor
metadata:
  name: argocd-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics
  endpoints:
  - port: metrics
    interval: 30s

Application Health Checks

# ArgoCD health checks
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  source:
    repoURL: https://github.com/myorg/my-app
    path: chart
    targetRevision: HEAD
    helm:
      valueFiles:
      - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  info:
  - name: 'Health Status'
    value: 'Healthy'

Disaster Recovery

Backup Strategies

# Velero configuration for GitOps backup
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: argocd-backup
  namespace: velero
spec:
  includedNamespaces:
  - argocd
  includedResources:
  - applications.argoproj.io
  - appprojects.argoproj.io
  storageLocation: aws-backup-location
  ttl: 720h0m0s

Recovery Procedures

# ArgoCD disaster recovery
# 1. Restore ArgoCD from backup
velero restore create argocd-restore --from-backup argocd-backup

# 2. Verify ArgoCD is running
kubectl get pods -n argocd

# 3. Force sync all applications
kubectl get applications -n argocd -o name | xargs kubectl patch -n argocd --type merge -p '{"operation":"sync"}'

Security Best Practices

Access Control

# ArgoCD RBAC configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.csv: |
    p, role:admin, *, *, *, allow
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, */*, allow
    g, my-org:developers, role:developer

Secret Management

# SOPS with ArgoCD
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
spec:
  source:
    repoURL: https://github.com/bitnami-labs/sealed-secrets
    path: .
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Performance Optimization

Sync Optimization

# ArgoCD sync optimization
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: optimized-app
  annotations:
    argocd.argoproj.io/sync-options: PruneLast=true
    argocd.argoproj.io/sync-wave: "5"
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - PrunePropagationPolicy=foreground
    - RespectIgnoreDifferences=true
  source:
    repoURL: https://github.com/myorg/my-app
    path: .
    kustomize:
      commonLabels:
        app.kubernetes.io/managed-by: argocd

Resource Management

# Kustomize resource optimization
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: production
commonLabels:
  environment: production
  team: platform
commonAnnotations:
  argocd.argoproj.io/sync-wave: "10"
resources:
- ../../base
patchesStrategicMerge:
- deployment-patch.yaml
- service-patch.yaml
images:
- name: my-app
  newTag: v1.2.3

Troubleshooting Common Issues

Sync Failures

# Debug ArgoCD sync issues
# Check application status
argocd app get my-app

# View sync history
argocd app history my-app

# Force refresh
argocd app get my-app --hard-refresh

# View logs
kubectl logs -n argocd deployment/argocd-application-controller

Configuration Drift

# Detect configuration drift
kubectl get applications -n argocd

# Force sync to correct drift
argocd app sync my-app

# Check for differences
argocd app diff my-app

Performance Issues

# Monitor ArgoCD performance
kubectl top pods -n argocd

# Check resource usage
kubectl describe deployment argocd-application-controller -n argocd

# Scale ArgoCD components
kubectl scale deployment argocd-application-controller --replicas=3 -n argocd

Advanced Patterns

GitOps at Scale

  • ApplicationSets: Manage multiple applications with templates
  • App of Apps: Hierarchical application management
  • Clusters as Code: Manage multiple clusters from Git

Multi-Cloud GitOps

# Crossplane with ArgoCD for multi-cloud
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: crossplane-aws
spec:
  source:
    repoURL: https://github.com/myorg/infrastructure
    path: aws
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: crossplane-system

GitOps with Machine Learning

# ML model deployment with GitOps
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ml-model
spec:
  source:
    repoURL: https://github.com/myorg/ml-models
    path: production
    targetRevision: HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: ml-production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Conclusion

Advanced GitOps workflows provide the foundation for reliable, scalable, and secure application delivery. By implementing these patterns and best practices, organizations can achieve:

  • Reliability: Automated reconciliation ensures consistent deployments
  • Security: Complete audit trail and policy enforcement
  • Scalability: Patterns that work across teams and environments
  • Observability: Full visibility into deployment status and health

The key to successful GitOps adoption is starting with core principles and gradually incorporating advanced patterns as your organization matures.