How do you observe and alert on GitOps sync failures in production?

GitOps operators expose rich telemetry specifically for monitoring sync and reconciliation health. The two main surfaces are Prometheus metrics and the operators' own notification controllers.

Argo CD observability:

• Argo CD exposes Prometheus metrics on port 8082. Key metrics: argocd_app_info (labels include sync_status and health_status), argocd_app_sync_total{phase="Failed"}, and argocd_app_reconcile_duration_seconds.
• The Argo CD Notifications Controller sends alerts based on trigger conditions (SyncFailed, AppOutOfSync, AppDegraded) to Slack, PagerDuty, email, or any webhook — configured via argocd-notifications-cm ConfigMap.
• argocd app list and argocd app get <name> for CLI-based status checks.

Flux observability:

• Each Flux controller exposes metrics. Key: gotk_reconcile_error_total (per controller, per object name), gotk_reconcile_duration_seconds, gotk_resource_info (with ready and suspended labels).
• flux get all and flux events --watch for real-time status.
• The Flux Notification Controller uses Provider and Alert CRs (stored in Git) to route events from any Flux resource type to any webhook or messaging service.

Example Prometheus alert rule for Argo CD:

- alert: ArgoCDAppSyncFailed
  expr: increase(argocd_app_sync_total{phase="Failed"}[5m]) > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Argo CD sync failed for app {{ $labels.name }}"