Skip to content

How varTrack Works

varTrack is built around a simple idea: your Git repository is the source of truth for configuration, and any datasource that holds config values should automatically stay in sync with it.

The pipeline

Every sync — whether triggered by a webhook or the CLI — runs through the same pipeline. A CLI sync brings the file itself, so it skips fetching from Git: it starts at the rule for its datasource and goes straight to parsing.

PAYLOAD       →   ETL             →   SYNC
──────────────────────────────────────────────
resolve rule      fetch from Git      pick strategy
match platform    parse file          write to sinks
                  flatten keys        prune stale keys
                  validate CUE        rollback on error

Stage 1 — Payload

When a webhook arrives at the gateway-service, varTrack resolves:

  • Which platform sent the event (GitHub, GitLab, ...)
  • Which rule applies: the one for the platform that sent the event and the datasource in the webhook URL (a rule is identified by that pair). Its repositories and exclude_repositories decide whether the pushed repository is synced, and file_name or file_path_map which files
  • Which environment to target (from branch name, PR number, tag, or explicit branch_map)

Stage 2 — ETL

The orchestrator-service:

  1. Reads the files at the pushed commit, from a bare mirror of the repository kept under GIT_CACHE_DIR (at most 20 repos, least recently used removed first)
  2. Parses the file (YAML / JSON / TOML / .env / HCL / XML / INI)
  3. Extracts the subtree under the vartrack root key (if present)
  4. Detects the environment key pattern (branch-keyed or flat)
  5. Flattens nested config to { "key.subkey": "value" } using Rust BFS/DFS
  6. Applies variables_map overlay if configured
  7. Validates against the CUE schema for this tenant

Stage 3 — Sync

Writes the validated flat map to every configured sink:

  • Picks the sync strategy (AUTO, GIT_UPSERT_ALL or GIT_SMART_REPAIR)
  • Runs writes to all matching sinks in parallel
  • Prunes stale keys once per environment, after all its files are written, if prune is enabled on the rule
  • Rolls back (deletes written data) if a write fails partway through

Services

Service Language Port Role
gateway-service Go 5657 HTTP webhook receiver, platform routing, gRPC forwarder
orchestrator-service Python + Rust 8000 / 50051 ETL pipeline, CUE validation, multi-sink writer
watcher-service Go 9091 Drift detection, self-heal trigger
vt Go — CLI for manual sync and CI/CD

Data flow diagram

                    ┌──────────────────────────────────────┐
  GitHub push  ───► │          gateway-service             │
                    │  ┌────────────────────────────────┐  │
                    │  │  verify signature               │  │
                    │  │  match rule                     │  │
                    │  │  rate limit                     │  │
                    │  └─────────────┬──────────────────┘  │
                    └────────────────┼─────────────────────┘
                                     │ gRPC
                    ┌────────────────▼─────────────────────┐
                    │        orchestrator-service           │
                    │  ┌────────────────────────────────┐  │
                    │  │  read files at pushed commit    │  │
                    │  │  parse + flatten (Rust)         │  │
                    │  │  validate CUE schema            │  │
                    │  │  write to sinks                 │──┼──► MongoDB
                    │  └────────────────────────────────┘  │──► Redis
                    └──────────────────────────────────────┘──► ZooKeeper
                                                               ──► S3 / ...
                    ┌──────────────────────────────────────┐
                    │          watcher-service              │
                    │  every 60 s, per self_heal rule:     │
                    │  CheckDatasource (gRPC) — the        │
                    │  orchestrator compares sink vs Git   │
                    │  drifted → SyncDatasource (gRPC)     │
                    └──────────────────────────────────────┘

Rust core

Hot-path operations (flattening and merging) are implemented in Rust and exposed to Python via PyO3. Pure-Python fallbacks activate automatically if the Rust extension is not compiled — so the service always works, just slightly slower without it.

Module Purpose
flatten BFS / DFS nested dict → flat key.subkey map
merge Variable map overlay + environment resolution
cue CUE binary subprocess wrapper
diff Added / removed / changed sets of two flat maps (the sinks compare in Python)

Next