Architecture

ATC operates as a lightweight stateless daemon that coordinates traffic routing policies directly through the Consul Config Entries API. It splits responsibility into two main operational loops: the Forwarder and the Redirector.

Operational Modes

1. Active State (Failover Routing)

When instances of a service exist in the local catalog, ATC's Forwarder ensures a service-resolver config entry exists. If a failover strategy is specified via tags, the forwarder configures Consul to forward requests to backup datacenters or standby services if local instances fail healthchecks or connection timeouts are reached.

2. Offline State (Redirect Routing)

When a service goes completely offline (all instances deregistered from Consul), catalog tags disappear. The Redirector detects this deletion. It retrieves the service's redirect policy from its in-memory cache (or falls back to metadata on the existing config entry) and rewrites the Consul service-resolver entry to a Redirect type, gracefully routing traffic to the backup target.

Installation & Quickstart

ATC is written in Go and compiles to a single static binary. It can be run on VM-based environments alongside Consul agents or deployed to Kubernetes containers.

1. Build from Source

Building ATC requires Go 1.22+ and Node.js (for compiling the embedded live dashboard frontend).

# Clone the repository
git clone https://github.com/atcprojectio/atc.git
cd atc

# Compile the frontend and Go binary
make build

This generates the static binary under dist/atc.

2. Install using Homebrew

ATC can be installed on macOS and Linux using Homebrew via the official tap:

# Tap the repository
brew tap atcprojectio/tap

# Install ATC
brew install atc

3. Running the Daemon

Launch the server command by passing the path to the predefined strategies configuration file:

atc server --config deploy/strategies.yaml

Available CLI Flags

4. Deploying to Kubernetes (Helm)

An official versioned Helm chart is published directly to GitHub Container Registry (GHCR) as an OCI artifact. You can install it on your Kubernetes cluster using the following command:

helm install atc oci://ghcr.io/atcprojectio/charts/atc --version <version>

Alternatively, you can install it using the local templates provided under deploy/helm/atc:

helm install atc ./deploy/helm/atc --values ./deploy/helm/atc/values.yaml

Configuration settings like Consul addresses, predefined strategies, and resources are configurable in the values.yaml file (or using the --set flag with OCI installs).

5. Deploying to Nomad

A Nomad job definition is available at deploy/nomad/atc.nomad.hcl. To run ATC in an active-passive HA mode with two instances, run:

nomad job run ./deploy/nomad/atc.nomad.hcl

Consul 2.0.1 Demo Project

The ATC ecosystem includes a fully automated, multi-datacenter federated Consul 2.0.1 demo environment hosted in the atc-demo repository. It showcases how ATC dynamically generates and updates service-resolver configuration entries during service failovers and redirects.

Demo Features

• Consul 2.0.1 WAN Federation: Runs two Consul servers (dc1 and dc2) federated over WAN.
• HTTP Echo Servers: Mock payment services running in both datacenters to simulate actual routing targets.
• Smart Traffic Client: A lightweight Python traffic client that queries Consul and fetches the active routing output, verifying routing decisions in real-time.
• Automated CLI Walkthrough: A single command (make run-demo) runs a complete failover, redirect, recovery, and purge lifecycle.

Running the Demo

Ensure you have Docker and Python 3 installed, then clone the repository and execute the following tasks:

# Clone the demo repository
git clone https://github.com/atcprojectio/atc-demo.git
cd atc-demo

# Pull the ATC and Consul Docker images
make pull

# Start the Consul 2.0.1 cluster and Mock backends
make up

# Run the interactive routing walkthrough
make run-demo

Demo Lifecycle Output

When running the walkthrough, you will see output demonstrating dynamic failover:

1. Local DC1 Healthy: Traffic resolves locally and prints:
   [Traffic Router] Local DC1 healthy -> Routing to payment-service in DC1... [DC1 Instance] Payment processed successfully!
2. Simulated Outage (DC1 offline): ATC immediately rewrites the resolver to a Redirect to dc2. Traffic client prints:
   [Traffic Router] Redirect Rule Active -> Target DC: dc2, Service: payment-service [DC2 Instance] Payment processed successfully!
3. Recovery (DC1 restored): ATC restores the Failover rule and traffic client resumes local routing.

Strategies Configuration

ATC admins define predefined routing strategies in a YAML configuration file. This file separates policies into failover strategies (for active services) and redirect strategies (for deleted/offline services).

YAML Schema

# Consul connection overrides
consul_namespace: "finance" # Scopes all config/watcher actions to this namespace
write_rate_limit: "1s"      # Coalesce events to prevent API write thrashing

strategies:
  failover:
    # A standard failover strategy routing to another datacenter
    standard-failover:
      connect_timeout: "10s"
      targets:
        - datacenter: "dc2"
          namespace: "finance" # Optional target namespace scope

    # Multi-datacenter targets with fallback services
    multi-region-failover:
      connect_timeout: "5s"
      targets:
        - datacenter: "dc2"
        - datacenter: "dc3"
          service: "fallback-service"

  redirect:
    # Redirect traffic to the same service name in dc2
    standard-redirect:
      datacenter: "dc2"
      namespace: "marketing" # Optional target namespace redirect

    # Redirect to a disaster recovery web service in dc3
    geo-redirect:
      service: "geo-fallback"
      datacenter: "dc3"

Strategy Fields

Failover Strategy

• connect_timeout: The connect timeout duration (e.g. 5s, 10s) applied to the resolver.
• targets: A list of targets to failover to. If service is omitted, it defaults to the active service's name. If datacenter is omitted, it defaults to the primary fallback datacenter. If namespace is specified, it scopes the target resolver to that namespace.

Redirect Strategy

• service: The target service to redirect traffic to. Defaults to the offline service's name.
• datacenter: The target datacenter for redirection.
• namespace: The target namespace for redirection.

Configuration Validation & Linting (GitOps)

ATC provides a configuration validation tool to verify strategy configuration syntax and semantic rules before deploying changes to production. This is highly recommended for GitOps workflows to prevent loading invalid configurations at runtime.

To validate a configuration file, run the validate subcommand:

atc validate --config deploy/strategies.yaml

If the configuration is valid, it prints a success message and exits with status 0. If any validation rules fail, it prints detailed error messages to standard error and exits with status 1.

Validation Rules

• Dampening Periods: dampening_period and min_dampening_period must be valid Go duration strings (e.g. 5s, 1m). Dampening period cannot be less than the min dampening period.
• HA Leader Election: If HA is enabled, session_ttl must be a valid Go duration string between 10s and 24h (which corresponds to Consul's session limits).
• Failover Strategies: Each strategy's connect_timeout (if specified) must be a valid Go duration string.
• Redirect Strategies: Each strategy must define at least one target selector field (such as service, datacenter, namespace, or service_subset).

CI/CD Integration Example

Add a validation check to your GitHub Actions pre-commit or deployment workflow to lint strategies configuration files:

name: Lint ATC Config
on: [pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate ATC Configuration
        run: |
          go run main.go validate --config deploy/strategies.yaml

Consul Service Integration

Service teams opt-in to ATC automation and associate failover/redirect strategies using tags in their Consul service registration.

Opt-in Tags

• atc.enabled=true: Enrolls the service in ATC reconciliation. Without this tag, ATC will not touch the service.
• atc.failover=<strategy-name>: Binds the service to a predefined failover strategy defined in the config. If omitted, and a strategy named default is defined under strategies.failover, ATC automatically falls back to that default strategy.
• atc.redirect=<strategy-name>: Binds the service to a predefined redirect strategy defined in the config. If omitted, and a strategy named default is defined under strategies.redirect, ATC automatically falls back to that default strategy.

YAML Registration Example

service:
  name: "payments-service"
  port: 8443
  tags:
    - "atc.enabled=true"
    - "atc.failover=multi-region-failover"
    - "atc.redirect=geo-redirect"

Consul Metadata Management

ATC ensures that Consul config entry metadata conforms strictly to the strategy currently in effect in the service resolver:

• When Active: The resolver is configured for failover. The metadata reports failover-strategy and removes redirect-strategy.
• When Offline: The resolver is configured for redirect. The metadata reports redirect-strategy and removes failover-strategy.

Oscillation Dampening (Hysteresis)

ATC includes built-in oscillation dampening (hysteresis) to prevent configuration churn in Consul. If a service's health flaps or catalog state oscillates rapidly, ATC debounces the writes to protect the Consul leader from excessive write operations.

How Hysteresis Works

When a state change occurs (e.g. service goes offline or recovers), ATC schedules the configuration write after a dampening period. If a new state change arrives before the dampening timer fires (e.g. the service recovers during the dampening window), the pending write is canceled, and the transition is debounced.

Dampening Settings

• Global Default: Set via dampening_period in the main config (e.g. 5s).
• Service Tag Override: Bypassed or customized on a per-service basis using the tag atc.dampening=<duration> (e.g. atc.dampening=10s or atc.dampening=0s).
• Immediate Updates: A tag value of 0s or 0 disables dampening, triggering immediate failover or redirects.
• Safety Boundary: Set via min_dampening_period in the main config. All custom tags are clamped to this minimum limit to prevent users from bypassing operator safety settings.

Active-Passive High Availability

To run ATC in a high-availability production environment, multiple ATC daemons can be deployed in active-passive mode. Leadership is coordinated dynamically through Consul KV Session locks.

HA Architecture

• Target-Scoped Locks: Leadership is scope-isolated per reconciler component (e.g., atc/leader/lock/forwarder and atc/leader/lock/redirector). Node instances running a single component compete only for that component's lock. Node instances running multiple components compete for and acquire locks for each workload independently, preventing split-workload deadlocks and workload starvation.
• Active Workload: Reconcilers for a component run only when the local node acquires the leadership lock for that component.
• Standby Nodes: Passive standby instances keep their HTTP/metrics servers active to serve read-only APIs and dashboards, but suspend reconciler watches for any modules where they do not hold the lock.
• Failover: If the active instance for a workload fails or experiences network partitions, the Consul session lock expires (governed by the session TTL). A standby instance instantly acquires leadership for that component and resumes reconciliation with zero data loss.

HA Configuration

ha:
  enabled: true
  lock_key: "atc/leader/lock"
  session_ttl: "15s"

HA Leadership CLI Commands

ATC provides dedicated administrative subcommands under the leader namespace to query status and force stand-down lock owners:

1. Query Leadership Status

Prints module-level leadership states, active lock keys, current leader nodes, and session IDs:

atc leader status

2. Force Unlock Module Locks

Forces the stand-down of the active leader of a component by destroying its associated Consul session:

atc leader force-unlock --module=<forwarder|redirector>

WAN Federation Indicators

ATC queries the local Consul agent's WAN coordinate/member list (via the `/api/federation` endpoint) to verify that configured target datacenters are WAN-federated and reachable. Status badges are rendered directly in the failover visualization path on the dashboard:

• ● Green Indicator: Datacenter is healthy and WAN-federated (status is alive).
• ▲ Red Indicator: Gossip connection to the datacenter has failed (status is failed).
• ⚠️ Yellow Warning: Datacenter is completely missing from WAN members list (status is unfederated).

Advanced Features

ATC includes advanced configuration capabilities for enterprise workloads, including dry-run simulation, dynamic hot-reloading, write rate limiting, and override expiration TTLs.

1. Dry-Run Mode

In dry-run mode, ATC runs the watcher and reconciler engines normally but bypasses writing or deleting configuration entries in the Consul KV registry. All actions that would be taken are written to the logs with a [DRY RUN] prefix. Enable dry-run via configuration or CLI flags:

dry_run: true

atc server --config deploy/strategies.yaml --dry-run

2. Dynamic Configuration Hot-Reloading

ATC supports watching its central configuration files for changes. If strategies, namespaces, or rate limits are updated in the configuration file, the changes are applied automatically without restarting the process. This is governed by the fsnotify watch loops. Alternatively, operators can trigger hot-reloading by:

• Sending a SIGHUP signal to the running daemon process.
• Sending an authenticated POST request to the /api/reload endpoint.
• Calling the reload_config Model Context Protocol (MCP) tool.

3. Manual Override Expiration (TTL)

When applying a manual failover override or redirect via the UI dashboard, REST API, or MCP tools, a Time-To-Live (TTL) duration (e.g. 5m, 1h, 24h) can be specified. ATC records the expiration timestamp in the configuration entry's metadata as atc-override-expires-at. A background sweeper task checks active overrides and automatically purges them once expired, restoring default routing policies.

4. Write Rate Limiting

To prevent API write thrashing and protect the Consul cluster from overloading in large environments, a token-bucket rate limiter is applied to all state-changing Consul API updates. Rate limits can be configured in the YAML file or via command-line flags:

# Limit write operations to Consul to 10 per second
write_rate_limit: "10/s"

atc server --config deploy/strategies.yaml --write-rate-limit="10/s"

Forwarder Module

The Forwarder manages the active state of enrolled services in the Consul registry. It ensures that the mesh routing layers are prepared to divert failing traffic instantly.

Behavior Loop

1. Listens to Consul catalog updates from the watcher.
2. Identifies active services tagged with atc.enabled=true.
3. Caches configured strategy tags (failover & redirect) in-memory for downstream recovery.
4. Evaluates the designated atc.failover strategy. If the strategy contains no custom targets, it defaults to routing to the primary remote datacenter.
5. Checks if the existing service-resolver entry in Consul matches the strategy. If it does not, or does not exist, the forwarder updates/creates it.
6. Sets metadata to only include failover-strategy, ensuring the inactive redirect policy is not reported in Consul.

Redirector Module

The Redirector handles the lifecycle transition when a service goes completely offline. It switches the service-resolver from load-balancing mode to direct redirection.

Behavior Loop

1. Listens to catalog updates and identifies services that have been deleted/deregistered from the catalog.
2. Filters deleted services that possess a resolver entry created by atc.
3. Determines the redirect strategy:
   • Reads the strategy name from the in-memory cache populated while the service was active.
   • If the cache is empty (e.g. following a daemon restart), it falls back to the last known redirect-strategy in the resolver entry metadata.
4. Updates the Consul service-resolver entry to a Redirect resolver block.
5. Sets the metadata map on the entry to only report redirect-strategy, removing failover-strategy.

OpenTelemetry Instrumentation

ATC is built with native cloud-observability support. It uses the OpenTelemetry SDK to emit structured logs, standard runtime and application metrics, and distributed trace spans.

Trace Spans

All reconciliation runs, watchers, and API handlers generate tracing spans. They propagate context using standard headers, allowing developers to trace reconciliation latency and Consul client overhead.

Metrics Exposed

Local Observability Stack

The ATC developer Docker Compose environment is maintained in the atc-demo repository. It spins up a fully integrated sandbox containing Consul, ATC, and the LGTM (Loki, Grafana, Tempo, Mimir/Prometheus) observability stack.

Starting the Stack

To run the developer environment, clone the demo repository and bring up the services:

# Clone the demo repository
git clone https://github.com/atcprojectio/atc-demo.git
cd atc-demo

# Spin up Consul infrastructure and mock services
make up-infra

# Spin up the LGTM observability stack
make up-obs

# (Optional) Spin up the containerized ATC daemons
make up-atc

This launches the following endpoints locally:

• Grafana: http://localhost:3000 (preconfigured with Loki, Prometheus, and Tempo datasources)
• Consul UI: http://localhost:8500
• ATC Dashboard: http://localhost:8088
• Prometheus: http://localhost:9090

Docker Architecture

The ATC binary sends OTLP metrics, traces, and logs directly to the OpenTelemetry Collector container listening on port 4317 (gRPC) and 4318 (HTTP). The Collector filters and distributes these signals to Prometheus, Tempo, and Loki respectively.

Live Dashboard UI

The ATC Web Dashboard is a premium glassmorphic React interface embedded directly inside the Go binary. It provides ATC administrators and platform teams with a real-time visualization of service mesh routing paths.

Features

• Real-time status indicators: Instantly view if a service is actively balancing local instances, failing over, or redirecting.
• Dynamic target path visualizers: Traces and visualizes the target paths of active failover configurations or redirect paths.
• Orphan Purging: Allows platform administrators to clean up orphaned resolver configurations from Consul directly via the dashboard with a single click.

API Reference

ATC exposes REST API endpoints for monitoring and lifecycle actions on port 8088.

1. GET /api/services

Returns a list of all services enrolled in ATC, their current operational status, active resolver types, strategy definitions, and targets.

Response Schema

[
  {
    "name": "test-service",
    "tags": [
      "atc.enabled=true",
      "primary",
      "atc.failover=multi-region-failover",
      "atc.redirect=geo-redirect"
    ],
    "resolver_type": "failover",
    "status": "active",
    "failover_strategy": "multi-region-failover",
    "redirect_strategy": "geo-redirect",
    "failover_targets": [
      {
        "service": "test-service",
        "datacenter": "dc2"
      },
      {
        "service": "fallback-service",
        "datacenter": "dc3"
      }
    ]
  }
]

2. DELETE /api/services?name=<service-name>

Purges the service-resolver configuration entry for the specified service from Consul. Used to clean up orphaned entries or clear manual overrides.

Query Parameters

• name (required): The name of the service to purge.

3. POST /api/overrides

Manually overrides service routing behavior. Writes a Consul service-resolver configuration entry tagged with "created-by": "atc-override", causing ATC automated reconcilers to bypass this service. To restore automated watcher reconciliation, call the DELETE /api/services?name=<service-name> endpoint.

Request Body

{
  "service": "payment-service",
  "type": "failover|redirect",
  "target_dc": "dc2",
  "duration": "1h"
}

The optional duration parameter specifies a Time-To-Live (TTL) for the manual override (e.g. "5m", "15m", "1h", "24h"). Once expired, a background task loop automatically purges the override resolver from Consul.

Response Codes

• 200 OK: Override successfully applied.
• 400 Bad Request: Invalid body, missing fields, or invalid type/duration format.
• 500 Internal Server Error: Consul write failed.

4. POST /api/reload

Dynamically triggers a configuration file hot-reload, re-reading the active config file from disk, unmarshalling new settings, and dynamically propagating them to all submodules (forwarder and redirector) without restarting the daemon process.

Response Codes

• 200 OK: Configuration successfully reloaded.
• 500 Internal Server Error: Reload failed or file reading error.

5. GET /api/strategies

Returns the JSON representation of all predefined failover and redirect strategy templates configured in the ATC daemon.

Response Schema

{
  "failover": {
    "standard-failover": {
      "connect_timeout": "15s",
      "targets": [
        {
          "service": "payment-service",
          "datacenter": "dc2"
        }
      ]
    }
  },
  "redirect": {
    "standard-redirect": {
      "service": "payment-service",
      "datacenter": "dc2"
    }
  }
}

6. GET /api/leader

Returns the detailed JSON representation of the cluster leadership status, current active leader node names, and component-specific Consul lock keys and sessions.

Response Schema

{
  "leader": true,
  "auth_enabled": false,
  "components": {
    "forwarder": true,
    "redirector": true
  },
  "local_node": "atc-node-01",
  "modules": {
    "forwarder": {
      "is_leader": true,
      "leader_node": "atc-node-01",
      "lock_key": "atc/leader/lock/forwarder",
      "session_id": "c1a9a8be-e086-455b-b9d9-2ea11e1f76f7"
}

7. GET /api/modules

Returns a JSON array of all active reconciliation modules enabled on this ATC daemon instance.

Response Schema

[
  "forwarder",
  "redirector"
]

8. GET /api/federation

Returns the WAN federation status of all configured target datacenters, including local Consul agent WAN member states.

Response Schema

{
  "dc2": {
    "status": "alive",
    "rtt": "1.24ms"
  }
}

9. GET /ready

A simple health check endpoint returning HTTP 200 OK if the daemon is running normally and accepting traffic, or HTTP 503 Service Unavailable when a shutdown is in progress.

Response Schema

{
  "status": "ok"
}

10. GET /services

Returns an ASCII representation of the active services catalog and their routing configurations (intended for quick terminal debugging).

Response Headers

Content-Type: text/plain; charset=utf-8

Model Context Protocol (MCP) Server

ATC natively exposes a Model Context Protocol (MCP) server over HTTP at /mcp on a dedicated listener port (default 8092) using Server-Sent Events (SSE). This allows AI coding assistants, autonomous agents, and CLI tools to interact with ATC dynamically, inspect routing policies, and apply manual failovers without socket starvation on the main dashboard port.

SSE Connection Details

The MCP server uses SSE (Server-Sent Events) for bi-directional communication. The initial SSE channel is opened at /mcp, and client tool-calls are posted back to the session-specific endpoint.

# Establish connection (SSE)
curl -N -H "Accept: application/json, text/event-stream" \
     -H "Authorization: Bearer <token>" \
     http://localhost:8092/mcp

# Call tools by posting JSON-RPC requests to the session endpoint

Available MCP Tools

ATC registers the following 10 tools on the MCP server:

Claude Desktop MCP Integration

To enable the ATC Model Context Protocol (MCP) server in Claude Desktop, bridge the Streamable HTTP SSE transport to Claude's stdio interface using the official mcp-remote bridge client.

1. Open the Claude Desktop configuration file:
   • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
   • Windows: %APPDATA%\Claude\claude_desktop_config.json
2. Add the atc configuration block under mcpServers pointing to the remote HTTP SSE endpoint:

{
  "mcpServers": {
    "atc": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "http://localhost:8092/mcp"
      ]
    }
  }
}

Authentication & Security

ATC includes built-in authentication and authorization middleware to secure all REST API and Model Context Protocol (MCP) transport endpoints in production (except the /health and /ready telemetry checks).

1. Enabling Authentication

Authentication settings are configured under the auth section block in the YAML config file:

auth:
  enabled: true
  static_keys:
    - "atc-super-secret-token"
  consul_token_delegation: true

2. Token Passing Methods

Clients can authenticate by passing a token using one of the following methods:

• Authorization Header: Authorization: Bearer <token>
• Custom Headers: X-ATC-Token: <token> or X-Consul-Token: <token>
• Query Parameter: ?token=<token>

3. Validation & Consul RBAC Delegation

When a request is received, ATC validates the token:

1. If the token matches any string in static_keys, authorization succeeds.
2. If consul_token_delegation is enabled, ATC forwards the token to the local Consul agent via the /v1/agent/self API. If Consul accepts the token as valid, authorization succeeds. Furthermore, for Consul read/write queries, ATC dynamically propagates this delegated token in the Consul context, enforcing user-level Consul ACLs natively.

4. High-Priority JSON Audit Logging

All state-changing actions (manual overrides and purges) write structured JSON logs directly to the process's standard output or error stream:

{"time":"2026-06-23T15:02:53+02:00","level":"INFO","msg":"audit event","module":"audit","audit":true,"client_ip":"127.0.0.1","actor":"atc-super...","action":"create_override","target_service":"payment-service","params":{"target_dc":"dc2","type":"failover"}}

Sensitive tokens are automatically masked in the actor attribute to prevent secret exposure in logs.