Custom Services

The Problem

The monitoring stack needed metrics and automation that no existing tool covered. Business APIs, DNS health checks, dynamic metric generation, automated health discovery, alert silencing based on business state. Each gap got its own service.

Exporters

• DNS Health Exporter - reads Cloud DNS zones and exposes health check status per zone as Prometheus metrics
• Dynamic Metrics Exporter - creates new metrics from label values of existing metrics via YAML config with hot-reload. For example, extracting Kubernetes node version labels into dedicated version tracking metrics
• Environment Status Exporter - scrapes a reporting API to expose which environments are live, active, or in demo mode. This data feeds into alerting and the status dashboard

Data Collection

• Health Check Exporter - auto-discovers services via Kubernetes API across all namespaces, runs concurrent health checks, and pushes structured results to VictoriaLogs. The data source for the Status Dashboard

Automation

• Silence Manager - checks environment business state from an API. If an environment has no live activity, it automatically creates silences in AlertManager to prevent false alerts. Part of the Alert Pipeline

Deployment

All services follow the same pattern - containerized with Docker, deployed via Rancher Fleet, output consumed by VictoriaMetrics or VictoriaLogs. Each one solves a specific problem that couldn't be solved with configuration alone.