Open Sourcing Our Prometheus Client

Of the many open-source projects we leverage, Prometheus is one we use virtually every day, from monitoring our own infrastructure to building a custom metering system. Prometheus is a project we really can't do without.

Not everyone sets out to build a Prometheus client. In our case we started off with the default Golang client, and over time it became clear that our needs quickly outgrew the standard client.

In this blog we will talk about some of our use cases for Prometheus, why we decided to write our own set of libraries, and why we think you should try it out.

The default route

We started out like most people, using prometheus/client_golang and a handful of community remote write libraries. Notably we also tried tally from Uber very briefly before dropping it. This worked fine initially. Metrics were collected, pushed upstream, and life was good.

Then we started building the CostGraph agent.

The agent runs on customer hosts, collects per-process CPU and memory metrics, and ships them to our backend via Prometheus remote write. We had a hard constraint from the start: no external Prometheus dependency. Whoever installs the agent should not need to run a Prometheus server alongside it — essentially, the agent needs to be self-contained.

A major problem is that most of the available clients are not robust.

"Not robust" for our use case means the clients do not handle retries: if there is a network error during a remote write, the metrics collected in that window are just gone, and there is no way to survive a process restart without dropping data. For a monitoring agent that customers depend on, that is not acceptable.

How we solved it

To combat this we created remote-write, a Go client for the Prometheus remote_write protocol which ships with two transport layers:

• Client for immediate synchronous delivery.
• DurableClient for local disk spooling, restart survival, and background draining.

Client

The synchronous client handles the common case. You give it a prometheus.Gatherer and it encodes, compresses, and pushes metrics via remote_write. Protobuf encoding, snappy compression, and automatic batch splitting are handled for you.

import (
    "github.com/prometheus/client_golang/prometheus"
    remotewrite "github.com/baselinehq/remote-write"
)

client, err := remotewrite.New(remotewrite.Config{
    UpstreamURL: "http://localhost:9090/api/v1/write",
})
if err != nil {
    log.Fatal(err)
}
defer client.Close()

err = client.Push(ctx, remotewrite.PushRequest{
    Gatherer: prometheus.DefaultGatherer,
    ExternalLabels: map[string]string{
        "job":      "my-daemon",
        "instance": hostname,
    },
})

The client also supports a Forward API for proxy use cases where you receive an incoming remote_write request and stream it to an upstream. In streaming mode there is no buffering — the body is forwarded directly.

DurableClient

The durable client is what we built for the CostGraph agent. It solves the problem that made us write this library in the first place.

When you call Push or Enqueue, the payload is persisted to a local disk queue before anything is sent upstream. A background drain loop picks up records and ships them with retry and backoff. If the network goes down, payloads accumulate on disk. When connectivity returns, they drain in FIFO order. No metrics are lost.

dc, err := remotewrite.NewDurable(remotewrite.DurableConfig{
    Client: remotewrite.Config{
        UpstreamURL:  "https://localhost:9090/api/v1/push",
        TenantHeader: "X-Scope-OrgID",
        Retry: &remotewrite.RetryConfig{
            MinWait: time.Second,
            MaxWait: 30 * time.Second,
        },
    },
    QueueDir:          "/var/lib/my-agent/remotewrite",
    QueueName:         "metrics",
    MaxInMemoryBlocks: 128,
    MaxPendingBytes:   10 * 1024 * 1024 * 1024, // 10 GiB
    SendConcurrency:   1,
})
if err != nil {
    log.Fatal(err)
}
defer dc.Close()

go dc.Run(context.Background())

// Push periodically
ticker := time.NewTicker(15 * time.Second)
for range ticker.C {
    dc.Push(ctx, remotewrite.PushRequest{
        TenantID: "tenant-a",
        Gatherer: prometheus.DefaultGatherer,
        ExternalLabels: map[string]string{
            "job":      "my-agent",
            "instance": hostname,
        },
    })
}

The durable layer does not require Prometheus internals or a second process. It is self-contained, just like the agent.

On performance

Not only did we want more from our Prometheus client — performance was critical from day one. remote-write is optimized for low allocation counts and minimal GC pressure. A few of the things we do:

• Pooled io.ReadCloser wrappers and timer pools to reduce allocations under sustained load.
• Retry buffers with size classes to avoid repeated large allocations.
• Reusable protobuf, label, and compression output buffers across encode cycles.
• Automatic batch splitting to avoid oversized payloads while minimizing HTTP round trips.

Here's a quick snapshot of some numbers on an Apple M3 Pro:

Closing

The library is available at this repo. We use it in production every day across the CostGraph agent and operator.

If you are working with Prometheus remote write and need something more robust than what is currently available, give it a try. Contributions and feedback are welcome.