Monitoring LND with systemd, Without Containers (2) - Scraping lndmon with Prometheus

Part 2 of the series. Part 1 built lndmon and left one exporter per node running under systemd: node A on 127.0.0.1:9092, node B on 127.0.0.1:9093.

Installing Prometheus

Checking what apt offers

Unlike Bitcoin Core, LND, and lndmon, Prometheus is packaged in Ubuntu. Before installing, apt-cache policy shows which version the package would bring:

apt-cache policy prometheus

Two things follow from that. First, a package from Ubuntu’s archive arrives already verified: apt checks the archive’s signature on every download, which replaces the checksum and signature steps the previous series did by hand for Bitcoin Core and LND. Second, universe is the community-maintained part of Ubuntu, and the packaged version is from the Prometheus 2.53 line, not the newest upstream release. For a lab that needs Prometheus to scrape two endpoints and answer queries, that’s enough, and it keeps updates in apt’s hands.

The unit the package ships

The package doesn’t just install a binary. It also creates a prometheus system user and a systemd unit, the two things the previous series wrote by hand for bitcoind and lnd. systemctl cat shows the unit:

systemctl cat prometheus

The file lives in /usr/lib/systemd/system/, where packages put their units, not /etc/systemd/system/ like mine. That matters for changes: editing it directly would be overwritten by the next package update, so local changes go in a drop-in under /etc/systemd/system/prometheus.service.d/, through sudo systemctl edit prometheus.

The service settings:

Then a whole block marked # systemd hardening-options. My units had four hardening lines; this one has twenty:

Prometheus only needs to make HTTP requests, write its database, and serve queries, so the package can afford to take away everything else. The same list is a useful checklist for tightening my own bitcoind, lnd, and lndmon units later.

The flags file

The unit takes its command-line flags from /etc/default/prometheus:

cat /etc/default/prometheus

ARGS="" is empty, so Prometheus runs with its defaults. The comments warn about escaping: the value passes through the shell and then systemd, so a regex backslash has to be written four times. With no flags set, the configuration file, the data directory, and the listen address all come from defaults, and --help shows what they are:

prometheus --help 2>&1 | rg -A 2 -- '--config.file|--storage.tsdb.path|--web.listen-address'

2>&1 is there because --help prints to standard error; it sends that into the pipe. rg -A 2 shows each matching flag plus the two lines after it, and -- tells rg the pattern that follows isn’t one of its own options, since it starts with dashes.

The first two fit the lab as they are. The third doesn’t. 0.0.0.0 means every interface, so Prometheus’s API would be reachable from outside the VM, unlike everything else here, which binds to 127.0.0.1. Keeping the same rule means setting the address explicitly:

ARGS="--web.listen-address=127.0.0.1:9090"

After setting ARGS and restarting Prometheus (a listen address is read only at startup, so reload isn’t enough), ss shows where it actually listens:

sudo ss -lntp '( sport = :9090 )'

Prometheus listens on 127.0.0.1:9090 only, the same as every other service in the lab. The 0.0.0.0:* under “Peer Address” isn’t the listen address; it means any peer may connect, which on a loopback socket means only programs inside the VM.

The sample configuration

The scrape targets go in /etc/prometheus/prometheus.yml. The package ships a sample:

cat /etc/prometheus/prometheus.yml

Each job_name becomes a job label on every series that job scrapes, which is how queries tell one kind of target from another.

Two lines in the sample collide with this lab:

• localhost:9093 under alertmanagers. 9093 is Alertmanager’s usual port, but in this VM it’s where lndmon-b serves node B’s metrics. Left as is, Prometheus would try to deliver alerts to lndmon. There are no alert rules yet, so nothing is sent. For now I’m removing the alerting block entirely, and Alertmanager will get its own port when the alerting part of this series adds it.
• localhost:9100 for the node job. It only works if prometheus-node-exporter is installed and running. Otherwise the target shows as down, which is noise on a page that should show failures clearly.

So the plan for the configuration: keep the prometheus and node jobs, add the two lndmon exporters, A on 9092 and B on 9093, and drop the alerting block.

Backing up the original

Before editing, a copy of the package’s file:

sudo cp -a /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.bak

-a (archive) copies the file’s attributes along with its contents: owner, group, mode, and timestamps. A plain sudo cp would leave the copy owned by root with default permissions, so restoring it later could quietly change who can read the configuration. With -a, putting the backup back restores the file exactly as the package installed it.

The new configuration

I replaced the sample with a shorter file of my own:

sudo tee /etc/prometheus/prometheus.yml >/dev/null <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: node
    static_configs:
      - targets: ['127.0.0.1:9100']

  - job_name: lndmon
    static_configs:
      - targets: ['127.0.0.1:9092']
        labels:
          node: a
      - targets: ['127.0.0.1:9093']
        labels:
          node: b
EOF

Both exporters serve the same kind of metrics, so they share one job, lndmon, and a node label tells them apart. Every series from 9092 carries job="lndmon", node="a", and every series from 9093 job="lndmon", node="b". A dashboard panel can then show lnd_peer_count once per node with a single query, instead of one query per job.

Compared with the sample, localhost became 127.0.0.1, matching every other address in this lab, and three sections are gone: alerting, because of the 9093 collision; rule_files, since there are no rules yet; and external_labels, which only matters when talking to other systems. They come back when the alerting part needs them.

Checking it before Prometheus reads it

The package also installs promtool, Prometheus’s own checker. It plays the part systemd-analyze verify played for unit files:

promtool check config /etc/prometheus/prometheus.yml

SUCCESS means the YAML parses and every key is one Prometheus knows, with valid values. That’s worth checking first because a reload with a broken file doesn’t take Prometheus down; it keeps running on the old configuration and logs the error, which is easy to miss. promtool checks syntax and structure, not whether the targets are reachable. That’s what the targets page shows after the reload.

Reloading and checking the targets

The unit’s ExecReload sends SIGHUP, so the new targets take effect without restarting Prometheus. Then I asked Prometheus itself whether it can reach both exporters, through its query API:

sudo systemctl reload prometheus
curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job="lndmon"}' |
  python3 -m json.tool

The answer has one result per exporter:

Both exporters are being scraped, and each series carries the job and node labels from the configuration, plus instance, the address Prometheus scraped. The first number in value is the time of the sample as a Unix timestamp; the second is the value itself, as a string, which is how the API returns numbers.

up needs reading carefully, though. It says whether Prometheus could scrape lndmon, not whether LND is healthy: an exporter can answer while the node behind it is down. When node B is stopped later in this series, comparing up{node="b"} with B’s LND metrics will show which one notices first.

The same query for the other two jobs:

curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job="node"}' |
  python3 -m json.tool
curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job="prometheus"}' |
  python3 -m json.tool

The node target is up, so the worry from the sample configuration didn’t apply: prometheus-node-exporter is running and serving the VM’s own metrics on 9100. Prometheus is scraping itself too. All four targets, the two lndmon exporters, the host, and Prometheus itself, report up as 1. Only the lndmon results carry a node label, since only those targets were given one.

Where this part ends

Prometheus is collecting from four targets every 15 seconds: both lndmon exporters, labeled node="a" and node="b", the VM’s own metrics through node exporter, and itself. Everything listens on loopback only. The data is there; what’s missing is a view of it, which is Grafana, next.

Next: Dashboards with Grafana.