Monitoring LND with systemd, Without Containers (1) - Exporting LND Metrics with lndmon

This series picks up where Running LND with systemd, Without Containers left off: two LND nodes in one Lima VM, lnc (A) and lnb (B), with an open channel and a settled payment. A holds 986,530 sat in the channel and B holds 10,000.

The next step is practicing failures: stopping a node, breaking its connection to Bitcoin Core, and seeing how it recovers. Before breaking anything, I want a record of what healthy looks like, so the first part of this series sets up monitoring, with the same rule as before: binaries and systemd, no containers.

The stack has three pieces:

Put together, inside the one Lima VM:

flowchart LR
  subgraph VM["Lima VM: everything on 127.0.0.1"]
    direction LR
    LA["lnd (node A)<br/>gRPC :10009"]
    LB["lnd-b (node B)<br/>gRPC :10010"]
    MA["lndmon-a<br/>metrics :9092"]
    MB["lndmon-b<br/>metrics :9093"]
    P["Prometheus<br/>:9090"]
    G["Grafana<br/>:3000"]
    MA -- "gRPC + TLS<br/>readonly.macaroon" --> LA
    MB -- "gRPC + TLS<br/>readonly.macaroon" --> LB
    P -- "HTTP GET /metrics<br/>every 15s" --> MA
    P -- "HTTP GET /metrics<br/>every 15s" --> MB
    G -- "PromQL queries" --> P
  end
  U["Browser on the Mac"] --> G

Each arrow points from the side that starts the connection. Three things the diagram makes explicit:

Grafana never talks to the nodes or to lndmon. It only asks Prometheus, so everything on a dashboard comes from what Prometheus has stored.

lndmon on its own doesn’t store or display anything; it turns a node’s state into numbers Prometheus can collect. Together, the three make it easy to see a node’s current status and spot abnormal activity, and later to alert on it. I covered the Kubernetes version of the same idea in Metrics Collector: Prometheus on LND.

Installing lndmon

Getting the source

lndmon is built from source, so the first step is the repository:

cd ~
command -v git go
mkdir -p ~/src
cd ~/src
git clone https://github.com/lightninglabs/lndmon.git
cd lndmon
git rev-parse HEAD

Building and installing it

With Git already present and Go installed, building lndmon is one go build, and installing it is the same install used for bitcoind and lnd:

command -v git go
git --version
go version

cd ~/src/lndmon
git rev-parse HEAD

go build -mod=readonly -o ./lndmon ./cmd/lndmon

sudo install -o root -g root -m 0755 ./lndmon /usr/local/bin/lndmon

command -v lndmon
lndmon --help

command -v finds lndmon at /usr/local/bin/lndmon, and --help runs and prints its options, so the binary works.

Go compiles lndmon into one self-contained binary, so the build tools stay in my home directory and only the finished lndmon goes into /usr/local/bin. The service will never need Go or the source tree to run.

Reading lndmon’s options

The full --help output shows what lndmon needs to be told:

One port detail is easy to misread. 9092 is lndmon’s own metrics port, the address Prometheus scrapes, not Prometheus itself, which serves its own interface on another port. With two nodes in the same VM, two lndmon instances can’t both listen on 9092:

Connecting lndmon to LND

The two files lndmon needs

lndmon talks to LND over the same gRPC API lncli uses, and that connection is TLS-encrypted. Two files from LND’s data directory make it work:

TLS and the macaroon answer different questions. TLS protects the connection and proves the server’s identity; the macaroon proves the client is allowed to ask. Only the certificate leaves LND’s directory, never tls.key, the private key that goes with it.

Both files were created by LND under /var/lib/lnd, the 0700 directory only lnd can enter. Before copying them anywhere, I checked that they exist where I expected, as the lnd user:

sudo -u lnd sh -c '
  for file in \
    /var/lib/lnd/tls.cert \
    /var/lib/lnd/data/chain/bitcoin/regtest/readonly.macaroon
  do
    if test -r "$file"; then
      printf "readable: %s\n" "$file"
    else
      printf "not readable: %s\n" "$file"
    fi
  done
'

The script’s messages are in Korean in the screenshot; “읽기 가능” means “readable”.

test -r is true when the file exists and the current user can read it. Running the loop through sudo -u lnd sh -c makes lnd the current user, so both readable lines confirm the files are there and belong to LND’s side. The macaroon sits under the chain and network, bitcoin/regtest, because LND keeps separate macaroons per network.

A first run by hand

Before writing any service, a quick manual run shows whether lndmon can reach LND at all. For this one test I ran it as lnd, which can already read both files where they are:

sudo -u lnd lndmon \
  --lnd.host=127.0.0.1:10009 \
  --lnd.network=regtest \
  --lnd.tlspath=/var/lib/lnd/tls.cert \
  --lnd.macaroondir=/var/lib/lnd/data/chain/bitcoin/regtest \
  --lnd.macaroonname=readonly.macaroon \
  --prometheus.listenaddr=127.0.0.1:9092 \
  --prometheus.logdir=/var/lib/lnd/lndmon-log

Every option from the table above is set for node A: its gRPC address, regtest instead of the mainnet default, its certificate and read-only macaroon, metrics on 127.0.0.1:9092, and a log directory lnd can write to.

So lndmon can talk to LND over TLS with the read-only macaroon, and it’s serving metrics. Nothing is collecting them yet; that’s Prometheus’s job, once it’s installed.

Running as lnd let lndmon read both files in place, without copying anything.

Checking the metrics endpoint

Prometheus active! is lndmon’s own claim. With lndmon still running in the first terminal, I opened a second shell into the VM from the Mac and fetched the endpoint directly, the way Prometheus will:

limactl shell local-lnd
curl -fsS http://127.0.0.1:9092/metrics | head -n 30

The endpoint answers in the Prometheus text format covered in Metrics Collector: Prometheus on LND: a # HELP line describing each metric, a # TYPE line saying whether it’s a gauge, counter, or summary, then the values.

These first lines are about lndmon itself rather than the node. Every Go program that exposes Prometheus metrics gets the go_* set for free: garbage collection pauses, goroutines, memory. go_info{version="go1.26.0"} also records the Go toolchain lndmon was built with. The node’s own numbers, its channels, peers, balances, and chain state, come further down the same output, and those are what the dashboard will use.

The node’s own metrics

Filtering out the comments and lndmon’s own process metrics leaves what lndmon learned from LND:

curl -fsS http://127.0.0.1:9092/metrics |
  rg -v '^(#|go_|process_|promhttp_|$)'

rg -v prints every line that doesn’t match the pattern, so it drops comment lines, blank lines, and the go_, process_, and promhttp_ metrics that describe lndmon rather than the node.

Every number here is something the previous series already saw through lncli or bitcoin-cli, now as a metric:

Some of these rows come from further down the same output than the screenshot shows.

Two cautions when reading them. bandwidth_outgoing_sat matches the channel balance, but it isn’t the most A can pay in one go: the 10,000 sat channel reserve and the commitment fee still limit that, as the previous series showed. And the full output also has network graph statistics, some of which show NaN or 1.7976931348623157e+308, the largest possible floating-point number. Those shouldn’t be read as fee values. How lndmon computes those statistics is something to check separately; on a graph of two nodes and one channel they don’t describe anything useful yet. I’m not charting them, and the dashboard sticks to the node and channel metrics above.

The per-channel metrics carry labels that identify the channel: chan_id="115448720982016" is the short channel ID that payinvoice showed as CHAN_OUT (block 105, transaction 1, output 0), peer is node B’s public key, initiator="true" says A opened it, and status="active".

These are the numbers the dashboard will chart, and the baseline from the end of the previous series, A at 986,530 sat and B at 10,000, is already in them.

Running lndmon with systemd

The unit for node A

With the manual run working, the same command goes into a unit file, lndmon-a.service, built like the bitcoind and lnd units from the previous series:

sudo tee /etc/systemd/system/lndmon-a.service >/dev/null <<'EOF'
[Unit]
Description=LND A metrics exporter
Wants=lnd.service
After=lnd.service
StartLimitIntervalSec=0

[Service]
Type=simple
User=lnd
Group=lnd
ExecStart=/usr/local/bin/lndmon \
  --lnd.host=127.0.0.1:10009 \
  --lnd.network=regtest \
  --lnd.tlspath=/var/lib/lnd/tls.cert \
  --lnd.macaroondir=/var/lib/lnd/data/chain/bitcoin/regtest \
  --lnd.macaroonname=readonly.macaroon \
  --prometheus.listenaddr=127.0.0.1:9092 \
  --prometheus.logdir=/var/lib/lnd/lndmon-log
Restart=on-failure
RestartSec=5
UMask=0077
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=full

[Install]
WantedBy=multi-user.target
EOF

[Unit], how this service relates to others:

[Service], how lndmon runs:

[Install]:

Running lndmon as lnd is the simple choice: no copies, nothing to keep in sync when LND regenerates its certificate. The trade-off is that lndmon then has everything lnd can read, including admin.macaroon and the wallet database, even though it only ever uses the read-only macaroon. The tighter version is a separate lndmon account holding copies of just tls.cert and readonly.macaroon, so a compromised exporter couldn’t reach anything else. The ProtectSystem and ProtectHome lines limit what it can do outside /var/lib/lnd, but not inside it.

Checking the unit

The same check as for every unit before it, then a reload so systemd knows the new file:

sudo systemd-analyze verify /etc/systemd/system/lndmon-a.service
sudo systemctl daemon-reload

Nothing in lndmon-a.service was flagged. The two warnings are the same Ubuntu XFS units that appeared when verifying bitcoind.service and lnd.service in the previous series, and daemon-reload returned silently.

Starting it and checking the metrics

With the manual run from earlier stopped, so port 9092 is free again, systemd starts lndmon as a service. status shows how it’s running, and a narrower curl checks the four metrics the first dashboard will lean on:

sudo systemctl start lndmon-a
sudo systemctl status lndmon-a --no-pager -l
curl -fsS http://127.0.0.1:9092/metrics |
  rg '^lnd_(chain_synced|peer_count|channels_active_total|channels_open_balance_sat) '

The trailing space in the rg pattern matters: it matches only metric names followed directly by their value, so it skips longer names that start the same way and the labeled per-channel lines. The service-run lndmon reports the same state the manual run did, so node A’s exporter is in place.

Then enable, so node A’s exporter comes back on its own after a reboot:

sudo systemctl enable lndmon-a

The symlink in multi-user.target.wants/ is the same mechanism as for bitcoind in the previous series: at boot, systemd starts everything linked there. Because the unit also says Wants=lnd.service, booting into lndmon-a pulls in LND too.

Node B

Node B is the second LND from the previous series, set up the same way under its own account, lnd-b, with its data in /var/lib/lnd-b. Its exporter follows A’s pattern, so the first step is the same file check, now as lnd-b:

sudo -u lnd-b sh -c '
  for file in \
    /var/lib/lnd-b/tls.cert \
    /var/lib/lnd-b/data/chain/bitcoin/regtest/readonly.macaroon
  do
    if test -r "$file"; then
      printf "readable: %s\n" "$file"
    else
      printf "not readable: %s\n" "$file"
    fi
  done
'

As before, “읽기 가능” in the screenshot means “readable”.

Both of node B’s files are there and readable by lnd-b. They are B’s own certificate and macaroon, not copies of A’s: each LND generates its own when it first starts, and B’s exporter has to present B’s macaroon to B’s gRPC port.

Then the same manual run as for A, with B’s values:

sudo -u lnd-b lndmon \
  --lnd.host=127.0.0.1:10010 \
  --lnd.network=regtest \
  --lnd.tlspath=/var/lib/lnd-b/tls.cert \
  --lnd.macaroondir=/var/lib/lnd-b/data/chain/bitcoin/regtest \
  --lnd.macaroonname=readonly.macaroon \
  --prometheus.listenaddr=127.0.0.1:9093 \
  --prometheus.logdir=/var/lib/lnd-b/lndmon-log

Everything that names a node changes; --lnd.network and the macaroon name stay the same. The startup log matches A’s, ending in Prometheus active!.

From a second shell, B’s endpoint on 9093:

curl -fsS http://127.0.0.1:9093/metrics |
  rg '^lnd_(chain_synced|peer_count|channels_active_total|channels_open_balance_sat) '

The two exporters describe the two ends of one channel. Both nodes are synced, each has one peer and one active channel, and the balances are the previous series’ baseline split exactly: 986,530 sat on A’s side, 10,000 on B’s. Together they account for the channel’s 996,530 sat after the commitment costs.

B’s unit is A’s with the node-specific values swapped:

Everything else, the restart policy, StartLimitIntervalSec=0, and the four hardening lines, is identical, and means the same as in A’s tables. Each exporter depends on its own node’s LND, so stopping node B later will affect lndmon-b and leave lndmon-a alone.

Then the same steps as for A, with the manual run stopped first so port 9093 is free:

sudo systemd-analyze verify /etc/systemd/system/lndmon-b.service
sudo systemctl daemon-reload
sudo systemctl start lndmon-b
systemctl status lndmon-b --no-pager

verify flagged nothing new, and lndmon-b is active (running) as its own process, PID 34686, next to A’s exporter.

This status ran without -l, and it shows why that flag has been on every other status in these series: long lines are cut off with …, including the command line and the journal messages, and systemd prints a hint to use -l. The start-up messages can still be read in full with journalctl -u lndmon-b.

The service-run exporter on 9093 reports what the manual run did:

curl -fsS http://127.0.0.1:9093/metrics |
  rg '^lnd_(chain_synced|peer_count|channels_active_total|channels_open_balance_sat) '

Synced, one peer, one active channel, 10,000 sat on B’s side. Both exporters now run under systemd, A’s on 9092 and B’s on 9093, ready for Prometheus to scrape.

Where this part ends

Both nodes now export their state as Prometheus metrics, each from its own lndmon service on loopback: node A on 127.0.0.1:9092, node B on 127.0.0.1:9093. The numbers match what lncli showed at the end of the previous series. Nothing is collecting them yet; that’s Prometheus, in the next part.

Next: Scraping lndmon with Prometheus.