Running LND with systemd, Without Containers (3) - Running bitcoind as a systemd Service

Part 3 of the series. Part 2 installed Bitcoin Core and gave it its own user, directories, and configuration.

The unit file

This is where systemd takes over the job Kubernetes used to do. A unit file tells systemd how to start bitcoind, as which user, when to restart it, and what it may touch. I wrote it with tee:

sudo tee /etc/systemd/system/bitcoind.service > /dev/null <<'EOF'
[Unit]
Description=Bitcoin Core regtest node
After=network.target

[Service]
Type=simple
User=bitcoin
Group=bitcoin
ExecStart=/usr/local/bin/bitcoind -conf=/etc/bitcoin/bitcoin.conf -datadir=/var/lib/bitcoind
Restart=on-failure
RestartSec=5
TimeoutStopSec=300
UMask=0077
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=full

[Install]
WantedBy=multi-user.target
EOF

sudo tee writes the file as root, which a plain sudo cat > file can’t do because the redirection runs in my own shell. > /dev/null stops tee from echoing the file back. Quoting 'EOF' keeps the shell from expanding anything inside the text.

Restart=on-failure is the piece that replaces a Kubernetes restart policy, and the last four lines play the role a pod’s securityContext did: they limit what the process can do, enforced by the kernel.

Checking the unit and starting it

Before starting anything, systemd-analyze verify checks the unit file. Then systemd reloads its unit files and starts the service.

sudo systemd-analyze verify /etc/systemd/system/bitcoind.service
sudo systemctl daemon-reload
sudo systemctl start bitcoind

The two warnings aren’t about bitcoind.service. verify also loads units that ship with Ubuntu, and two of them, for XFS filesystem scrubbing, still use CPUAccounting=, an option newer systemd has dropped. Nothing in my unit was flagged, and both daemon-reload and start returned without output, which means they succeeded.

Checking that it’s running

systemctl status shows whether the service started, how it’s running, and its latest log lines:

sudo systemctl status bitcoind --no-pager -l

--no-pager prints straight to the terminal, and -l stops systemd from cutting long lines short.

The log lines are what an isolated regtest node should say. The worker threads start, the DNS seed lookup goes to dummySeed.invalid and finds nothing because regtest has no seeds, and no fixed seeds are added. With listen=0 and no peers, the node is running on its own, which is what this lab wants. The log carries two clocks: the journal’s local time (KST) at the front, and bitcoind’s own UTC timestamp after it.

Talking to it over RPC

A running process isn’t proof that the node answers. bitcoin-cli asks it over RPC:

sudo -u bitcoin bitcoin-cli -conf=/etc/bitcoin/bitcoin.conf -datadir=/var/lib/bitcoind getblockchaininfo

The command runs as bitcoin for the same reason as before: with no rpcuser set, RPC authenticates with the .cookie file in the data directory, and only bitcoin can read it. -conf and -datadir point bitcoin-cli at the same configuration and data directory the service uses, so it finds the regtest RPC port and the cookie.

initialblockdownload looks odd next to verificationprogress: 1, but it follows from time. The node decides it’s caught up only when its tip is recent, and its tip is a block from 2011. Mining the first new block will clear it.

The node is up, reachable over RPC by its own service account, and waiting for its first block.

A shortcut for bitcoin-cli

Typing sudo -u bitcoin, the configuration path, and the data directory before every RPC call gets old fast. A shell function wraps them, the same way lnc wrapped lncli in my Kubernetes posts:

btc() {
  sudo -u bitcoin bitcoin-cli \
    -conf=/etc/bitcoin/bitcoin.conf \
    -datadir=/var/lib/bitcoind \
    "$@"
}

"$@" passes along every argument given to btc, each one kept as a separate word with its quoting intact, so btc getblockchaininfo runs the full command above. A function defined at the prompt lasts only for that shell session; adding it to ~/.bashrc makes it available in every new one.

A miner wallet and the first blocks

On regtest there are no miners but me. I created a wallet called miner, took a new address from it, and mined blocks to that address.

btc -named createwallet wallet_name=miner load_on_startup=true
MINER_ADDRESS=$(btc -rpcwallet=miner getnewaddress)
printf '%s\n' "$MINER_ADDRESS"
btc generatetoaddress 101 "$MINER_ADDRESS"

The address starts with bcrt1, the prefix for native SegWit addresses on regtest. Mainnet uses bc1 and testnet tb1, so a regtest address can’t be mistaken for a real one.

Why 101 blocks: a block reward can’t be spent until 100 more blocks are built on top of it. Mining 101 makes the reward from the first block spendable, which is what the miner will use later to fund the Lightning nodes. The wallet’s files live in the data directory, under /var/lib/bitcoind/regtest/wallets/miner, which is one reason that directory is 0700.

Asking the node again, through btc this time, shows what the 101 blocks changed:

btc getblockchaininfo
btc -rpcwallet=miner getbalances

The wallet’s balances show the maturity rule from above at work:

These are regtest coins with no value, but they behave like real ones: 50 BTC is what the miner can actually spend to fund the Lightning nodes.

Surviving a restart

Before LND comes anywhere near this node, the chain has to survive bitcoind going down and coming back. I saved the tip’s hash, restarted the service, and asked again:

BEFORE_RESTART_HASH=$(btc getbestblockhash)
sudo systemctl restart bitcoind
btc -rpcwait -rpcwaittimeout=60 getblockchaininfo

systemctl restart stops bitcoind, which flushes its databases to disk within the TimeoutStopSec window, then starts it again. Right after the restart the RPC server isn’t up yet, so -rpcwait makes bitcoin-cli keep retrying instead of failing at once, and -rpcwaittimeout=60 gives up after a minute.

After the restart, the node reports the same height, 101, and the same tip, 279b42…b577, as before it. initialblockdownload stays false and chainwork and size_on_disk are unchanged. The blocks live in /var/lib/bitcoind, not in the process, so a restart costs nothing. This is the property a PersistentVolume gave my pods, here provided by nothing more than a directory the service owns.

Comparing the saved hash against the tip after the restart, and checking the wallet:

AFTER_RESTART_HASH=$(btc getbestblockhash)
printf 'before: %s\nafter:  %s\n' \
  "$BEFORE_RESTART_HASH" "$AFTER_RESTART_HASH"
btc listwallets
btc -rpcwallet=miner getbalances

The two hashes are identical, character for character. listwallets shows miner already loaded without my touching it, which is load_on_startup=true doing its job, and the balances match what they were before the restart: 50 BTC spendable, 5,000 BTC immature, synced to height 101.

Starting at boot

Everything so far ran on systemctl start, which starts the service once. With the unit, the RPC, and the restart all checked, I enabled it so it comes up on its own whenever the VM boots:

sudo systemctl enable bitcoind
systemctl is-enabled bitcoind

enable doesn’t start anything. It creates a symlink in multi-user.target.wants/, the directory systemd reads to decide what to start when the system reaches its normal multi-user state at boot. That’s the WantedBy=multi-user.target line from the unit’s [Install] section taking effect. is-enabled now answers enabled, where systemctl status showed disabled earlier.

Next: Installing LND and its service account.