[bug]: Cleaning circuits from disk for closed channels takes 20+ minutes

[benthecarman]

Background

On the mutinynet signet faucet node, it has >8k inactive peers/channels. On startup it takes 20+ minutes to get past Cleaning circuits from disk for closed channels.

Is there a way I can clean up these peers or skip this step on start up or do I just need to force close a ton of channels

Your environment

• version of lnd 0.19.3
• which operating system (uname -a on *Nix) ubuntu
• version of btcd, bitcoind, or other backend bitcoind
• any other relevant environment details in docker, boltdb backend

Steps to reproduce

start up the node

Expected behaviour

happens fast

Actual behaviour

takes forever

[ziggie1984]

could you provide a goroutine profile (cpu+memory), because normally I would expect that should not last very long because the circuit map is normally cleared after each forward so this map shouldn't be that long. Can you also provide the logs most interested in how many circuits were cleaned in this operation which lasted +20 min see also this log line:

lnd/htlcswitch/circuit_map.go

Line 482 in 6c9e0f3

log.Infof("Finished cleaning: num_closed_channel=%v, "+

[ziggie1984]

@benthecarman could you provide details otherwise I close the issue because it is hard for me reproduce it without the logs.

[benthecarman]

Hey sorry haven't wanted to give myself 30 mins of downtime just to get some logs but was planning some maintenance tonight, should be able to grab them

[ziggie1984]

Thank you for your reply. No hurry at all - I just wanted to make sure you still have an eye on this issue, provide us the logs whenever you plan a downtime 🙏

[benthecarman]

Took 25 mins today

lnd               | 2025-09-18 21:40:14.800 [INF] BTWL: Started rescan from block 00000030baeedfa3b0d1ce06c9fd5d2dd4042cc0a4e5ac4b0e8cd5d1f6a48925 (height 2459768) for 387005 addrs, 11 outpoints
lnd               | 2025-09-18 21:40:14.976 [INF] HSWC: Finished cleaning: num_closed_channel=10661, num_circuits=0, num_keystone=0
lnd               | 2025-09-18 21:40:14.976 [INF] HSWC: Restoring in-memory circuit state from disk
lnd               | 2025-09-18 21:40:15.141 [INF] HSWC: Payment circuits loaded: num_pending=109, num_open=109
lnd               | 2025-09-18 21:40:16.500 [INF] NTFN: New block: height=2459821, sha=000000be99f7938e12123e36f72d5afaf39b9c62a67dd960621e97143f6ed342
lnd               | 2025-09-18 21:40:17.022 [INF] BTWL: Catching up block hashes to height 2459821, this might take a while
lnd               | 2025-09-18 21:40:17.023 [INF] BTWL: Done catching up block hashes
lnd               | 2025-09-18 21:40:17.023 [INF] BTWL: Finished rescan for 387005 addresses (synced to block 000000be99f7938e12123e36f72d5afaf39b9c62a67dd960621e97143f6ed342, height 2459821)
lnd               | 2025-09-18 21:40:17.025 [INF] BTWL: 9a30f952ab1e811338704ad8b95cb1b1d05703fdc5e025e1958346b200af643c: broadcast failed because of: bad txns inputs missingorspent
lnd               | 2025-09-18 21:40:17.026 [INF] BTWL: Removed invalid transaction: 9a30f952ab1e811338704ad8b95cb1b1d05703fdc5e025e1958346b200af643c
lnd               | 2025-09-18 21:40:17.028 [INF] BTWL: a4b1008ebfc354a6a58903fd0a939c82e548dd4c83d558839e28d035633e837b: tx already confirmed
lnd               | 2025-09-18 21:40:21.574 [INF] HSWC: Trimming open circuits for chan_id=1633383:4:0, start_htlc_id=0
lnd               | 2025-09-18 21:40:21.574 [INF] HSWC: Trimming open circuits for chan_id=914965:1:1, start_htlc_id=0
.. there are then a ton of these trimming logs

[ziggie1984]

Thank you for your logs, I think we can definitely improve here, it seems like our approach of fetching all the closed channels into memory is the bottleneck.

[ziggie1984]

are you running your node on a low power device, or is the setup a beefy hardware ?

[benthecarman]

its a $100 a month VPS, 6cpus

[ziggie1984]

Just to make sure I undestand you correctly it takes 25 min between these two log lines:

Cleaning circuits from disk for closed channels => +25 min => lnd | 2025-09-18 21:40:14.976 [INF] HSWC: Finished cleaning: num_closed_channel=10661, num_circuits=0, num_keystone=0

[benthecarman]

yes