chatochi_notes/lightning_node_management.md

Lightning Node Management

My umbrel node is almost in sync with the blockchain so soon I will be able to put my new lightning node into use. These past days I have been consuming a lot of material on how to prepare the lightning node, with the idea in mind of making a successful routing node.

These are the topics that I have in mind:

• UPS for uptime
• Channel opening strategy
• Channel rebalancing strategy
• Pricing strategy

UPS for uptime

I have learnt through different sources that it is important to have a "good reputation" in order to have more chances of peers connecting to my node. Now, reputation is rather fuzzy, and varies from simply people observing your node on 1ML or similar pages, to a variety of rankings and scores such as BOS, Lightning Terminal Tiers, 1ML node rank, etc.

One of the components of most of these scores, and more fundamentally, a strong signal of node quality, is uptime. Plain and simple, never go down, and if you do, don't spend much time down.

In order to ensure that my node does a good job in terms of availability, I have been studying the possiblity of using a UPS. In general terms, an outage is the only relevant (as in both probable and impactful) event that could put my node down.

I have been reading up a bit on UPS. By design, they are designed to only provide a few minutes of availability to securely put services down in the machine that is going to go dry. But since both Banky and my router are super low consumption devices, I'm actually hoping that a decent UPS could provide a handful of hours of activity if needed. Notice I'm also including the router in the equation because, obviously, if the internet connection goes down, keeping Banky on is not enough.

So I need to:

• Decide what UPS I should get. The main points here are:
  • Price
  • Battery capacity
  • That it has two sockets.
• Confirm that, in the event of a house-level power outage, the router still keeps connecting to WAN as long as it has electricity. It would also be good to check with a building/street level outage, but I think simulating that goes beyond my current capacities.
• Measure empirically how long does the setup last until the node disconnects.

---

After a couple of conversations with people that roughly knew what they were talking about, I have concluded that indeed, a decent UPS should be able to keep banky + the home router up and running for more than an hour. Calculting the specifics is pretty irrelevant, since there are a lot of real world factors that will change results and it's not all that important.

I'll simply buy a nice one from pccomponentes and that's it.

Channel Opening Strategy

Once I get funds in my lightning node, I need to start opening channels. As I do it, they will all be unbalanced, with all the funds sitting in the local side. I'll have to compensate for that since I don't expect anyone opening towards me initially (I'm not gonna showcase myself all over the place to get people's attention). But I'll discuss that in another section.

So, I need to open channels to people. The key questions are:

• With who?
• How many sats to put into each channel?

Regarding channel size: generally, the larger, the better. It helps reduce the impact of onchain fees, offers more opportunities for keeping things balanced and also enables larger transactions to go through. The downside of making one channel large is the opportunity cost of having a few smaller channels, which increase the variety of nodes.

From what I've been reading, anything smaller than 1M sats gives yields more trouble than benefit. So I will keep that as a minimum size for the channels I open. Question is, how big should the bigger channels be?

I'm copying a table from the statistics page in 1ML, on 13/01/22:

Percentile	Capacity in Satoshis
1	__.__2.000
5	__._24.900
25	__.200.000
50	_1.000.000
75	_3.000.000
95	14.995.600
99	50.000.000

So, it's clear that going above 0.1 BTC is a pretty rare exercise. I'm planning on allocating 0.2 BTC in total during my first experiments, so I think I will distribute them in the following way:

How many channels	Capacity per channel
1	5.000.000
3	2.500.000
7	1.000.000
11	19.500.000

The missing half-million will be used as petty cash for fees.

The next question is: who to connect to?

Intuitively, the goal of any routing node intending to place itself in the network should be to find a place in the graph that's poorly connected but has a big demand for moving funds. Since reading the future is impossible, and there is no history of transactions available for anyone to make forecasts on, it's impossible to assess the demand.

Thus, the only thing one can try is to put itself between different nodes that are relevant in terms of traffic, but are not properly connected. This means that simply connecting to the largest nodes won't work, because they are already very well connected between themselves. I think that, instead, I should try to connect to both some of the large nodes, but also to a few smaller ones. Given the channel capacity strategy that I present above, it naturally follows that to have a few, large channels for large nodes and many, smaller channels for smaller nodes.

Ok, this is as far as my intuition goes. Now I'm going to do a bit of research to try to find out if there are any other factors I'm missing in my thinking.

---

I came across this brilliant post by Jameson Lopp . Awesome stuff, as it tends to be when he writes it. In it, it gives a thorough explanation of his own experience trying to run a routing lightning node. There is plenty of knowledge to get from it, but the main points I'm keeping in mind for now are:

• Avoid channels smaller than 1M sats completely. If possible, do larger than 10M sats.
• The rebalance tool by carsten otto works nicely, but only if you don't also have an automatic fee setter (or if you do, just make sure that rebalancing and fee setting happen not so frequently and with time between both).
• The swaps on lightningnetwork.plus are a good idea and not some random internet scam, so I should look into how to use that.

A few links to interesting stuff that I got from Lopp's post:

• An article by Alex Bosworth , with the usual thoughts. One new idea I had not seen anywhere though: using my own node to make payments on the lightning network is a great way to rebalance channels that need inbound liquidity.
• Node match tool to find nodes that increase connectivity of another node (mine, basically).
• Lightning Terminal page to get scores on any node. Good for judging candidates for channel opening.
• Lightning conductor, which offers liquidity services including a pretty convenient services to open an incoming channel with an on-chain transaction.

First night...

I'm writing this as I wait for some channels to open successfully. Finally, today was the day I started my routing node!

As planned, I moved 0.2 BTC from cold storage to the umbrel wallet. I looked for a pretty looking node and opened a channel with them and, voila, it worked! I opened another, smaller one with another smaller node, but Thunderhub didn't let me. I learned that Thunderhub only allows you to have one pending channel opening at a time. So I had to wait a bit to create a second one.

Once I successfully opened those two channels, I headed to lightningnetwork.plus and joined a triangle. It was almost by accident. I was checking the triangle swap page when I hit the application button and, boom, it was closed. The page was giving me some instructions on how to proceed and I kind of just followed through without much thinking. It felt like heaven when I saw my incoming partner channel opening towards me. This swaps thingy is great! The only issue I see is that my partners quality is pretty low. One of them is pretty old, but has almost no capacity. The other one is super new, but is growing pretty fast. Oh well, I guess it's better than nothing!

I have realised that the fancier swaps are restricted by channel number and total capacity, so it seems that I should try to step up my game slowly, always trying to get the best possible partners and then letting my fresh open channels mature a bit so that I earn some street creed that allows me to get into more exclusive swaps.

I can already tell it's gonna be quite a few nights of little sleep...

Plebnet tribe

Plebnet this, plebnet that... I kept reading about the plebs everywhere. Apparently it's just this bunch of people gathering in telegram to open channels and help each other out. Old school forum style, like in the P2P piracy times.

Today I stumbled upon the plebnet wiki and I found quite a few useful things. When reading this and similar resources, I'm realising that there's a lot of people doing exactly the same discovery path I am going through with the lightning network. It's very exciting to see this happening. Will it blow up exponentially? I keep on dreaming awake with that idea.

Anyways, for the cool stuff:

• I read their hardening page. It seems pretty standard linux security, so I don't feel I need to take any extra steps in this topic.
• A small guide on how to upgrade umbrel. It seems umbrel is a rather popular choice.

The self-routing trick...

I was reading a post on the lightningnetwork subreddit when I spotted a brilliant idea. Like all routing nodes wannabe, I will be having troubles pretty often with getting remote balance. The only chances to get some are swaps (which only provides at the beginning), services like LOOP (which come with a hefty fee) and natural flow of payments (which will only happen with a lot of time, effort and capital).

And there comes this guy and says: "you can also get remote balance by spending your money". Brilliant! It's completely true. If I ever need to pay for something through the Lightning Network, I can just do it through the node and afterwards refill it through the main chain with new btc. Since I can do my grocery shopping with bitrefill, that means I have like 300.000sats of weekly rebalancing power.

So things would go this way:

1. Buy a 100€ grocery voucher through bitrefill. Pay the invoice through an unbalanced channel of my node. Now I owe Counterweight LTD 100€.
2. Go grocery shopping with that, cause I need to eat.
3. When I owe Counterweight LTD a reasonable amount, like 500€, get some BTC with my own money and send it to Counterweight LTD wallet.
4. Done! No debts outstanding and the channels have been rebalanced.

It's just great. Basically, I'm playing LOOP's role with myself, so I get the same benefit without having to pay their huge fees. Plus, I do my groceries with btc. How badass is that?

There's a bit of fee paying involved, but it's orders of magnitude inferior to what I would incurr with the other methods to get liquidity, so not much of a problem.

I just need to be tidy with keeping track of the balances and that should be it.

The funny guy

Quite a random, but fun story. I was checking my node's lightning transactions when I saw a 10sat one. I was confused at first, and my first thought is somebody, somehow, for some reason, had charged me a fee (in the lightning world, it feels like every breath comes with a fee).

But reading more carefully, I realised it was an incoming transactions. Who the hell sent me 10 sats just because? And then I checked the message field and it said "Check out my node, Baystar!", together with a public key. Lovely, this guy is just sending random spam messages to people to get some attention. The lightning network keeps on feeling like the wild wild west, where funny things just happen out of the blue.

A professional knows its price

As I am beginning and trying things out, I haven't paid much attention to the fees in my channels. I have set all the base fees to 0, following the movement to support Pickhardt payments, and I have set my fee rates 20ppm, which I have identified as being relatively low, so that I can check how much flows through my node.

But at some point, I will want to get more serious about fee management. Specially, taking into account the impact on balancing. Overall, balanced channels are desirable since they give more paths to the network, and thus make the node more capable of routing in different directions. At the entire node level, it's also important to have a general balance between local and remote, because that limits the total capacity of the node (i.e. if you have 10M sats of outbound capacity, but no inbound capacity at all, you can't route anything. The other way around holds true as well.).

To achieve this, I have been looking into charge-lnd. The tool is a small python script that can read data from lnd and modify fees accordingly. There are some instructions on how to set it up with umbrel, which is awesome. The tool is configurable through a very simple config file that allows several policies to exist, so a pretty intelligent behaviour can be achieved with relatively simple effort.

The link to the tool is: github I also found this example gist showcasing some example policies, which I think will be very useful: the gist

Eventually, I need to set this up. But I will still play around manually for a while to get a feel on the behaviour of flows with different fees.

Setting up uptime kuma

As part of my efforts to be professional and keep the node with the best possible uptime, I should always know if the node is up. Since checking constantly myself is really something I don't want to do, I decided to use uptime kuma to do the job for me.

To get this done, I installed uptime kuma with docker in navaja. Since it won't hold any critical data, I'm not really concerned about backing it up or anything like that.

Once that was ready, I played a bit with the different pinging options. Things were difficult, because with banky behind an onion address, regular HTTP and TCP options didn't work. I am not sure if that was because it's plain impossible to set that up, or simply I was doing it wrong because I'm an ignorant. In any case, I discovered that uptime kuma has a cool option in which you reverse roles: uptime kuma is constantly waiting to receive a heartbeat from the monitored service at a certain schedule. If it doesn't, it considers the service to be done. I went for that option, and I simply had to set a curl call in banky with cron to run every minute. Easy peasy.

Next was to prepare some alerts in case banky goes down, because otherwise I would be stuck having to check uptime kuma constantly, so I would still be at the starting point.

The easiest option I could find was to use a telegram bot. To get this done, I set a telegram bot by sending a message to the Bot Father to create a new bot, which I called uptima_kuma_bot. Lovely typo there, but doesn't matter. The auth token for the bot is 5184325884:AAHGsULQXWNVy-BHaP8d-itKOK-FwEHQSuI. Then, I set up a group between me and the bot, and got the chat id by accessing this endpoint . With that, I managed to configure uptime kuma to send a message everytime banky goes down or recovers. The whole thing is suboptimal because, technically, umbrel services could go down while banky stays up, and the uptime kuma won't notify me. But as the meme used to say, it's something.

My sunday checklist

Keeping an eye on the node is important, but I haven't put much structure into it. I enjoy messing around with it, but I don't want to develop and addictive habit of checking on it all day long.

So, I have decided to:

• Set up an automatic monitor with uptime kuma that will trigger warnings if the service stops working.
• Keep all the node-chores for sunday afternoon and just breeze through them.

To support the second point, here is my checklist of chores:

• Check transactions and forwards. Put any that happened into the accounting book.
• Buy any grocery vouchers that are needed.
• Send any BTC from other wallets into here that is needed.
• Check the uptime statistics of all my channel peers in thunderhub. Judge if any of them should be removed because they are behaving like shit.
• Open channels/swaps in lnplus if necessary.
• Check up on 1ml to see how well is the node ranking.
• Check up on lightning terminal to see how is the node being judged.
• Check up on lightning pool to see if it's finally T1.
• Check that uptime kuma is running properly.
• Check if there are any relevant umbrel updates.
• Check the content of the lightningnetwork and umbrel subreddits.
• Put anything relevant in the accounting book again.
• Backup the channel states.

Automating fee adjustment

I finally found a way to use charge-lnd with Umbrel. I followed the instructions here . My own custom stuff changes were:

• I put my config file in: /media/data_disk/charge.conf
• I wrote my docker run statement as follows:

docker run --rm -it --network=umbrel_main_network 
-e GRPC_LOCATION=10.21.21.9:10009 
-e LND_DIR=/data/.lnd 
-e CONFIG_LOCATION=/app/charge.config 
-v /media/data_disk/umbrel/lnd:/data/.lnd 
-v /media/data_disk/charge.config:/app/charge.config accumulator/charge-lnd:latest

I've been reading a bit and apparently it's not a good idea to run price adjustments very frequently given that it makes it difficult for other nodes to plan their routes, so automated software will see it as a low quality attribute for routing nodes. I have decided to leave a weekly cronjob for now, so the fees will update automatically every thursday at 10pm.

A small job

Today I was randomly wandering through internet when I visited microlancer.io, a small website I had already seen once or twice. The page hosts tasks, posted by users, which come attached with Bitcoin bounties as compensation.

Most tasks are crap, spammish-like stuff. But today I found one that simple said "Open a lightning channel with my node". The task poster indicated in the description that he would pay 2500 sats to whoever opened a channel of at least 750K sats of girth and kept it up for a month. His node's pubkey was also posted. I checked it up and saw it was a very small node, with barely any capacity. Way to small to be an interesting partner. But, 2500 sats was more than enough to cover for the costs of opening and closing a channel with him, so I decided I would give the task a shot to experiment.

I signed-up at microlancer.io and offered myself for the task. The task poster accepted (which came with him putting the 2500 sats in an escrow handled through microlancer), and I went ahead and created a 1 million sats channel with him. Afterwards, the poster marked the task as done and the escrow funds were sent to me. I had received 2500 sats for opening the channel!

That was a very nice experience. I think the channel will probably be useless, but I will keep my promise and leave it up for at least a month. And the money I got from this, even after accounting for channel opening and closing fees, is several times more than what I got routing payments! I feel it has been like a glimpse of the feeling I would get by providing liquidity in Lightning Pool. I'm looking forward to get the node up to Tier1 so that the party can get started.

https://microlancer.io/task/view/2038 The small node's pubkey: 02c94a8c40c64c511228c0ca4b393ccd0738d75e9229073ceb6328246de32b9a8b

Getting scientifc with fees

After going above 0.5 BTC in capacity, 20+ channel count and starting to use automated fee setting with charge-lnd, I have started to see an increase in the frequency and diversity of routed payments. It seems that the current size is getting to the point where routing activity flows in several directions so that channels stay balanced and things "flow nicely".

Given that I don't expected to have any channel opening/closing activity for the next couple of weeks, I have decided I will run an experiment on prices for the next two weeks. The plan is to hold two different pricing models, each during one week, to see the impact on frequency and size of routing, as well as overall fees earned.

The first week will be the "expensive" week. During that week, I will set:

• The encourage-routing fee to 25.
• The discourage-routing fee to 500.
• The proportional fees between 100 and 300.

The second week will be the "cheap" week. During that week, I will set:

• The encourage-routing fee to 10.
• The discourage-routing fee to 500.
• The proportional fees between 15 and 50.

Afterwards I will compare:

• The count of forwards
• The total forwarded amount
• The total sats earned
• The mean transaction size

First week results

The expensive week has had very clear and strong effects. Traffic pretty much stopped. Only two forwards for the entire period.

• The count of forwards: 3
• The total forwarded amount: 80014
• The total sats earned: 0.710 sats (710 millisats)
• The mean transaction size: 26671
• Effective mean fee: 9ppm

A couple of links to node-recommending tools that I want to try: https://github.com/Gridflare/lndpytools https://github.com/bitromortac/lndmanage#setup

A guide on how to put the watchtower in place: https://bitcoinmagazine.com/guides/how-to-set-up-watchtower-lightning-node