chatochi_notes/lightning_node_management.md

# Lightning Node Management

My umbrel node is almost in sync with the blockchain so soon I will be able to
put my new lightning node into use. These past days I have been consuming a lot
of material on how to prepare the lightning node, with the idea in mind of
making a successful routing node.

These are the topics that I have in mind:

- UPS for uptime
- Channel opening strategy
- Channel rebalancing strategy
- Pricing strategy

## UPS for uptime

I have learnt through different sources that it is important to have a "good
reputation" in order to have more chances of peers connecting to my node. Now,
reputation is rather fuzzy, and varies from simply people observing your node
on 1ML or similar pages, to a variety of rankings and scores such as BOS,
Lightning Terminal Tiers, 1ML node rank, etc.

One of the components of most of these scores, and more fundamentally, a strong
signal of node quality, is uptime. Plain and simple, never go down, and if you
do, don't spend much time down.

In order to ensure that my node does a good job in terms of availability, I
have been studying the possiblity of using a UPS. In general terms, an outage
is the only relevant (as in both probable and impactful) event that could put
my node down.

I have been reading up a bit on UPS. By design, they are designed to only
provide a few minutes of availability to securely put services down in the
machine that is going to go dry. But since both Banky and my router are super
low consumption devices, I'm actually hoping that a decent UPS could provide a
handful of hours of activity if needed. Notice I'm also including the router in
the equation because, obviously, if the internet connection goes down, keeping
Banky on is not enough.

So I need to:

- Decide what UPS I should get. The main points here are:
    - Price
    - Battery capacity
    - That it has two sockets.
- Confirm that, in the event of a house-level power outage, the router still
  keeps connecting to WAN as long as it has electricity. It would also be good
  to check with a building/street level outage, but I think simulating that
  goes beyond my current capacities.
- Measure empirically how long does the setup last until the node disconnects.

---

After a couple of conversations with people that roughly knew what they were
talking about, I have concluded that indeed, a decent UPS should be able to
keep banky + the home router up and running for more than an hour. Calculting
the specifics is pretty irrelevant, since there are a lot of real world factors
that will change results and it's not all that important.

I'll simply buy a nice one from pccomponentes and that's it.

## Channel Opening Strategy

Once I get funds in my lightning node, I need to start opening channels. As I
do it, they will all be unbalanced, with all the funds sitting in the local
side. I'll have to compensate for that since I don't expect anyone opening
towards me initially (I'm not gonna showcase myself all over the place to get
people's attention). But I'll discuss that in another section.

So, I need to open channels to people. The key questions are:

- With who?
- How many sats to put into each channel?

Regarding channel size: generally, the larger, the better. It helps reduce the
impact of onchain fees, offers more opportunities for keeping things balanced
and also enables larger transactions to go through. The downside of making one
channel large is the opportunity cost of having a few smaller channels, which
increase the variety of nodes.

From what I've been reading, anything smaller than 1M sats gives yields more
trouble than benefit. So I will keep that as a minimum size for the channels I
open. Question is, how big should the bigger channels be?

I'm copying a table from the statistics page in 1ML, on 13/01/22:

| Percentile | Capacity in Satoshis |
| --- |----------------------|
| 1 | __.__2.000           |
| 5 | __._24.900           |
| 25 | __.200.000           |
| 50 | _1.000.000           |
| 75 | _3.000.000           |
| 95 | 14.995.600           |
| 99 | 50.000.000           |

So, it's clear that going above 0.1 BTC is a pretty rare exercise. I'm planning
on allocating 0.2 BTC in total during my first experiments, so I think I will
distribute them in the following way:

| How many channels | Capacity per channel |
|-------------------|----------------------|
| 1                 | 5.000.000            |
| 3                 | 2.500.000            |
| 7                 | 1.000.000            |
| 11                | 19.500.000           |

The missing half-million will be used as petty cash for fees.

The next question is: who to connect to?

Intuitively, the goal of any routing node intending to place itself in the
network should be to find a place in the graph that's poorly connected but has
a big demand for moving funds. Since reading the future is impossible, and
there is no history of transactions available for anyone to make forecasts on,
it's impossible to assess the demand.

Thus, the only thing one can try is to put itself between different nodes that
are relevant in terms of traffic, but are not properly connected. This means
that simply connecting to the largest nodes won't work, because they are
already very well connected between themselves. I think that, instead, I should
try to connect to both some of the large nodes, but also to a few smaller ones.
Given the channel capacity strategy that I present above, it naturally follows
that to have a few, large channels for large nodes and many, smaller channels
for smaller nodes.

Ok, this is as far as my intuition goes. Now I'm going to do a bit of research
to try to find out if there are any other factors I'm missing in my thinking.

----

I came
across [this brilliant post by Jameson Lopp](https://blog.lopp.net/lightning-network-liquidity-management-guide/)
. Awesome stuff, as it tends to be when he writes it. In it, it gives a
thorough explanation of his own experience trying to run a routing lightning
node. There is plenty of knowledge to get from it, but the main points I'm
keeping in mind for now are:

- Avoid channels smaller than 1M sats completely. If possible, do larger than
  10M sats.
- The rebalance tool by carsten otto works nicely, but only if you don't also
  have an automatic fee setter (or if you do, just make sure that rebalancing
  and fee setting happen not so frequently and with time between both).
- The swaps on [lightningnetwork.plus](https://lightningnetwork.plus) are a
  good idea and not some random internet scam, so I should look into how to use
  that.

A few links to interesting stuff that I got from Lopp's post:

- [An article by Alex Bosworth](https://github.com/alexbosworth/run-lnd/blob/master/LIQUIDITY.md)
  , with the usual thoughts. One new idea I had not seen anywhere though: using
  my own node to make payments on the lightning network is a great way to
  rebalance channels that need inbound liquidity.
- [Node match tool](https://moneni.com/nodematch) to find nodes that increase
  connectivity of another node (mine, basically).
- [Lightning Terminal page](https://terminal.lightning.engineering/#/) to get
  scores on any node. Good for judging candidates for channel opening.
- [Lightning conductor](https://lightningconductor.net/receipt), which offers
  liquidity services including a pretty convenient services to open an incoming
  channel with an on-chain transaction.

## First night...

I'm writing this as I wait for some channels to open successfully. Finally,
today was the day I started my routing node!

As planned, I moved 0.2 BTC from cold storage to the umbrel wallet. I looked
for a pretty looking node and opened a channel with them and, voila, it worked!
I opened another, smaller one with another smaller node, but Thunderhub didn't
let me. I learned that Thunderhub only allows you to have one pending channel
opening at a time. So I had to wait a bit to create a second one.

Once I successfully opened those two channels, I headed to
lightningnetwork.plus and joined a triangle. It was almost by accident. I was
checking the triangle swap page when I hit the application button and, boom, it
was closed. The page was giving me some instructions on how to proceed and I
kind of just followed through without much thinking. It felt like heaven when I
saw my incoming partner channel opening towards me. This swaps thingy is great!
The only issue I see is that my partners quality is pretty low. One of them is
pretty old, but has almost no capacity. The other one is super new, but is
growing pretty fast. Oh well, I guess it's better than nothing!

I have realised that the fancier swaps are restricted by channel number and
total capacity, so it seems that I should try to step up my game slowly, always
trying to get the best possible partners and then letting my fresh open
channels mature a bit so that I earn some street creed that allows me to get
into more exclusive swaps.

I can already tell it's gonna be quite a few nights of little sleep...

## Plebnet tribe

Plebnet this, plebnet that... I kept reading about the plebs everywhere.
Apparently it's just this bunch of people gathering in telegram to open
channels and help each other out. Old school forum style, like in the P2P
piracy times.

Today I stumbled upon the [plebnet wiki](https://plebnet.wiki/wiki/Main_Page)
and I found quite a few useful things. When reading this and similar resources,
I'm realising that there's a lot of people doing exactly the same discovery
path I am going through with the lightning network. It's very exciting to see
this happening. Will it blow up exponentially? I keep on dreaming awake with
that idea.

Anyways, for the cool stuff:

- I read their hardening page. It seems pretty standard linux security, so I
  don't feel I need to take any extra steps in this topic.
- [A small guide](https://plebnet.wiki/wiki/Upgrading_Umbrel_Node) on how to
  upgrade umbrel. It seems umbrel is a rather popular choice.

## The self-routing trick...

I was reading a post on the lightningnetwork subreddit when I spotted a
brilliant idea. Like all routing nodes wannabe, I will be having troubles
pretty often with getting remote balance. The only chances to get some are
swaps (which only provides at the beginning), services like LOOP (which come
with a hefty fee) and natural flow of payments (which will only happen with a
lot of time, effort and capital).

And there comes this guy and says: "you can also get remote balance by spending
your money". Brilliant! It's completely true. If I ever need to pay for
something through the Lightning Network, I can just do it through the node and
afterwards refill it through the main chain with new btc. Since I can do my
grocery shopping with bitrefill, that means I have like 300.000sats of weekly
rebalancing power.

So things would go this way:

1. Buy a 100€ grocery voucher through bitrefill. Pay the invoice through an
   unbalanced channel of my node. Now I owe Counterweight LTD 100€.
2. Go grocery shopping with that, cause I need to eat.
3. When I owe Counterweight LTD a reasonable amount, like 500€, get some BTC
   with my own money and send it to Counterweight LTD wallet.
4. Done! No debts outstanding and the channels have been rebalanced.

It's just great. Basically, I'm playing LOOP's role with myself, so I get the
same benefit without having to pay their huge fees. Plus, I do my groceries
with btc. How badass is that?

There's a bit of fee paying involved, but it's orders of magnitude inferior to
what I would incurr with the other methods to get liquidity, so not much of a
problem.

I just need to be tidy with keeping track of the balances and that should be
it.

## The funny guy

Quite a random, but fun story. I was checking my node's lightning transactions
when I saw a 10sat one. I was confused at first, and my first thought is
somebody, somehow, for some reason, had charged me a fee (in the lightning
world, it feels like every breath comes with a fee).

But reading more carefully, I realised it was an _incoming_ transactions. Who
the hell sent me 10 sats just because? And then I checked the message field and
it said  "Check out my node, Baystar!", together with a public key. Lovely,
this guy is just sending random spam messages to people to get some attention.
The lightning network keeps on feeling like the wild wild west, where funny
things just happen out of the blue.

## A professional knows its price

As I am beginning and trying things out, I haven't paid much attention to the
fees in my channels. I have set all the base fees to 0, following the movement
to support Pickhardt payments, and I have set my fee rates 20ppm, which I have
identified as being relatively low, so that I can check how much flows through
my node.

But at some point, I will want to get more serious about fee management.
Specially, taking into account the impact on balancing. Overall, balanced
channels are desirable since they give more paths to the network, and thus make
the node more capable of routing in different directions. At the entire node
level, it's also important to have a general balance between local and remote,
because that limits the total capacity of the node (i.e. if you have 10M sats
of outbound capacity, but no inbound capacity at all, you can't route anything.
The other way around holds true as well.).

To achieve this, I have been looking into `charge-lnd`. The tool is a small
python script that can read data from `lnd` and modify fees accordingly. There
are some instructions on how to set it up with umbrel, which is awesome. The
tool is configurable through a very simple config file that allows several
policies to exist, so a pretty intelligent behaviour can be achieved with
relatively simple effort.

The link to the tool is: [github](https://github.com/accumulator/charge-lnd)
I also found this example gist showcasing some example policies, which I think
will be very
useful:  [the gist](https://gist.github.com/ziggie1984/48a67f3ee3cd0616e40620dc372ac3fe)

Eventually, I need to set this up. But I will still play around manually for a
while to get a feel on the behaviour of flows with different fees.

## Setting up uptime kuma

As part of my efforts to be professional and keep the node with the best
possible uptime, I should always know if the node is up. Since checking
constantly myself is really something I don't want to do, I decided to use
uptime kuma to do the job for me.

To get this done, I installed uptime kuma with docker in navaja. Since it won't
hold any critical data, I'm not really concerned about backing it up or
anything like that.

Once that was ready, I played a bit with the different pinging options. Things
were difficult, because with banky behind an onion address, regular HTTP and
TCP options didn't work. I am not sure if that was because it's plain
impossible to set that up, or simply I was doing it wrong because I'm an
ignorant. In any case, I discovered that uptime kuma has a cool option in which
you reverse roles: uptime kuma is constantly waiting to _receive_ a heartbeat
from the monitored service at a certain schedule. If it doesn't, it considers
the service to be done. I went for that option, and I simply had to set a curl
call in banky with cron to run every minute. Easy peasy.

Next was to prepare some alerts in case banky goes down, because otherwise I
would be stuck having to check uptime kuma constantly, so I would still be at
the starting point.

The easiest option I could find was to use a telegram bot. To get this done, I
set a telegram bot by sending a message
to [the Bot Father](https://t.me/BotFather) to create a new bot, which I called
uptima_kuma_bot. Lovely typo there, but doesn't matter. The auth token for the
bot is 5184325884:AAHGsULQXWNVy-BHaP8d-itKOK-FwEHQSuI. Then, I set up a group
between me and the bot, and got the chat id by
accessing [this endpoint](https://api.telegram.org/bot5184325884:AAHGsULQXWNVy-BHaP8d-itKOK-FwEHQSuI/getUpdates)
. With that, I managed to configure uptime kuma to send a message everytime
banky goes down or recovers. The whole thing is suboptimal because,
technically, umbrel services could go down while banky stays up, and the uptime
kuma won't notify me. But as the meme used to say, it's something.

## My sunday checklist

Keeping an eye on the node is important, but I haven't put much structure into
it. I enjoy messing around with it, but I don't want to develop and addictive
habit of checking on it all day long.

So, I have decided to:

- Set up an automatic monitor with uptime kuma that will trigger warnings if
  the service stops working.
- Keep all the node-chores for sunday afternoon and just breeze through them.

To support the second point, here is my checklist of chores:

- Check transactions and forwards. Put any that happened into the accounting
  book.
- Buy any grocery vouchers that are needed.
- Send any BTC from other wallets into here that is needed.
- Check the uptime statistics of all my channel peers in thunderhub. Judge if
  any of them should be removed because they are behaving like shit.
- Open channels/swaps in lnplus if necessary.
- Check up on 1ml to see how well is the node ranking.
- Check up on lightning terminal to see how is the node being judged.
- Check up on lightning pool to see if it's finally T1.
- Check that uptime kuma is running properly.
- Check if there are any relevant umbrel updates.
- Check the content of the lightningnetwork and umbrel subreddits.
- Put anything relevant in the accounting book again.
- Backup the channel states.

## Automating fee adjustment

I finally found a way to use `charge-lnd` with Umbrel. I followed the
instructions
[here](https://community.getumbrel.com/t/guide-installing-charge-lnd-in-a-docker-to-automate-your-fee-policies/2187)
. My own custom stuff changes were:

- I put my config file in: `/media/data_disk/charge.conf`
- I wrote my docker run statement as follows:

```commandline
docker run --rm -it --network=umbrel_main_network 
-e GRPC_LOCATION=10.21.21.9:10009 
-e LND_DIR=/data/.lnd 
-e CONFIG_LOCATION=/app/charge.config 
-v /media/data_disk/umbrel/lnd:/data/.lnd 
-v /media/data_disk/charge.config:/app/charge.config accumulator/charge-lnd:latest
```

I've been reading a bit and apparently it's not a good idea to run price
adjustments very frequently given that it makes it difficult for other nodes to
plan their routes, so automated software will see it as a low quality attribute
for routing nodes. I have decided to leave a weekly cronjob for now, so the
fees will update automatically every thursday at 10pm.

## A small job

Today I was randomly wandering through internet when I visited microlancer.io,
a small website I had already seen once or twice. The page hosts tasks, posted
by users, which come attached with Bitcoin bounties as compensation.

Most tasks are crap, spammish-like stuff. But today I found one that simple
said "Open a lightning channel with my node". The task poster indicated in the
description that he would pay 2500 sats to whoever opened a channel of at least
750K sats of girth and kept it up for a month. His node's pubkey was also
posted. I checked it up and saw it was a very small node, with barely any
capacity. Way to small to be an interesting partner. But, 2500 sats was more
than enough to cover for the costs of opening and closing a channel with him,
so I decided I would give the task a shot to experiment.

I signed-up at microlancer.io and offered myself for the task. The task poster
accepted (which came with him putting the 2500 sats in an escrow handled
through microlancer), and I went ahead and created a 1 million sats channel
with him. Afterwards, the poster marked the task as done and the escrow funds
were sent to me. I had received 2500 sats for opening the channel!

That was a very nice experience. I think the channel will probably be useless,
but I will keep my promise and leave it up for at least a month. And the money
I got from this, even after accounting for channel opening and closing fees, is
several times more than what I got routing payments! I feel it has been like a
glimpse of the feeling I would get by providing liquidity in Lightning Pool.
I'm looking forward to get the node up to Tier1 so that the party can get
started.

https://microlancer.io/task/view/2038
The small node's pubkey:
02c94a8c40c64c511228c0ca4b393ccd0738d75e9229073ceb6328246de32b9a8b

## Getting scientifc with fees

After going above 0.5 BTC in capacity, 20+ channel count and starting to use
automated fee setting with charge-lnd, I have started to see an increase in the
frequency and diversity of routed payments. It seems that the current size is
getting to the point where routing activity flows in several directions so that
channels stay balanced and things "flow nicely".

Given that I don't expected to have any channel opening/closing activity for
the next couple of weeks, I have decided I will run an experiment on prices for
the next two weeks. The plan is to hold two different pricing models, each
during one week, to see the impact on frequency and size of routing, as well as
overall fees earned.

The first week will be the "expensive" week. During that week, I will set:

- The encourage-routing fee to 25.
- The discourage-routing fee to 500.
- The proportional fees between 100 and 300.

The second week will be the "cheap" week. During that week, I will set:

- The encourage-routing fee to 10.
- The discourage-routing fee to 500.
- The proportional fees between 15 and 50.

Afterwards I will compare:

- The count of forwards
- The total forwarded amount
- The total sats earned
- The mean transaction size


### First week results

The expensive week has had very clear and strong effects. Traffic pretty much
stopped. Only two forwards for the entire period.

- The count of forwards: 3
- The total forwarded amount: 80014
- The total sats earned: 0.710 sats (710 millisats)
- The mean transaction size: 26671
- Effective mean fee: 9ppm 


## Automatic back ups and getting ready for the worse

There are two options. This one is the only one that keeps channels alive:

    This method was confirmed only by one Umbrel user that did it with success. But remember, he wasn’t using the Raspberry Pi version of Umbrel, was on a machine with regular Linux and Umbrel installed on it.
    
    This procedure is ONLY for experimented Linux users! If you don’t know to use Linux you better stay away.
    
    The procedure is simple. Are only these 2 files and they are located in:
    
    <lnd folder>/data/chain/bitcoin/mainnet/wallet.db
    
    <lnd folder>/data/graph/mainnet/channel.db
    
    You have to construct the data/… folders yourself and then copying the files to them, before starting up lnd.
    
        So, take another machine/drive and install Umbrel, fresh. Use your 24 words seed to restore the onchain AEZEED wallet. Leave it to start the sync a bit and construct the rest of the folders for LND. Then stop it.
        Go to the old drive and locate those 2 files.
        Copy them into the same path in the new node.
        Optional, if you have the blockchain data OK, files integrity is fine, you can copy also the blockchain so you can save time. If you think is corrupted, you better just forget it and let the sync to be done in normal/natural way.
        Start the node, leave it to sync and… voila, your old LN channels are there alive and not closed.
        If there is a problem in restarting the node, just run: sudo scripts/configure
    
    REMINDER: these files have to be the latest version that was online! If you use an older backup you can lose funds being punished for cheating with an older version of your channels.

To keep the two files always backup, I'm going to automatically backthem up in
my own git server in navaja.

The second option is the "standard" recovery proposed by umbrel, which recovers
all funds but closes channels in the process of doing so.

    It is suppose that you already have the 24 words seed and the channel.backup file obtained previously. If you didn’t make that backup, but you still have access to your old node drive, you can find it in /home/umbrel/umbrel/lnd/data/chain/bitcoin/mainnet/channel.backup or you can request a copy from Umbrel devs.
    
    Install a new instance of Umbrel. Start the dashboard page and you will be prompted to use previously backup 24 words seed.
    
    Once you’ve restored from the 24 words, it might take a few minutes to a few hours for it to scan all of your previous Bitcoin (on-chain) transactions and balances. Meanwhile, here’s how you can restore the funds in your Lightning channels.
    
    STEP 1: COPY OVER THE CHANNEL BACKUP FILE FROM YOUR COMPUTER TO YOUR UMBREL.
    
    Enter using SSH 139 and run this:
    
    scp <path/to/your/channel/backup/file> umbrel@umbrel.local:/home/umbrel/umbrel/lnd/channel.backup
    Here more details about using scp command 1
    
        Replace <path/to/your/channel/backup/file> with the exact path to channel backup file on your computer
    
    The password is moneyprintergobrrr, except on version 0.3.3 or later where the password is your personal user password instead.
    
    STEP 2: RECOVER FUNDS
    
    cd ~/umbrel && ./bin/lncli restorechanbackup --multi_file /data/.lnd/channel.backup
    
    After you run this, wait for 1 minute. You should now be able to see your channels being closed on http://umbrel.local/lightning 35.
    You should wait patiently until the funds are coming back to your onchain wallet. It will take at least 40 blocks. You can see/check the details of closing channels in the troubleshooting guide 35.

So for this one, I also need to copy one file to keep it backup.



## Checkpoint 27/03/2022

- February events
  - Added 0.15 liquidity
  - Automation of fee setting
  - Automation of backups
  - First rewarded liquidity provision through microlancer
  - First categorization of good node in lighting terminal
- P&L
- Going ahead
  - Reach Tier1 in lightning pool to start renting liquidity
  - Make march purchases and add to banky
  - From that point on, I will also put skin in the game
  - Leverage with Bankinter

A couple of links to node-recommending tools that I want to try:
https://github.com/Gridflare/lndpytools
https://github.com/bitromortac/lndmanage#setup

A guide on how to put the watchtower in place:
https://bitcoinmagazine.com/guides/how-to-set-up-watchtower-lightning-node