jlspc [ARCHIVE] on Nostr: 📅 Original date posted:2023-04-18 📝 Original message: > One thing that confuses ...
📅 Original date posted:2023-04-18
📝 Original message:
> One thing that confuses me about the paper is how to think about routing
> to a "channel" rather than a node -- ie the payment from "E->FG->A" where
> "FG" isn't "F" or "G", but "both of them".
Yes, I found it very difficult to think about, and I kept getting confused between concepts like "user", "node", "channel", and "factory".
The thing that works best for me is to create a clear definition of each of these terms, along with the term "party".
I also think it's best to imagine a world in which there are hierarchical channels, but there are no "factories" per se.
The distinction is meaningful in the sense that a hierarchical channel requires that there are exactly *two* *fixed* "parties" to which the hierarchical channel pays out, and all changes to the balances paid to those parties are made via HTLCs.
In contrast, a "factory" can pay out to an arbitrary number of users or groups of users, and changes to the balances paid to those users or groups of users are made via updating the state of the factory (typically without an HTLC being required).
So, in a world with hierarchical channels, but without factories, we have the following terms:
* User: a person
* Party: a fixed group consisting of one or more users
* Node: a party
- so "party" and "node" are synonyms, with "party" emphasizing the human side of things and "node" emphasizing the graph side of things, such as when discussing the Lightning routing graph
* Physical channel: a layer 2 construct that is funded by an on-chain or an off-chain UTXO, is owned by 2 parties, pays a balance to each of the parties, and updates the parties' balances through HTLCs and a channel protocol
- a physical channel can be non-hierarchical (where each of the 2 parties is a single user) or hierarchical (where at least one party consists of more than 1 user)
- (Footnote: The concept of an "off-chain UTXO" is itself a bit tricky, as it often refers to one of multiple possible transaction outputs, where the transactions or their ancestors conflict, so only one of them can actually be instantiated. For example, a Lightning channel (using the current penalty protocol) can be said to create an off-chain UTXO that pays to one of the users, even though it's not clear which of the users' Commitment transactions will actually be instantiated. In general, an off-chain UTXO is a guaranteed way of getting a certain portion of a specific on-chain UTXO's funds. Let's skip going down this rabbit hole any further and just assume we understand these issues.)
* Logical Channel: a layer 2 construct that consists of all of the physical channels owned by a specific pair of parties
- the size (capacity) of a logical channel is the sum of the sizes of their physical channels
- (Footnote: It's possible, with a significant amount of software development work that I in no way discount, to route a payment through a logical channel where the payment traverses multiple physical channels at the same time. This is done by using separate HTLCs, all sharing the same secret, in each of the physical channels that the payment traverses. I can write more about this if that would be helpful.)
* Lightning Network (LN): a layer 2 network consisting of parties and logical channels between parties
- the LN can be represented as a directed graph, where each party is a node and each logical channel is pair of arcs (one in each direction) between two nodes
- the LN can be used to make payments from one node to another node via a path through the graph
- the purpose of the payment could be to send bitcoin from one user to another (in this case, the payment's source and destination nodes are both 1-user parties)
- the purpose of the payment could be to increase or decrease the size of a logical channel (in this case, the payment's source and/or destination node is a multi-user party)
- (Footnote: Here's another great opportunity for confusion, as the source and/or destination of the payment is a multi-user party, which is defined as more than two users, and yet the purpose of the payment is to resize a logical channel. Obviously, a group of users is different from a logical channel (as shown in the definitions above), so how does making a payment to or from a group of users change the size of a logical channel? The answer is that the logical channel being resized is the one owned by the group of users.)
* Factory: a term that doesn't exist in this world view
Sorry if all of this seems overly pedantic.
It's the only way I've been able to keep these concepts straight in my mind.
I've also found that drawings help clarify these issues, which is why the paper has 20 drawings, many of which are examples of payments and channel graphs.
> It feels like there's a whole
> mass of complications hidden in there from a routing perspective; like how
> do you link "FG" back with "F" and "G", how do you decide fees, how do
> you communicate fees/max_htlc/etc.
In addition to the assumptions and definitions above, we need to use Rusty Russell's idea for channel_update_v2 messages [1].
Regarding the specific issues you raised:
Q: How do you link "FG" back with "F" and "G"?
A: In terms of gossiping and the channel graph, you don't explicitly link "FG" with "F" and "G". "FG", "F" and "G" each have their own identity in the LN's Onion network and each one appears as a separate node in the routing graph, with no explicit linkage between them. Every logical channel in the channel graph is announced with channel_update_v2 messages that provide the channel's current capacity and refer back to an on-chain UTXO. It's possible, but not necessary, that some of the logical channels between "FG" and other nodes, between "F" and other nodes, and between "G" and other nodes, use the same on-chain UTXO for their channel_update_v2 messages.
Q: How do you decide fees?
A: All of the users that own a given logical channel negotiate the routing fees that they want to charge. These routing fees are of two types: 1) a fee for devoting capital (by being the "payer" or "payee" as defined in the paper), and 2) a fee for providing signatures without devoting capital. For example, in order to traverse the logical channel ((A,B),(C,D)) (where A, B, C and D are dedicated users, A is the "payer", and C is the "payee"), C will get a payment for devoting capital and D will get a payment for providing signatures. It's reasonable to assume that the sum of these fees will be larger than the fees charged by most non-hierarchical channels, so it's reasonable to assume that the logical channel ((A,B),(C,D)) will be used more often as the first or last hop in a payment (in order to resize (A,B) or (C,D), but not both of them), rather than as an intermediate hop in a payment.
Q: How do you communicate fees/max_htlc/etc.?
A: Just like you do today, with the only difference being that a specific user within each node should handle the Onion traffic for that node and (and should forward it to the other users within that node, if it's a multi-user node). This user could be called the "leader" for the node. I didn't describe this detail in the paper, but I could add it if you think it would be helpful.
> I think it also implies that channel
> capacity is no longer really something you can gossip very sensibly --
Given the above definitions and assumptions, including the use of channel_update_v2 messages, I think you can gossip channel capacity quite reasonably.
As is shown in the paper, just routing through a logical channel does *not* change its capacity (except for a slight increase due to fees). The only time a channel capacity changes significantly is when it is being resized by being the source or destination of a LN payment. The new capacity will be broadcast in the next weekly channel_update_v2 message. Obviously, there's some potential for an unexpected inability for a channel to route requested payments before the channel_update_v2 message showing a reduced capacity has been received, but nothing actually breaks. Of course, it may be worth sending out a channel_update_v2 message as soon as a channel is resized.
> if you have a factory ((A,B),C,D,E) then every payment through AB to C
> or D or E will decrease AB's channel capacity.
First, let's translate this example into one without factories (as that's the world I'm envisioning). Let's assume we have logical channals ((A,B),C), ((A,B),D) and ((A,B),D).
Let's assume that C sends a payment to some other random user/node G that is routed through AB. In this case, the logical channel between A and B does *not* change capacity (except for growing slightly due to fees)! This is shown in Figures 8 and 9 of the paper (with "BC" in those figures playing the role of "AB" described here). That's the beauty of using logical, rather than physical, channels in the channel graph.
It took me a long time to see this and I obviously have not succeeded in explaining it very well. Please let me know if this example isn't clear, in which case I'll try to find a better way of explaining it.
> You could still gossip the
> capacity of the overall factory, and sum that to get an overall lightning
> network capacity, of course. But a lot of the ways of simplifying it
> also make it harder to do all the nice rebalancing.
I think the above approach solves these problems. Please let me know if I've missed something.
Finally, I realize that creating a world without factories doesn't sound like a good solution for scaling Bitcoin. However, I believe hierarchical channels largely solve the problem of resizing channels off-chain. This still leaves the problem of creating channels off-chain, but I don't think we need factories to solve that problem. The key observation is that we don't need to create a factory with *updatable* state in order to create a huge number of channels off-chain. Instead, we just need something that instantiates all of those off-chain channels in a way that guarantees the off-chain UTXOs creating those factories. I think the solution to this simpler problem is Timeout-Trees, but I still need to write that up. In any case, please consider hierarchical channels as being a partial solution to scaling Bitcoin and wait until I can make the case for Timeout-Trees before evaluating whether or not we need factories.
> Anyway, I've tried a few times now to put some thoughts together on that
> and come up with nothing that I'm happy with, so figured it might be at
> least worth posing explicitly as a problem...
Thanks!
I really appreciate your working through all of these issues.
Regards,
John
[1] https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003470.html
📝 Original message:
> One thing that confuses me about the paper is how to think about routing
> to a "channel" rather than a node -- ie the payment from "E->FG->A" where
> "FG" isn't "F" or "G", but "both of them".
Yes, I found it very difficult to think about, and I kept getting confused between concepts like "user", "node", "channel", and "factory".
The thing that works best for me is to create a clear definition of each of these terms, along with the term "party".
I also think it's best to imagine a world in which there are hierarchical channels, but there are no "factories" per se.
The distinction is meaningful in the sense that a hierarchical channel requires that there are exactly *two* *fixed* "parties" to which the hierarchical channel pays out, and all changes to the balances paid to those parties are made via HTLCs.
In contrast, a "factory" can pay out to an arbitrary number of users or groups of users, and changes to the balances paid to those users or groups of users are made via updating the state of the factory (typically without an HTLC being required).
So, in a world with hierarchical channels, but without factories, we have the following terms:
* User: a person
* Party: a fixed group consisting of one or more users
* Node: a party
- so "party" and "node" are synonyms, with "party" emphasizing the human side of things and "node" emphasizing the graph side of things, such as when discussing the Lightning routing graph
* Physical channel: a layer 2 construct that is funded by an on-chain or an off-chain UTXO, is owned by 2 parties, pays a balance to each of the parties, and updates the parties' balances through HTLCs and a channel protocol
- a physical channel can be non-hierarchical (where each of the 2 parties is a single user) or hierarchical (where at least one party consists of more than 1 user)
- (Footnote: The concept of an "off-chain UTXO" is itself a bit tricky, as it often refers to one of multiple possible transaction outputs, where the transactions or their ancestors conflict, so only one of them can actually be instantiated. For example, a Lightning channel (using the current penalty protocol) can be said to create an off-chain UTXO that pays to one of the users, even though it's not clear which of the users' Commitment transactions will actually be instantiated. In general, an off-chain UTXO is a guaranteed way of getting a certain portion of a specific on-chain UTXO's funds. Let's skip going down this rabbit hole any further and just assume we understand these issues.)
* Logical Channel: a layer 2 construct that consists of all of the physical channels owned by a specific pair of parties
- the size (capacity) of a logical channel is the sum of the sizes of their physical channels
- (Footnote: It's possible, with a significant amount of software development work that I in no way discount, to route a payment through a logical channel where the payment traverses multiple physical channels at the same time. This is done by using separate HTLCs, all sharing the same secret, in each of the physical channels that the payment traverses. I can write more about this if that would be helpful.)
* Lightning Network (LN): a layer 2 network consisting of parties and logical channels between parties
- the LN can be represented as a directed graph, where each party is a node and each logical channel is pair of arcs (one in each direction) between two nodes
- the LN can be used to make payments from one node to another node via a path through the graph
- the purpose of the payment could be to send bitcoin from one user to another (in this case, the payment's source and destination nodes are both 1-user parties)
- the purpose of the payment could be to increase or decrease the size of a logical channel (in this case, the payment's source and/or destination node is a multi-user party)
- (Footnote: Here's another great opportunity for confusion, as the source and/or destination of the payment is a multi-user party, which is defined as more than two users, and yet the purpose of the payment is to resize a logical channel. Obviously, a group of users is different from a logical channel (as shown in the definitions above), so how does making a payment to or from a group of users change the size of a logical channel? The answer is that the logical channel being resized is the one owned by the group of users.)
* Factory: a term that doesn't exist in this world view
Sorry if all of this seems overly pedantic.
It's the only way I've been able to keep these concepts straight in my mind.
I've also found that drawings help clarify these issues, which is why the paper has 20 drawings, many of which are examples of payments and channel graphs.
> It feels like there's a whole
> mass of complications hidden in there from a routing perspective; like how
> do you link "FG" back with "F" and "G", how do you decide fees, how do
> you communicate fees/max_htlc/etc.
In addition to the assumptions and definitions above, we need to use Rusty Russell's idea for channel_update_v2 messages [1].
Regarding the specific issues you raised:
Q: How do you link "FG" back with "F" and "G"?
A: In terms of gossiping and the channel graph, you don't explicitly link "FG" with "F" and "G". "FG", "F" and "G" each have their own identity in the LN's Onion network and each one appears as a separate node in the routing graph, with no explicit linkage between them. Every logical channel in the channel graph is announced with channel_update_v2 messages that provide the channel's current capacity and refer back to an on-chain UTXO. It's possible, but not necessary, that some of the logical channels between "FG" and other nodes, between "F" and other nodes, and between "G" and other nodes, use the same on-chain UTXO for their channel_update_v2 messages.
Q: How do you decide fees?
A: All of the users that own a given logical channel negotiate the routing fees that they want to charge. These routing fees are of two types: 1) a fee for devoting capital (by being the "payer" or "payee" as defined in the paper), and 2) a fee for providing signatures without devoting capital. For example, in order to traverse the logical channel ((A,B),(C,D)) (where A, B, C and D are dedicated users, A is the "payer", and C is the "payee"), C will get a payment for devoting capital and D will get a payment for providing signatures. It's reasonable to assume that the sum of these fees will be larger than the fees charged by most non-hierarchical channels, so it's reasonable to assume that the logical channel ((A,B),(C,D)) will be used more often as the first or last hop in a payment (in order to resize (A,B) or (C,D), but not both of them), rather than as an intermediate hop in a payment.
Q: How do you communicate fees/max_htlc/etc.?
A: Just like you do today, with the only difference being that a specific user within each node should handle the Onion traffic for that node and (and should forward it to the other users within that node, if it's a multi-user node). This user could be called the "leader" for the node. I didn't describe this detail in the paper, but I could add it if you think it would be helpful.
> I think it also implies that channel
> capacity is no longer really something you can gossip very sensibly --
Given the above definitions and assumptions, including the use of channel_update_v2 messages, I think you can gossip channel capacity quite reasonably.
As is shown in the paper, just routing through a logical channel does *not* change its capacity (except for a slight increase due to fees). The only time a channel capacity changes significantly is when it is being resized by being the source or destination of a LN payment. The new capacity will be broadcast in the next weekly channel_update_v2 message. Obviously, there's some potential for an unexpected inability for a channel to route requested payments before the channel_update_v2 message showing a reduced capacity has been received, but nothing actually breaks. Of course, it may be worth sending out a channel_update_v2 message as soon as a channel is resized.
> if you have a factory ((A,B),C,D,E) then every payment through AB to C
> or D or E will decrease AB's channel capacity.
First, let's translate this example into one without factories (as that's the world I'm envisioning). Let's assume we have logical channals ((A,B),C), ((A,B),D) and ((A,B),D).
Let's assume that C sends a payment to some other random user/node G that is routed through AB. In this case, the logical channel between A and B does *not* change capacity (except for growing slightly due to fees)! This is shown in Figures 8 and 9 of the paper (with "BC" in those figures playing the role of "AB" described here). That's the beauty of using logical, rather than physical, channels in the channel graph.
It took me a long time to see this and I obviously have not succeeded in explaining it very well. Please let me know if this example isn't clear, in which case I'll try to find a better way of explaining it.
> You could still gossip the
> capacity of the overall factory, and sum that to get an overall lightning
> network capacity, of course. But a lot of the ways of simplifying it
> also make it harder to do all the nice rebalancing.
I think the above approach solves these problems. Please let me know if I've missed something.
Finally, I realize that creating a world without factories doesn't sound like a good solution for scaling Bitcoin. However, I believe hierarchical channels largely solve the problem of resizing channels off-chain. This still leaves the problem of creating channels off-chain, but I don't think we need factories to solve that problem. The key observation is that we don't need to create a factory with *updatable* state in order to create a huge number of channels off-chain. Instead, we just need something that instantiates all of those off-chain channels in a way that guarantees the off-chain UTXOs creating those factories. I think the solution to this simpler problem is Timeout-Trees, but I still need to write that up. In any case, please consider hierarchical channels as being a partial solution to scaling Bitcoin and wait until I can make the case for Timeout-Trees before evaluating whether or not we need factories.
> Anyway, I've tried a few times now to put some thoughts together on that
> and come up with nothing that I'm happy with, so figured it might be at
> least worth posing explicitly as a problem...
Thanks!
I really appreciate your working through all of these issues.
Regards,
John
[1] https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003470.html