Alex Myers [ARCHIVE] on Nostr: đź“… Original date posted:2022-04-21 đź“ť Original message: Hello Bastien, Thank you ...
đź“… Original date posted:2022-04-21
đź“ť Original message:
Hello Bastien,
Thank you for your feedback. I hope you don't mind I let it percolate for a while.
> Eclair doesn't do any rate-limiting. We wanted to "feel the pain" before adding
> anything, and to be honest we haven't really felt it yet.
I understand the “feel the pain first” approach, but attempting set reconciliation has forced me to confront the issue a bit early.
My thoughts on sync were that set-reconciliation would only be used once a node had fully synced gossip through traditional means (initial_routing_sync / gossip_queries.) There should be many levers to pull in order to help maintain sync after this. I'm going to have to experiment with them a bit before I can claim they are sufficient, but I'm optimistic.
> One thing that may help here from an implementation's point of view is to avoid
> sending a disabled channel update every time a channel goes offline. What
> eclair does to avoid spamming is to only send a disabled channel update when
> someone actually tries to use that channel. Of course, if people choose this
> offline node in their route, you don't have a choice and will need to send a
> disabled channel update, but we've observed that many channels come back
> online before we actually need to use them, so we're saving two channel updates
> (one to disable the channel and one to re-enable it). I think all implementations
> should do this. Is that the case today?
> We could go even further, and when we receive an htlc that should be relayed
> to an offline node, wait a bit to give them an opportunity to come online instead
> of failing the htlc and sending a disabled channel update. Eclair currently doesn'tdo that, but it would be very easy to add.
Core-Lightning also delays sending disabled channel updates in an effort to minimize unnecessary gossip. I hadn’t considered an additional delay before failing an htlc on a disabled channel. That will be interesting to explore in the context of transient disconnects of Tor v3 nodes.
I like the idea of a block_height in the channel update tlv. That would be sufficient to enable a simple rate-limit heuristic for this application anyway. Allowing leeway for the chain tip is no problem. I would also expect most implementations to hold a couple updates in reserve, defaulting to predated updates when available. This would allow a “burst” functionality similar to the current LND/CLN rate-limit, but the responsibility is now placed on the originating node to provide that allowance.
Cheers,
Alex
------- Original Message -------
On Friday, April 15th, 2022 at 2:15 AM, Bastien TEINTURIER <bastien at acinq.fr> wrote:
> Good morning Alex,
>
>> I’ve been investigating set reconciliation as a means to reduce bandwidth
>
>> and redundancy of gossip message propagation.
>
> Cool project, glad to see someone working on it! The main difficulty here will
> indeed be to ensure that the number of differences between sets is bounded.
> We will need to maintain a mechanism to sync the whole graph from scratch
> for new nodes, so the minisketch diff must be efficient enough otherwise nodes
> will just fall back to a full sync way too often (which would waste a lot of
> bandwidth).
>
>> Picking several offending channel ids, and digging further, the majority of these
>
>> appear to be flapping due to Tor or otherwise intermittent connections.
>
> One thing that may help here from an implementation's point of view is to avoid
> sending a disabled channel update every time a channel goes offline. What
> eclair does to avoid spamming is to only send a disabled channel update when
> someone actually tries to use that channel. Of course, if people choose this
> offline node in their route, you don't have a choice and will need to send a
> disabled channel update, but we've observed that many channels come back
> online before we actually need to use them, so we're saving two channel updates
> (one to disable the channel and one to re-enable it). I think all implementations
> should do this. Is that the case today?
>
> We could go even further, and when we receive an htlc that should be relayed
> to an offline node, wait a bit to give them an opportunity to come online instead
> of failing the htlc and sending a disabled channel update. Eclair currently doesn't
> do that, but it would be very easy to add.
>
>> - A common listing of current default rate limits across lightning network implementations.
>
> Eclair doesn't do any rate-limiting. We wanted to "feel the pain" before adding
> anything, and to be honest we haven't really felt it yet.
>
>> which will use a common, simple heuristic to accept or reject a gossip message.
>
>> (Think one channel update per block, or perhaps one per block_height << 5.)
>
> I think it would be easy to come to agreement between implementations and
> restrict channel updates to at most one every N blocks. We simply need to add
> the `block_height` in a tlv in `channel_update` and then we'll be able to actually
> rate-limit based on it. Given how much time it takes to upgrade most of the
> network, it may be a good idea to add the `block_height` tlv now in the spec,
> and act on it later? Unless your work requires bigger changes in channel update
> in which case it will probably be a new message.
>
> Note that it will never be completely accurate though, as different nodes can
> have different blockchain tips. My nodes may be one or two blocks late compared
> to the node that emits the channel update. We need to allow a bit of leeway there.
>
> Cheers,
> Bastien
>
> Le jeu. 14 avr. 2022 Ă 23:06, Alex Myers <alex at endothermic.dev> a Ă©crit :
>
>> Hello lightning developers,
>>
>> I’ve been investigating set reconciliation as a means to reduce bandwidth and redundancy of gossip message propagation. This builds on some earlier work from Rusty using the minisketch library [1]. The idea is that each node will build a sketch representing it’s own gossip set. Alice’s node will encode and transmit this sketch to Bob’s node, where it will be merged with his own sketch, and the differences produced. These differences should ideally be exactly the latest missing gossip of both nodes. Due to size constraints, the set differences will necessarily be encoded, but Bob’s node will be able to identify which gossip Alice is missing, and may then transmit exactly those messages.
>>
>> This process is relatively straightforward, with the caveat that the sets must otherwise match very closely (each sketch has a maximum capacity for differences.) The difficulty here is that each node and lightning implementation may have its own rules for gossip acceptance and propagation. Depending on their gossip partners, not all gossip may propagate to the entire network.
>>
>> Core-lightning implements rate limiting for incoming channel updates and node announcements. The default rate limit is 1 per day, with a burst of 4. I analyzed my node’s gossip over a 14 day period, and found that, of all publicly broadcasting half-channels, 18% of them fell afoul of our spam-limiting rules at least once. [2]
>>
>> Picking several offending channel ids, and digging further, the majority of these appear to be flapping due to Tor or otherwise intermittent connections. Well connected nodes may be more susceptible to this due to more frequent routing attempts, and failures resulting in a returned channel update (which otherwise might not have been broadcast.)A slight relaxation of the rate limit resolves the majority of these cases.
>>
>> A smaller subset of channels broadcast frequent channel updates with minor adjustments to htlc_maximum_msat and fee_proportional_millionths parameters. These nodes appear to be power users, with many channels and large balances. I assume this is automated channel management at work.
>>
>> Core-Lightning has updated rate-limiting in the upcoming release to achieve a higher acceptance of incoming gossip, however, it seems that a broader discussion of rate limits may now be worthwhile. A few immediate ideas:
>>
>> - A common listing of current default rate limits across lightning network implementations.
>>
>> - Internal checks of RPC input to limit or warn of network propagation issues if certain rates are exceeded.
>>
>> - A commonly adopted rate-limit standard.
>>
>> My aim is a set reconciliation gossip type, which will use a common, simple heuristic to accept or reject a gossip message. (Think one channel update per block, or perhaps one per block_height << 5.) See my github for my current draft. [3] This solution allows tighter consensus, yet suffers from the same problem as original anti-spam measures – it remains somewhat arbitrary. I would like to start a conversation regarding gossip propagation, channel_update and node_announcement usage, and perhaps even bandwidth goals for syncing gossip in the future (how about a million channels?) This would aid in the development of gossip set reconciliation, but could also benefit current node connection and routing reliability more generally.
>>
>> Thanks,
>>
>> Alex
>>
>> [1] https://github.com/sipa/minisketch
>>
>> [2] https://github.com/endothermicdev/lnspammityspam/blob/main/sampleoutput.txt
>>
>> [3] https://github.com/endothermicdev/lightning-rfc/blob/gossip-minisketch/07-routing-gossip.md#set-reconciliation
>>
>> _______________________________________________
>> Lightning-dev mailing list
>> Lightning-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20220421/a2ddf84f/attachment-0001.html>;
đź“ť Original message:
Hello Bastien,
Thank you for your feedback. I hope you don't mind I let it percolate for a while.
> Eclair doesn't do any rate-limiting. We wanted to "feel the pain" before adding
> anything, and to be honest we haven't really felt it yet.
I understand the “feel the pain first” approach, but attempting set reconciliation has forced me to confront the issue a bit early.
My thoughts on sync were that set-reconciliation would only be used once a node had fully synced gossip through traditional means (initial_routing_sync / gossip_queries.) There should be many levers to pull in order to help maintain sync after this. I'm going to have to experiment with them a bit before I can claim they are sufficient, but I'm optimistic.
> One thing that may help here from an implementation's point of view is to avoid
> sending a disabled channel update every time a channel goes offline. What
> eclair does to avoid spamming is to only send a disabled channel update when
> someone actually tries to use that channel. Of course, if people choose this
> offline node in their route, you don't have a choice and will need to send a
> disabled channel update, but we've observed that many channels come back
> online before we actually need to use them, so we're saving two channel updates
> (one to disable the channel and one to re-enable it). I think all implementations
> should do this. Is that the case today?
> We could go even further, and when we receive an htlc that should be relayed
> to an offline node, wait a bit to give them an opportunity to come online instead
> of failing the htlc and sending a disabled channel update. Eclair currently doesn'tdo that, but it would be very easy to add.
Core-Lightning also delays sending disabled channel updates in an effort to minimize unnecessary gossip. I hadn’t considered an additional delay before failing an htlc on a disabled channel. That will be interesting to explore in the context of transient disconnects of Tor v3 nodes.
I like the idea of a block_height in the channel update tlv. That would be sufficient to enable a simple rate-limit heuristic for this application anyway. Allowing leeway for the chain tip is no problem. I would also expect most implementations to hold a couple updates in reserve, defaulting to predated updates when available. This would allow a “burst” functionality similar to the current LND/CLN rate-limit, but the responsibility is now placed on the originating node to provide that allowance.
Cheers,
Alex
------- Original Message -------
On Friday, April 15th, 2022 at 2:15 AM, Bastien TEINTURIER <bastien at acinq.fr> wrote:
> Good morning Alex,
>
>> I’ve been investigating set reconciliation as a means to reduce bandwidth
>
>> and redundancy of gossip message propagation.
>
> Cool project, glad to see someone working on it! The main difficulty here will
> indeed be to ensure that the number of differences between sets is bounded.
> We will need to maintain a mechanism to sync the whole graph from scratch
> for new nodes, so the minisketch diff must be efficient enough otherwise nodes
> will just fall back to a full sync way too often (which would waste a lot of
> bandwidth).
>
>> Picking several offending channel ids, and digging further, the majority of these
>
>> appear to be flapping due to Tor or otherwise intermittent connections.
>
> One thing that may help here from an implementation's point of view is to avoid
> sending a disabled channel update every time a channel goes offline. What
> eclair does to avoid spamming is to only send a disabled channel update when
> someone actually tries to use that channel. Of course, if people choose this
> offline node in their route, you don't have a choice and will need to send a
> disabled channel update, but we've observed that many channels come back
> online before we actually need to use them, so we're saving two channel updates
> (one to disable the channel and one to re-enable it). I think all implementations
> should do this. Is that the case today?
>
> We could go even further, and when we receive an htlc that should be relayed
> to an offline node, wait a bit to give them an opportunity to come online instead
> of failing the htlc and sending a disabled channel update. Eclair currently doesn't
> do that, but it would be very easy to add.
>
>> - A common listing of current default rate limits across lightning network implementations.
>
> Eclair doesn't do any rate-limiting. We wanted to "feel the pain" before adding
> anything, and to be honest we haven't really felt it yet.
>
>> which will use a common, simple heuristic to accept or reject a gossip message.
>
>> (Think one channel update per block, or perhaps one per block_height << 5.)
>
> I think it would be easy to come to agreement between implementations and
> restrict channel updates to at most one every N blocks. We simply need to add
> the `block_height` in a tlv in `channel_update` and then we'll be able to actually
> rate-limit based on it. Given how much time it takes to upgrade most of the
> network, it may be a good idea to add the `block_height` tlv now in the spec,
> and act on it later? Unless your work requires bigger changes in channel update
> in which case it will probably be a new message.
>
> Note that it will never be completely accurate though, as different nodes can
> have different blockchain tips. My nodes may be one or two blocks late compared
> to the node that emits the channel update. We need to allow a bit of leeway there.
>
> Cheers,
> Bastien
>
> Le jeu. 14 avr. 2022 Ă 23:06, Alex Myers <alex at endothermic.dev> a Ă©crit :
>
>> Hello lightning developers,
>>
>> I’ve been investigating set reconciliation as a means to reduce bandwidth and redundancy of gossip message propagation. This builds on some earlier work from Rusty using the minisketch library [1]. The idea is that each node will build a sketch representing it’s own gossip set. Alice’s node will encode and transmit this sketch to Bob’s node, where it will be merged with his own sketch, and the differences produced. These differences should ideally be exactly the latest missing gossip of both nodes. Due to size constraints, the set differences will necessarily be encoded, but Bob’s node will be able to identify which gossip Alice is missing, and may then transmit exactly those messages.
>>
>> This process is relatively straightforward, with the caveat that the sets must otherwise match very closely (each sketch has a maximum capacity for differences.) The difficulty here is that each node and lightning implementation may have its own rules for gossip acceptance and propagation. Depending on their gossip partners, not all gossip may propagate to the entire network.
>>
>> Core-lightning implements rate limiting for incoming channel updates and node announcements. The default rate limit is 1 per day, with a burst of 4. I analyzed my node’s gossip over a 14 day period, and found that, of all publicly broadcasting half-channels, 18% of them fell afoul of our spam-limiting rules at least once. [2]
>>
>> Picking several offending channel ids, and digging further, the majority of these appear to be flapping due to Tor or otherwise intermittent connections. Well connected nodes may be more susceptible to this due to more frequent routing attempts, and failures resulting in a returned channel update (which otherwise might not have been broadcast.)A slight relaxation of the rate limit resolves the majority of these cases.
>>
>> A smaller subset of channels broadcast frequent channel updates with minor adjustments to htlc_maximum_msat and fee_proportional_millionths parameters. These nodes appear to be power users, with many channels and large balances. I assume this is automated channel management at work.
>>
>> Core-Lightning has updated rate-limiting in the upcoming release to achieve a higher acceptance of incoming gossip, however, it seems that a broader discussion of rate limits may now be worthwhile. A few immediate ideas:
>>
>> - A common listing of current default rate limits across lightning network implementations.
>>
>> - Internal checks of RPC input to limit or warn of network propagation issues if certain rates are exceeded.
>>
>> - A commonly adopted rate-limit standard.
>>
>> My aim is a set reconciliation gossip type, which will use a common, simple heuristic to accept or reject a gossip message. (Think one channel update per block, or perhaps one per block_height << 5.) See my github for my current draft. [3] This solution allows tighter consensus, yet suffers from the same problem as original anti-spam measures – it remains somewhat arbitrary. I would like to start a conversation regarding gossip propagation, channel_update and node_announcement usage, and perhaps even bandwidth goals for syncing gossip in the future (how about a million channels?) This would aid in the development of gossip set reconciliation, but could also benefit current node connection and routing reliability more generally.
>>
>> Thanks,
>>
>> Alex
>>
>> [1] https://github.com/sipa/minisketch
>>
>> [2] https://github.com/endothermicdev/lnspammityspam/blob/main/sampleoutput.txt
>>
>> [3] https://github.com/endothermicdev/lightning-rfc/blob/gossip-minisketch/07-routing-gossip.md#set-reconciliation
>>
>> _______________________________________________
>> Lightning-dev mailing list
>> Lightning-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20220421/a2ddf84f/attachment-0001.html>;