Matt Corallo [ARCHIVE] on Nostr: 📅 Original date posted:2022-06-29 📝 Original message: Thanks Bastien for ...
📅 Original date posted:2022-06-29
📝 Original message:
Thanks Bastien for writing this up! This is a pretty trivial and straightforward way to rate-limit
onion messages in a way that allows legitimate users to continue using the system in spite of some
bad actors trying (and failing, due to being rate-limited) to DoS the network.
I do think any spec for this shouldn't make any recommendations about willingness to relay onion
messages for anonymous no-channel third parties, if anything deliberately staying mum on it and
allowing nodes to adapt policy (and probably rate-limit no-channel third-parties before they rate
limit any peer they have a channel with). Ultimately, we have to assume that nodes will attempt to
send onion messages by routing through the existing channel graph, so there's little reason to worry
too much about ensuring ability to relay for anonymous parties.
Better yet, as Val points out, requiring a channel to relay onion messages puts a very real,
nontrivial (in a world of msats) cost to getting an onion messaging channel. Better yet, with
backpressure ability to DoS onion message links isn't denominated in number of messages, but instead
in number of channels you are able to create, making the backpressure system equivalent to today's
HTLC DoS considerations, whereas explicit payment allows an attacker to pay much less to break the
system.
As for the proposal to charge for onion messages, I'm still not at all sure where its coming from.
It seems to flow from a classic "have a hammer (a system to make micropayments for things), better
turn this problem into a nail (by making users pay for it)" approach, but it doesn't actually solve
the problem at hand.
Even if you charge for onion messages, users may legitimately want to send a bunch of payments in
bulk, and trivially overflow a home or Tor nodes' bandwidth. The only response to that, whether its
a DoS attack or a legitimate user, is to rate-limit, and to rate-limit in a way that tells the user
sending the messages to back off! Sure, you could do that by failing onion messages with an error
that updates the fee you charge, but you're ultimately doing a poor-man's (or, I suppose,
rich-man's) version of what Bastien proposes, not adding some fundamental difference.
Ultimately, paying suffers from the standard PoW-for-spam issue - you cannot assign a reasonable
cost that an attacker cares about without impacting the system's usability due to said cost. Indeed,
making it expensive enough to mount a months-long DDoS without impacting legitimate users be pretty
easy - at 1msat per relay of a 1366 byte onion message you can only saturate an average home users'
30Mbps connection for 30 minutes before you rack up a dollar in costs, but if your concern is
whether someone can reasonably trivially take out the network for minutes at a time to make it have
perceptibly high failure rates, no reasonable cost scheme will work. Quite the opposite - the only
reasonable way to respond is to respond to a spike in traffic while maintaining QoS is to rate-limit
by inbound edge!
Ultimately, what we have here is a networking problem, that has to be solved with networking
solutions, not a costing problem, which can be solved with payment. I can only assume that the
desire to add a cost to onion messages ultimately stems from a desire to ensure every possible
avenue for value extraction is given to routing nodes, but I think that desire is misplaced in this
case - the cost of bandwidth is diminutive compared to other costs of routing node operation,
especially when you consider sensible rate-limits as proposed in Bastien's email.
Indeed, if anyone were proposing rate-limits which would allow anything close to enough bandwidth
usage to cause "lightning is turning into Tor and has Tor's problems" to be a legitimate concern I'd
totally agree we should charge for its use. But no one is, nor has anyone ever seriously, to my
knowledge, proposed such a thing. If lightning messages get deployed and start eating up even single
Mbps's on a consistent basis on nodes, we can totally revisit this, its not like we are shutting the
door to any possible costing system if it becomes necessary, but rate-limiting has to happen either
way, so we should start there and see if we need costing, not jump to costing on day one, hampering
utility.
Matt
On 6/29/22 8:22 PM, Olaoluwa Osuntokun wrote:
> Hi t-bast,
>
> Happy to see this finally written up! With this, we have two classes of
> proposals for rate limiting onion messaging:
>
> 1. Back propagation based rate limiting as described here.
>
> 2. Allowing nodes to express a per-message cost for their forwarding
> services, which is described here [1].
>
> I still need to digest everything proposed here, but personally I'm more
> optimistic about the 2nd category than the 1st.
>
> One issue I see w/ the first category is that a single party can flood the
> network and cause nodes to trigger their rate limits, which then affects the
> usability of the onion messages for all other well-behaving parties. An
> example, this might mean I can't fetch invoices, give up after a period of
> time (how long?), then result to a direct connection (perceived payment
> latency accumulated along the way).
>
> With the 2nd route, if an attacker floods the network, they need to directly
> pay for the forwarding usage themselves, though they may also directly cause
> nodes to adjust their forwarding rate accordingly. However in this case, the
> attacker has incurred a concrete cost, and even if the rates rise, then
> those that really need the service (notifying an LSP that a user is online
> or w/e) can continue to pay that new rate. In other words, by _pricing_ the
> resource utilization, demand preferences can be exchanged, leading to more
> efficient long term resource allocation.
>
> W.r.t this topic, one event that imo is worth pointing out is that a very
> popular onion routing system, Tor, has been facing a severe DDoS attack that
> has lasted weeks, and isn't yet fully resolved [2]. The on going flooding
> attack on Tor has actually started to affect LN (iirc over half of all
> public routing nodes w/ an advertised address are tor-only), and other
> related systems like Umbrel that 100% rely on tor for networking traversal.
> Funnily enough, Tor developers have actually suggested adding some PoW to
> attempt to mitigate DDoS attacks [3]. In that same post they throw around
> the idea of using anonymous tokens to allow nodes to give them to "good"
> clients, which is pretty similar to my lofty Forwarding Pass idea as relates
> to onion messaging, and also general HTLC jamming mitigation.
>
> In summary, we're not the first to attempt to tackle the problem of rate
> limiting relayed message spam in an anonymous/pseudonymous network, and we
> can probably learn a lot from what is and isn't working w.r.t how Tor
> handles things. As you note near the end of your post, this might just be
> the first avenue in a long line of research to best figure out how to handle
> the spam concerns introduced by onion messaging. From my PoV, it still seems
> to be an open question if the same network can be _both_ a reliable
> micro-payment system _and_ also a reliable arbitrary message transport
> layer. I guess only time will tell...
>
> > The `shared_secret_hash` field contains a BIP 340 tagged hash
>
> Any reason to use the tagged hash here vs just a plain ol HMAC? Under the
> hood, they have a pretty similar construction [4].
>
> [1]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003498.html
> <https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003498.html>;
> [2]: https://status.torproject.org/issues/2022-06-09-network-ddos/
> <https://status.torproject.org/issues/2022-06-09-network-ddos/>;
> [3]: https://blog.torproject.org/stop-the-onion-denial/
> <https://blog.torproject.org/stop-the-onion-denial/>;
> [4]: https://datatracker.ietf.org/doc/html/rfc2104 <https://datatracker.ietf.org/doc/html/rfc2104>;
>
> -- Laolu
>
>
>
> On Wed, Jun 29, 2022 at 1:28 AM Bastien TEINTURIER <bastien at acinq.fr <mailto:bastien at acinq.fr>> wrote:
>
> During the recent Oakland Dev Summit, some lightning engineers got together to discuss DoS
> protection for onion messages. Rusty proposed a very simple rate-limiting scheme that
> statistically propagates back to the correct sender, which we describe in details below.
>
> You can also read this in gist format if that works better for you [1].
>
> Nodes apply per-peer rate limits on _incoming_ onion messages that should be relayed (e.g.
> N/seconds with some burst tolerance). It is recommended to allow more onion messages from
> peers with whom you have channels, for example 10/seconds when you have a channel and 1/second
> when you don't.
>
> When relaying an onion message, nodes keep track of where it came from (by using the `node_id` of
> the peer who sent that message). Nodes only need the last such `node_id` per outgoing connection,
> which ensures the memory footprint is very small. Also, this data doesn't need to be persisted.
>
> Let's walk through an example to illustrate this mechanism:
>
> * Bob receives an onion message from Alice that should be relayed to Carol
> * After relaying that message, Bob stores Alice's `node_id` in its per-connection state with Carol
> * Bob receives an onion message from Eve that should be relayed to Carol
> * After relaying that message, Bob replaces Alice's `node_id` with Eve's `node_id` in its
> per-connection state with Carol
> * Bob receives an onion message from Alice that should be relayed to Dave
> * After relaying that message, Bob stores Alice's `node_id` in its per-connection state with Dave
> * ...
>
> We introduce a new message that will be sent when dropping an incoming onion message because it
> reached rate limits:
>
> 1. type: 515 (`onion_message_drop`)
> 2. data:
> * [`rate_limited`:`u8`]
> * [`shared_secret_hash`:`32*byte`]
>
> Whenever an incoming onion message reaches the rate limit, the receiver sends `onion_message_drop`
> to the sender. The sender looks at its per-connection state to find where the message was coming
> from and relays `onion_message_drop` to the last sender, halving their rate limits with that peer.
>
> If the sender doesn't overflow the rate limit again, the receiver should double the rate limit
> after 30 seconds, until it reaches the default rate limit again.
>
> The flow will look like:
>
> Alice Bob Carol
> | | |
> | onion_message | |
> |------------------------>| |
> | | onion_message |
> | |------------------------>|
> | | onion_message_drop |
> | |<------------------------|
> | onion_message_drop | |
> |<------------------------| |
>
> The `shared_secret_hash` field contains a BIP 340 tagged hash of the Sphinx shared secret of the
> rate limiting peer (in the example above, Carol):
>
> * `shared_secret_hash = SHA256(SHA256("onion_message_drop") || SHA256("onion_message_drop") || sphinx_shared_secret)`
>
> This value is known by the node that created the onion message: if `onion_message_drop` propagates
> all the way back to them, it lets them know which part of the route is congested, allowing them
> to retry through a different path.
>
> Whenever there is some latency between nodes and many onion messages, `onion_message_drop` may
> be relayed to the incorrect incoming peer (since we only store the `node_id` of the _last_ incoming
> peer in our outgoing connection state). The following example highlights this:
>
> Eve Bob Carol
> | onion_message | |
> |------------------------>| onion_message |
> | onion_message |------------------------>|
> |------------------------>| onion_message |
> | onion_message |------------------------>|
> |------------------------>| onion_message |
> |------------------------>|
> Alice | onion_message_drop |
> | onion_message | +----|
> |------------------------>| onion_message | |
> | |--------------------|--->|
> | | | |
> | | | |
> | | | |
> | onion_message_drop |<-------------------+ |
> |<------------------------| |
>
> In this example, Eve is spamming but `onion_message_drop` is propagated back to Alice instead.
> However, this scheme will _statistically_ penalize the right incoming peer (with a probability
> depending on the volume of onion messages that the spamming peer is generating compared to the
> volume of legitimate onion messages).
>
> It is an interesting research problem to find formulas for those probabilities to evaluate how
> efficient this will be against various types of spam. We hope researchers on this list will be
> interested in looking into it and will come up with a good model to evaluate that scheme.
>
> To increase the accuracy of attributing `onion_message_drop`, more data could be stored in the
> future if it becomes necessary. We need more research to quantify how much accuracy would be
> gained by storing more data and making the protocol more complex.
>
> Cheers,
> Bastien
>
> [1]https://gist.github.com/t-bast/e37ee9249d9825e51d260335c94f0fcf <https://gist.github.com/t-bast/e37ee9249d9825e51d260335c94f0fcf>;
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org <mailto:Lightning-dev at lists.linuxfoundation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
> <https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev>;
>
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
📝 Original message:
Thanks Bastien for writing this up! This is a pretty trivial and straightforward way to rate-limit
onion messages in a way that allows legitimate users to continue using the system in spite of some
bad actors trying (and failing, due to being rate-limited) to DoS the network.
I do think any spec for this shouldn't make any recommendations about willingness to relay onion
messages for anonymous no-channel third parties, if anything deliberately staying mum on it and
allowing nodes to adapt policy (and probably rate-limit no-channel third-parties before they rate
limit any peer they have a channel with). Ultimately, we have to assume that nodes will attempt to
send onion messages by routing through the existing channel graph, so there's little reason to worry
too much about ensuring ability to relay for anonymous parties.
Better yet, as Val points out, requiring a channel to relay onion messages puts a very real,
nontrivial (in a world of msats) cost to getting an onion messaging channel. Better yet, with
backpressure ability to DoS onion message links isn't denominated in number of messages, but instead
in number of channels you are able to create, making the backpressure system equivalent to today's
HTLC DoS considerations, whereas explicit payment allows an attacker to pay much less to break the
system.
As for the proposal to charge for onion messages, I'm still not at all sure where its coming from.
It seems to flow from a classic "have a hammer (a system to make micropayments for things), better
turn this problem into a nail (by making users pay for it)" approach, but it doesn't actually solve
the problem at hand.
Even if you charge for onion messages, users may legitimately want to send a bunch of payments in
bulk, and trivially overflow a home or Tor nodes' bandwidth. The only response to that, whether its
a DoS attack or a legitimate user, is to rate-limit, and to rate-limit in a way that tells the user
sending the messages to back off! Sure, you could do that by failing onion messages with an error
that updates the fee you charge, but you're ultimately doing a poor-man's (or, I suppose,
rich-man's) version of what Bastien proposes, not adding some fundamental difference.
Ultimately, paying suffers from the standard PoW-for-spam issue - you cannot assign a reasonable
cost that an attacker cares about without impacting the system's usability due to said cost. Indeed,
making it expensive enough to mount a months-long DDoS without impacting legitimate users be pretty
easy - at 1msat per relay of a 1366 byte onion message you can only saturate an average home users'
30Mbps connection for 30 minutes before you rack up a dollar in costs, but if your concern is
whether someone can reasonably trivially take out the network for minutes at a time to make it have
perceptibly high failure rates, no reasonable cost scheme will work. Quite the opposite - the only
reasonable way to respond is to respond to a spike in traffic while maintaining QoS is to rate-limit
by inbound edge!
Ultimately, what we have here is a networking problem, that has to be solved with networking
solutions, not a costing problem, which can be solved with payment. I can only assume that the
desire to add a cost to onion messages ultimately stems from a desire to ensure every possible
avenue for value extraction is given to routing nodes, but I think that desire is misplaced in this
case - the cost of bandwidth is diminutive compared to other costs of routing node operation,
especially when you consider sensible rate-limits as proposed in Bastien's email.
Indeed, if anyone were proposing rate-limits which would allow anything close to enough bandwidth
usage to cause "lightning is turning into Tor and has Tor's problems" to be a legitimate concern I'd
totally agree we should charge for its use. But no one is, nor has anyone ever seriously, to my
knowledge, proposed such a thing. If lightning messages get deployed and start eating up even single
Mbps's on a consistent basis on nodes, we can totally revisit this, its not like we are shutting the
door to any possible costing system if it becomes necessary, but rate-limiting has to happen either
way, so we should start there and see if we need costing, not jump to costing on day one, hampering
utility.
Matt
On 6/29/22 8:22 PM, Olaoluwa Osuntokun wrote:
> Hi t-bast,
>
> Happy to see this finally written up! With this, we have two classes of
> proposals for rate limiting onion messaging:
>
> 1. Back propagation based rate limiting as described here.
>
> 2. Allowing nodes to express a per-message cost for their forwarding
> services, which is described here [1].
>
> I still need to digest everything proposed here, but personally I'm more
> optimistic about the 2nd category than the 1st.
>
> One issue I see w/ the first category is that a single party can flood the
> network and cause nodes to trigger their rate limits, which then affects the
> usability of the onion messages for all other well-behaving parties. An
> example, this might mean I can't fetch invoices, give up after a period of
> time (how long?), then result to a direct connection (perceived payment
> latency accumulated along the way).
>
> With the 2nd route, if an attacker floods the network, they need to directly
> pay for the forwarding usage themselves, though they may also directly cause
> nodes to adjust their forwarding rate accordingly. However in this case, the
> attacker has incurred a concrete cost, and even if the rates rise, then
> those that really need the service (notifying an LSP that a user is online
> or w/e) can continue to pay that new rate. In other words, by _pricing_ the
> resource utilization, demand preferences can be exchanged, leading to more
> efficient long term resource allocation.
>
> W.r.t this topic, one event that imo is worth pointing out is that a very
> popular onion routing system, Tor, has been facing a severe DDoS attack that
> has lasted weeks, and isn't yet fully resolved [2]. The on going flooding
> attack on Tor has actually started to affect LN (iirc over half of all
> public routing nodes w/ an advertised address are tor-only), and other
> related systems like Umbrel that 100% rely on tor for networking traversal.
> Funnily enough, Tor developers have actually suggested adding some PoW to
> attempt to mitigate DDoS attacks [3]. In that same post they throw around
> the idea of using anonymous tokens to allow nodes to give them to "good"
> clients, which is pretty similar to my lofty Forwarding Pass idea as relates
> to onion messaging, and also general HTLC jamming mitigation.
>
> In summary, we're not the first to attempt to tackle the problem of rate
> limiting relayed message spam in an anonymous/pseudonymous network, and we
> can probably learn a lot from what is and isn't working w.r.t how Tor
> handles things. As you note near the end of your post, this might just be
> the first avenue in a long line of research to best figure out how to handle
> the spam concerns introduced by onion messaging. From my PoV, it still seems
> to be an open question if the same network can be _both_ a reliable
> micro-payment system _and_ also a reliable arbitrary message transport
> layer. I guess only time will tell...
>
> > The `shared_secret_hash` field contains a BIP 340 tagged hash
>
> Any reason to use the tagged hash here vs just a plain ol HMAC? Under the
> hood, they have a pretty similar construction [4].
>
> [1]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003498.html
> <https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003498.html>;
> [2]: https://status.torproject.org/issues/2022-06-09-network-ddos/
> <https://status.torproject.org/issues/2022-06-09-network-ddos/>;
> [3]: https://blog.torproject.org/stop-the-onion-denial/
> <https://blog.torproject.org/stop-the-onion-denial/>;
> [4]: https://datatracker.ietf.org/doc/html/rfc2104 <https://datatracker.ietf.org/doc/html/rfc2104>;
>
> -- Laolu
>
>
>
> On Wed, Jun 29, 2022 at 1:28 AM Bastien TEINTURIER <bastien at acinq.fr <mailto:bastien at acinq.fr>> wrote:
>
> During the recent Oakland Dev Summit, some lightning engineers got together to discuss DoS
> protection for onion messages. Rusty proposed a very simple rate-limiting scheme that
> statistically propagates back to the correct sender, which we describe in details below.
>
> You can also read this in gist format if that works better for you [1].
>
> Nodes apply per-peer rate limits on _incoming_ onion messages that should be relayed (e.g.
> N/seconds with some burst tolerance). It is recommended to allow more onion messages from
> peers with whom you have channels, for example 10/seconds when you have a channel and 1/second
> when you don't.
>
> When relaying an onion message, nodes keep track of where it came from (by using the `node_id` of
> the peer who sent that message). Nodes only need the last such `node_id` per outgoing connection,
> which ensures the memory footprint is very small. Also, this data doesn't need to be persisted.
>
> Let's walk through an example to illustrate this mechanism:
>
> * Bob receives an onion message from Alice that should be relayed to Carol
> * After relaying that message, Bob stores Alice's `node_id` in its per-connection state with Carol
> * Bob receives an onion message from Eve that should be relayed to Carol
> * After relaying that message, Bob replaces Alice's `node_id` with Eve's `node_id` in its
> per-connection state with Carol
> * Bob receives an onion message from Alice that should be relayed to Dave
> * After relaying that message, Bob stores Alice's `node_id` in its per-connection state with Dave
> * ...
>
> We introduce a new message that will be sent when dropping an incoming onion message because it
> reached rate limits:
>
> 1. type: 515 (`onion_message_drop`)
> 2. data:
> * [`rate_limited`:`u8`]
> * [`shared_secret_hash`:`32*byte`]
>
> Whenever an incoming onion message reaches the rate limit, the receiver sends `onion_message_drop`
> to the sender. The sender looks at its per-connection state to find where the message was coming
> from and relays `onion_message_drop` to the last sender, halving their rate limits with that peer.
>
> If the sender doesn't overflow the rate limit again, the receiver should double the rate limit
> after 30 seconds, until it reaches the default rate limit again.
>
> The flow will look like:
>
> Alice Bob Carol
> | | |
> | onion_message | |
> |------------------------>| |
> | | onion_message |
> | |------------------------>|
> | | onion_message_drop |
> | |<------------------------|
> | onion_message_drop | |
> |<------------------------| |
>
> The `shared_secret_hash` field contains a BIP 340 tagged hash of the Sphinx shared secret of the
> rate limiting peer (in the example above, Carol):
>
> * `shared_secret_hash = SHA256(SHA256("onion_message_drop") || SHA256("onion_message_drop") || sphinx_shared_secret)`
>
> This value is known by the node that created the onion message: if `onion_message_drop` propagates
> all the way back to them, it lets them know which part of the route is congested, allowing them
> to retry through a different path.
>
> Whenever there is some latency between nodes and many onion messages, `onion_message_drop` may
> be relayed to the incorrect incoming peer (since we only store the `node_id` of the _last_ incoming
> peer in our outgoing connection state). The following example highlights this:
>
> Eve Bob Carol
> | onion_message | |
> |------------------------>| onion_message |
> | onion_message |------------------------>|
> |------------------------>| onion_message |
> | onion_message |------------------------>|
> |------------------------>| onion_message |
> |------------------------>|
> Alice | onion_message_drop |
> | onion_message | +----|
> |------------------------>| onion_message | |
> | |--------------------|--->|
> | | | |
> | | | |
> | | | |
> | onion_message_drop |<-------------------+ |
> |<------------------------| |
>
> In this example, Eve is spamming but `onion_message_drop` is propagated back to Alice instead.
> However, this scheme will _statistically_ penalize the right incoming peer (with a probability
> depending on the volume of onion messages that the spamming peer is generating compared to the
> volume of legitimate onion messages).
>
> It is an interesting research problem to find formulas for those probabilities to evaluate how
> efficient this will be against various types of spam. We hope researchers on this list will be
> interested in looking into it and will come up with a good model to evaluate that scheme.
>
> To increase the accuracy of attributing `onion_message_drop`, more data could be stored in the
> future if it becomes necessary. We need more research to quantify how much accuracy would be
> gained by storing more data and making the protocol more complex.
>
> Cheers,
> Bastien
>
> [1]https://gist.github.com/t-bast/e37ee9249d9825e51d260335c94f0fcf <https://gist.github.com/t-bast/e37ee9249d9825e51d260335c94f0fcf>;
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org <mailto:Lightning-dev at lists.linuxfoundation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
> <https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev>;
>
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev