Matt Corallo [ARCHIVE] on Nostr: 📅 Original date posted:2021-08-15 📝 Original message: Thanks, AJ, for kicking ...
📅 Original date posted:2021-08-15
📝 Original message:
Thanks, AJ, for kicking off the thread.
I'm frankly still very confused why we're having these conversations now. In one particular class of applicable routing
algorithms you could use for lightning routing having a base fee makes the algorithm intractably slow, but:
a) to my knowledge, no one has (yet) done any follow-on work to investigate pulling many of the same heuristics Rene et
al use in a Dijkstras/A* algorithm with multiple passes or generating multiple routes in the same pass to see whether
you can emulate the results in a faster algorithm without the drawbacks here,
b) to my knowledge, no one has (yet) done any follow-on work to investigate mapping the base fee to other, more
flow-based-routing-compatible numbers, eg you could convert the base fee to a minimum fee by increasing the "effective"
proportional fees. From what others have commented, this may largely "solve" the issue.
c) to my knowledge, no one has (yet) done any follow-on work to analyze where the proposed algorithm may be most optimal
in the HTLC-value<->channel liquidity ratio ranges. We may find that the proposed algorithm only provides materially
better routing when the HTLC value approaches X% of common network channel liquidity, allowing us to only use it for
large-value payments where we can almost ignore the base fees entirely.
There's real cost to distorting the fee structures on the network away from the costs of node operators, especially as
we move towards requiring and using Real (tm) amounts of capital on routing nodes. If we're relying purely on hobbyists
forever who are operating out of goodwill, we should just remove all fees. If we think Lightning is going to involve
capital with real opportunity cost, matching fees to the costs is important, or at least important enough that we
shouldn't throw it away after one (pretty great) paper and limited further analysis.
Imagine we find some great way to address HTLC slot flooding/DoS attacks (or just chose to do it in a not-great way) by
charging for HTLC slot usage, now we can't fix a critical DoS issue because the routing algorithms we deployed can't
handle the new costing. Instead, we should investigate how we can apply the ideas here with the more complicated fee
structures we have.
Color me an optimist, but I'm quite confident with sufficient elbow grease and heuristics we can get 95% of the way
there. We can and should revisit these conversations if such exploration is done and we find that its not possible, but
until then this all feels incredibly premature.
Matt
On 8/14/21 21:00, Anthony Towns wrote:
> Hey *,
>
> There's been discussions on twitter and elsewhere advocating for
> setting the BOLT#7 fee_base_msat value [0] to zero. I'm just writing
> this to summarise my understanding in a place that's able to easily be
> referenced later.
>
> Setting the base fee to zero has a couple of benefits:
>
> - it means you only have one value to optimise when trying to collect
> the most fees, and one-dimensional optimisation problems are
> obviously easier to write code for than two-dimensional optimisation
> problems
>
> - when finding a route, if all the fees on all the channels are
> proportional only, you'll never have to worry about paying more fees
> just as a result of splitting a payment; that makes routing easier
> (see [1])
>
> So what's the cost? The cost is that there's no longer a fixed minimum
> fee -- so if you try sending a 1sat payment you'll pay 0.1% of the fee
> to send a 1000sat payment, and there may be fixed costs that you have
> in routing payments that you'd like to be compensated for (eg, the
> computational work to update channel state, the bandwith to forward the
> tx, or the opportunity cost for not being able to accept another htlc if
> you've hit your max htlcs per channel limit).
>
> But there's no need to explicitly separate those costs the way we do
> now; instead of charging 1sat base fee and 0.02% proportional fee,
> you can instead just set the 0.02% proportional fee and have a minimum
> payment size of 5000 sats (htlc_minimum_msat=5e6, ~$2), since 0.02%
> of that is 1sat. Nobody will be asking you to route without offering a
> fee of at least 1sat, but all the optimisation steps are easier.
>
> You could go a step further, and have the node side accept smaller
> payments despite the htlc minimum setting: eg, accept a 3000 sat payment
> provided it pays the same fee that a 5000 sat payment would have. That is,
> treat the setting as minimum_fee=1sat, rather than minimum_amount=5000sat;
> so the advertised value is just calculated from the real settings,
> and that nodes that want to send very small values despite having to
> pay high rates can just invert the calculation.
>
> I think something like this approach also makes sense when your channel
> becomes overloaded; eg if you have x HTLC slots available, and y channel
> capacity available, setting a minimum payment size of something like
> y/2/x**2 allows you to accept small payments (good for the network)
> when you're channel is not busy, but reserves the last slots for larger
> payments so that you don't end up missing out on profits because you
> ran out of capacity due to low value spam.
>
> Two other aspects related to this:
>
> At present, I think all the fixed costs are also incurred even when
> a htlc fails, so until we have some way of charging failing txs for
> incurring those costs, it seems a bit backwards to penalise successful
> txs who at least pay a proportional fee for the same thing. Until we've
> got a way of handling that, having zero base fee seems at least fair.
>
> Lower value HTLCs don't need to be included in the commitment transaction
> (if they're below the dust level, they definitely shouldn't be included,
> and if they're less than 1sat they can't be included), and as such don't
> incur all the same fixed costs that HTLCs that are committed too do.
> Having different base fees for microtransactions that incur fewer costs
> would be annoying; so having that be "amortised" into the proportional
> fee might help there too.
>
> I think eltoo can help in two ways by reducing the fixed costs: you no
> longer need to keep HTLC information around permanently, and if you do
> a multilevel channel factory setup, you can probably remove the ~400
> HTLCs per channel at any one time limit. But there's still other fixed
> costs, so I think that would just lower the fixed costs, not remove them
> altogether and isn't a fundamental change.
>
> I think the fixed costs for forwarding a HTLC are very small; something
> like:
>
> 0.02sats -- cost of permanently storing the HTLC info
> (100 bytes, $500/TB/year, 1% discount rate)
> 0.04sats -- compute and bandwidth cost for updating an HTLC ($40/month
> at linode, 1 second of compute)
>
> The opportunity cost of having HTLC slots or Bitcoin locked up until
> the HTLC succeeds/fails could be much more significant, though.
>
> Cheers,
> aj
>
> [0] https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md#the-channel_update-message
> [1] https://basefee.ln.rene-pickhardt.de/
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
📝 Original message:
Thanks, AJ, for kicking off the thread.
I'm frankly still very confused why we're having these conversations now. In one particular class of applicable routing
algorithms you could use for lightning routing having a base fee makes the algorithm intractably slow, but:
a) to my knowledge, no one has (yet) done any follow-on work to investigate pulling many of the same heuristics Rene et
al use in a Dijkstras/A* algorithm with multiple passes or generating multiple routes in the same pass to see whether
you can emulate the results in a faster algorithm without the drawbacks here,
b) to my knowledge, no one has (yet) done any follow-on work to investigate mapping the base fee to other, more
flow-based-routing-compatible numbers, eg you could convert the base fee to a minimum fee by increasing the "effective"
proportional fees. From what others have commented, this may largely "solve" the issue.
c) to my knowledge, no one has (yet) done any follow-on work to analyze where the proposed algorithm may be most optimal
in the HTLC-value<->channel liquidity ratio ranges. We may find that the proposed algorithm only provides materially
better routing when the HTLC value approaches X% of common network channel liquidity, allowing us to only use it for
large-value payments where we can almost ignore the base fees entirely.
There's real cost to distorting the fee structures on the network away from the costs of node operators, especially as
we move towards requiring and using Real (tm) amounts of capital on routing nodes. If we're relying purely on hobbyists
forever who are operating out of goodwill, we should just remove all fees. If we think Lightning is going to involve
capital with real opportunity cost, matching fees to the costs is important, or at least important enough that we
shouldn't throw it away after one (pretty great) paper and limited further analysis.
Imagine we find some great way to address HTLC slot flooding/DoS attacks (or just chose to do it in a not-great way) by
charging for HTLC slot usage, now we can't fix a critical DoS issue because the routing algorithms we deployed can't
handle the new costing. Instead, we should investigate how we can apply the ideas here with the more complicated fee
structures we have.
Color me an optimist, but I'm quite confident with sufficient elbow grease and heuristics we can get 95% of the way
there. We can and should revisit these conversations if such exploration is done and we find that its not possible, but
until then this all feels incredibly premature.
Matt
On 8/14/21 21:00, Anthony Towns wrote:
> Hey *,
>
> There's been discussions on twitter and elsewhere advocating for
> setting the BOLT#7 fee_base_msat value [0] to zero. I'm just writing
> this to summarise my understanding in a place that's able to easily be
> referenced later.
>
> Setting the base fee to zero has a couple of benefits:
>
> - it means you only have one value to optimise when trying to collect
> the most fees, and one-dimensional optimisation problems are
> obviously easier to write code for than two-dimensional optimisation
> problems
>
> - when finding a route, if all the fees on all the channels are
> proportional only, you'll never have to worry about paying more fees
> just as a result of splitting a payment; that makes routing easier
> (see [1])
>
> So what's the cost? The cost is that there's no longer a fixed minimum
> fee -- so if you try sending a 1sat payment you'll pay 0.1% of the fee
> to send a 1000sat payment, and there may be fixed costs that you have
> in routing payments that you'd like to be compensated for (eg, the
> computational work to update channel state, the bandwith to forward the
> tx, or the opportunity cost for not being able to accept another htlc if
> you've hit your max htlcs per channel limit).
>
> But there's no need to explicitly separate those costs the way we do
> now; instead of charging 1sat base fee and 0.02% proportional fee,
> you can instead just set the 0.02% proportional fee and have a minimum
> payment size of 5000 sats (htlc_minimum_msat=5e6, ~$2), since 0.02%
> of that is 1sat. Nobody will be asking you to route without offering a
> fee of at least 1sat, but all the optimisation steps are easier.
>
> You could go a step further, and have the node side accept smaller
> payments despite the htlc minimum setting: eg, accept a 3000 sat payment
> provided it pays the same fee that a 5000 sat payment would have. That is,
> treat the setting as minimum_fee=1sat, rather than minimum_amount=5000sat;
> so the advertised value is just calculated from the real settings,
> and that nodes that want to send very small values despite having to
> pay high rates can just invert the calculation.
>
> I think something like this approach also makes sense when your channel
> becomes overloaded; eg if you have x HTLC slots available, and y channel
> capacity available, setting a minimum payment size of something like
> y/2/x**2 allows you to accept small payments (good for the network)
> when you're channel is not busy, but reserves the last slots for larger
> payments so that you don't end up missing out on profits because you
> ran out of capacity due to low value spam.
>
> Two other aspects related to this:
>
> At present, I think all the fixed costs are also incurred even when
> a htlc fails, so until we have some way of charging failing txs for
> incurring those costs, it seems a bit backwards to penalise successful
> txs who at least pay a proportional fee for the same thing. Until we've
> got a way of handling that, having zero base fee seems at least fair.
>
> Lower value HTLCs don't need to be included in the commitment transaction
> (if they're below the dust level, they definitely shouldn't be included,
> and if they're less than 1sat they can't be included), and as such don't
> incur all the same fixed costs that HTLCs that are committed too do.
> Having different base fees for microtransactions that incur fewer costs
> would be annoying; so having that be "amortised" into the proportional
> fee might help there too.
>
> I think eltoo can help in two ways by reducing the fixed costs: you no
> longer need to keep HTLC information around permanently, and if you do
> a multilevel channel factory setup, you can probably remove the ~400
> HTLCs per channel at any one time limit. But there's still other fixed
> costs, so I think that would just lower the fixed costs, not remove them
> altogether and isn't a fundamental change.
>
> I think the fixed costs for forwarding a HTLC are very small; something
> like:
>
> 0.02sats -- cost of permanently storing the HTLC info
> (100 bytes, $500/TB/year, 1% discount rate)
> 0.04sats -- compute and bandwidth cost for updating an HTLC ($40/month
> at linode, 1 second of compute)
>
> The opportunity cost of having HTLC slots or Bitcoin locked up until
> the HTLC succeeds/fails could be much more significant, though.
>
> Cheers,
> aj
>
> [0] https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md#the-channel_update-message
> [1] https://basefee.ln.rene-pickhardt.de/
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>