René Pickhardt [ARCHIVE] on Nostr: 📅 Original date posted:2022-05-12 📝 Original message: Dear fellow lightning ...
📅 Original date posted:2022-05-12
📝 Original message:
Dear fellow lightning developers,
last week I have started a new repository [0] where I maintain a python
package that can be used to test (and more importantly simulate) the
improvements to payment delivery that we have suggested over the last years
(c.f. [1][2][3]). I kindly invite you to check it out.
Feedback & code review will be highly appreciated if you find the spare
time to do so. Similarly I will be delighted if you can provide a patch or
an extension if necessary. Note while this may already be very useful that
this is work in progress and there are already quite some issues open [4]
and I am supervising several Summer of Bitcoin projects [5] that I expect
to contribute to the repository over the next couple of weeks in various
ways.
You can easily install the **pre-alpha** package (version 0.0.0) with the
PythonPackageIndex via:
:~$ pip install pickhardtpayments
Example code that shows how to use this library can be found in the
`Readme.md` file and the `example` folder [6]. Unfortunately my
presentation at MIT Bitcoin Expo where I first talked about this is still
buried in a long Livestream video (Day 2 Track A starts at 1:16:19) and is
not yet available as a stand alone video but I will maintain a
`Resources.md` file where I link to useful talks & articles [7] and other
resources related to the topic.
One final and very significant way to speed up the computation of the min
cost flow solver that I have included since my last mail [3] was achieved
by pruning the network to remove edges that are highly unlikely to be part
of the payment flow. The current solution is pretty ad-hoc but I am working
on doing this in a more automated / reliable way[8]. In this way I am very
pleased to report that as of now and with the given size of the zeroBaseFee
subnet...
**...The min cost flow solver consistently takes less that 100ms to find
close to optimal flows!**
That being said: Please be aware that the entire runtime of the provided
code is currently much higher. The reason is that for mere convenience I
have used the very slow `networkx` library that is used to store the
ChannelGraph, UncertaintyNetwork and OracleLightningNetwork with a lot of
memory overhead to maintain those data structures and to copy the necessary
data to the min cost flow solver before starting the actual solver (c.f.
[9]).
For safety reasons I deliberately did not provide an API to do mainnet
tests. However to the fellow expert user - who knows what they are doing -
it should be fairly straight forward to conduct mainnet experiments via the
following two steps:
1.) Create your own wrapper to an Lightning Node that exposes the
`send_onion` call [10] in the same way how it is currently being done in
the `OracleLightingNetwok` class and bring your own wrapper to the
`SyncSimulatedPaymentSession` class instead of the oracle that we have
2.) You may wish to make the payment loop [11] inside the
`SyncSimulatedPaymentSession` async.
Of course delivering production ready mainnet code will require more work
as one needs to figure out other constraints like channel reserves, min/max
htlc sizes, available HTLC slots, offline peers, hanging htlcs (stuck
payments),...
Also if you want to do simulations with non uniform distributions of
liquidity or because your LSP has a crawled snapshot of actual Liquidity
you can just bring your own `OracleLightningNetwork` that encodes your
assumed or known ground truth about the network.
I hope all of this makes sense. Feel free to ask me anything that you need
and I am curious to see if this will be useful for you. Note that there is
a notebook that has basically the same code but more documentation and in
particular a glossary of terms that can currently be found at [12].
Thanks to all people that I have been collaborating with so far and to
NTNU, BitMEX, various anonymous donors and Patreons who all helped to
achieve this.
with kind regards Rene Pickhardt
[0]: https://github.com/renepickhardt/pickhardtpayments
[1]:
https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-March/002984.html
[2]: https://arxiv.org/abs/2107.05322
[3]:
https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-March/003510.html
[4]: https://github.com/renepickhardt/pickhardtpayments/issues
[5]:
https://www.summerofbitcoin.org/project-ideas-details?recordId=recJchpFa9tqSZkQ4
[6]: https://github.com/renepickhardt/pickhardtpayments/tree/main/examples
[7]:
https://github.com/renepickhardt/pickhardtpayments/blob/main/Resources.md
[8]: https://github.com/renepickhardt/pickhardtpayments/issues/1
[9]: https://github.com/renepickhardt/pickhardtpayments/issues/6
[10]:
https://github.com/renepickhardt/pickhardtpayments/blob/1121b48ec6bf5fb2dee2b1793f87d489ce3149e3/pickhardtpayments/OracleLightningNetwork.py#L37
[11]:
https://github.com/renepickhardt/pickhardtpayments/blob/1121b48ec6bf5fb2dee2b1793f87d489ce3149e3/pickhardtpayments/SyncSimulatedPaymentSession.py#L351
[12]:
https://github.com/renepickhardt/mpp-splitter/blob/pickhardt-payments-simulation-dev/Pickhardt-Payments-Simulation.ipynb
(includes
a glossary of terms and many explainations)
--
https://ln.rene-pickhardt.de <https://ln.rene-pickhardt.de>;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20220512/4eb88b86/attachment.html>;
📝 Original message:
Dear fellow lightning developers,
last week I have started a new repository [0] where I maintain a python
package that can be used to test (and more importantly simulate) the
improvements to payment delivery that we have suggested over the last years
(c.f. [1][2][3]). I kindly invite you to check it out.
Feedback & code review will be highly appreciated if you find the spare
time to do so. Similarly I will be delighted if you can provide a patch or
an extension if necessary. Note while this may already be very useful that
this is work in progress and there are already quite some issues open [4]
and I am supervising several Summer of Bitcoin projects [5] that I expect
to contribute to the repository over the next couple of weeks in various
ways.
You can easily install the **pre-alpha** package (version 0.0.0) with the
PythonPackageIndex via:
:~$ pip install pickhardtpayments
Example code that shows how to use this library can be found in the
`Readme.md` file and the `example` folder [6]. Unfortunately my
presentation at MIT Bitcoin Expo where I first talked about this is still
buried in a long Livestream video (Day 2 Track A starts at 1:16:19) and is
not yet available as a stand alone video but I will maintain a
`Resources.md` file where I link to useful talks & articles [7] and other
resources related to the topic.
One final and very significant way to speed up the computation of the min
cost flow solver that I have included since my last mail [3] was achieved
by pruning the network to remove edges that are highly unlikely to be part
of the payment flow. The current solution is pretty ad-hoc but I am working
on doing this in a more automated / reliable way[8]. In this way I am very
pleased to report that as of now and with the given size of the zeroBaseFee
subnet...
**...The min cost flow solver consistently takes less that 100ms to find
close to optimal flows!**
That being said: Please be aware that the entire runtime of the provided
code is currently much higher. The reason is that for mere convenience I
have used the very slow `networkx` library that is used to store the
ChannelGraph, UncertaintyNetwork and OracleLightningNetwork with a lot of
memory overhead to maintain those data structures and to copy the necessary
data to the min cost flow solver before starting the actual solver (c.f.
[9]).
For safety reasons I deliberately did not provide an API to do mainnet
tests. However to the fellow expert user - who knows what they are doing -
it should be fairly straight forward to conduct mainnet experiments via the
following two steps:
1.) Create your own wrapper to an Lightning Node that exposes the
`send_onion` call [10] in the same way how it is currently being done in
the `OracleLightingNetwok` class and bring your own wrapper to the
`SyncSimulatedPaymentSession` class instead of the oracle that we have
2.) You may wish to make the payment loop [11] inside the
`SyncSimulatedPaymentSession` async.
Of course delivering production ready mainnet code will require more work
as one needs to figure out other constraints like channel reserves, min/max
htlc sizes, available HTLC slots, offline peers, hanging htlcs (stuck
payments),...
Also if you want to do simulations with non uniform distributions of
liquidity or because your LSP has a crawled snapshot of actual Liquidity
you can just bring your own `OracleLightningNetwork` that encodes your
assumed or known ground truth about the network.
I hope all of this makes sense. Feel free to ask me anything that you need
and I am curious to see if this will be useful for you. Note that there is
a notebook that has basically the same code but more documentation and in
particular a glossary of terms that can currently be found at [12].
Thanks to all people that I have been collaborating with so far and to
NTNU, BitMEX, various anonymous donors and Patreons who all helped to
achieve this.
with kind regards Rene Pickhardt
[0]: https://github.com/renepickhardt/pickhardtpayments
[1]:
https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-March/002984.html
[2]: https://arxiv.org/abs/2107.05322
[3]:
https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-March/003510.html
[4]: https://github.com/renepickhardt/pickhardtpayments/issues
[5]:
https://www.summerofbitcoin.org/project-ideas-details?recordId=recJchpFa9tqSZkQ4
[6]: https://github.com/renepickhardt/pickhardtpayments/tree/main/examples
[7]:
https://github.com/renepickhardt/pickhardtpayments/blob/main/Resources.md
[8]: https://github.com/renepickhardt/pickhardtpayments/issues/1
[9]: https://github.com/renepickhardt/pickhardtpayments/issues/6
[10]:
https://github.com/renepickhardt/pickhardtpayments/blob/1121b48ec6bf5fb2dee2b1793f87d489ce3149e3/pickhardtpayments/OracleLightningNetwork.py#L37
[11]:
https://github.com/renepickhardt/pickhardtpayments/blob/1121b48ec6bf5fb2dee2b1793f87d489ce3149e3/pickhardtpayments/SyncSimulatedPaymentSession.py#L351
[12]:
https://github.com/renepickhardt/mpp-splitter/blob/pickhardt-payments-simulation-dev/Pickhardt-Payments-Simulation.ipynb
(includes
a glossary of terms and many explainations)
--
https://ln.rene-pickhardt.de <https://ln.rene-pickhardt.de>;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20220512/4eb88b86/attachment.html>;