Dr Maxim Orlovsky [ARCHIVE] on Nostr: 📅 Original date posted:2023-04-17 🗒️ Summary of this message: The author ...
📅 Original date posted:2023-04-17
🗒️ Summary of this message: The author thanks the recipient for their analysis and addresses small questions. They clarify the status of RGB implementations and mention documentation efforts.
📝 Original message:Hi David,
Thank you for taking time on doing analysis and writing comments.
I will address all small questions in this reply -- with a follow-up
e-mail dedicated to the technical question from your letter
regarding the problem of "on-publishable conditional statements seemingly
being insecure in multiparty protocols", which requires longer write-up.
> FYI: the RGB-WG organization page links to a repository whose latest
> release is 0.9 and whose latest commit is titled, "Release v.0.9.1", see
> https://github.com/RGB-WG/rgb-node/
Thank you for spotting; we hadn't updated the readme. Now its fixed.
> > Nevertheless, in 2021 we were able to present both RGB powered with a
> > Turing-complete virtual machine (AluVM) [2] and RGB had became
> > operational on
> > Lightning Network [3] using the LNP Node - a complete rust
> > re-implementation of
> > the Lightning protocol made by me at the Association [4].
>
> Could you clarify the status of these implementations?
The status of LNP implementation is "experimental": it is an instrument
for testing new protocols in the scope of the Lightning network, and
its main feature is in having lightweight and modular architecture.
LNP implementation is not anymore required for using RGB on LN - as
Federico already mentioned in his reply, Bitfinex team was able to
independently integrate RGB with existing LDK codebase [1] (I even
didn't know about that before they announced), and there is a WIP on
integrating RGB through CLN extensions. So the question of LNP node
readiness/completeness is unrelated to the question of RGB readiness,
features or technical properties; in my previous e-mail I just had
pointed out that at some (past) point in time we had to work on LN
re-implementation due to some restrictions we had back those days --
like with pay-to-contract commitments which were impossible to
implement in the existing nodes due to architecture limitations --
but with the change to taproot-based commitments in late 2021-
early 2022 this is not true anymore.
> While trying to
> learn about RGB, I noticed that you don't have much completed
> documentation. Previous reviewers also mentioned this and I saw that
> you suggested them to read the code or view your videos.
A lot of documentation was written and is being written. For instance,
if you look at the foundational crates we have in RGB, they are
well documented, containing more docs than the code itself, like in
<https://github.com/LNP-BP/client_side_validation/blob/master/single_use_seals/src/lib.rs>;
Also, we have a number of websites tracking the RGB docs, listed in
<https://rgb.tech/docs/>; and <https://rgb.tech/learn/>; - literally
dozens of them. So your information is outdated.
Of course, much more need to be written - but again, for a small team
of community & self-fundend non-profit with the budget comparable to
a coffee shop. I think we are doing all what we can. If community
needs more docs -- it is welcome to provide more funding or hands in
writing them.
> When reading your code for your LN implementation (LNP), I noticed it
> seemed to be missing a lot of things present in other LN implementations
> I regularly review. For example, I can't find where it supports
> creating or parsing onions, which seems to be a fundamental requirement
> for using LN.
It is there, literally for years, where it should be - in P2P protocol,
BOLT4, as directory and file names suggest:
<https://github.com/LNP-WG/lnp-core/blob/master/lnp2p/src/bolt/bolt4.rs>;
It is based on our other library which provides encryption:
<https://docs.rs/internet2/0.9.0/internet2/presentation/sphinx/index.html>;
> In trying to figure out how it works, I also noticed that
> I couldn't find either unit tests or integration tests---indeed several
> of your applications seem to almost entirely lack the string "test".
> For example, here are LNP-node and RGB-node compared to the four LN
> implementations I regularly monitor:
>
> /tmp/rgb-node$ git grep -i '\<test\>' | wc -l
> 7
RGB Node is not a part of the current RGB release -- starting from
v0.10 RGB do not require a background service. The node is still useful
in server-side environments - but 100% of mobile and most of desktop
users and wallet devs would never need to touch it; thus the update
of the node to v0.10 will come later. Anyway, there is no reason of
doing its extensive test coverage since it its role to be a wrapper
around RGB libraries (existing in other repositories) which contain
100% of RGB consensus and applied business logic. Node just manages
threads and file I/O - not the stuff which is test-covered first of
all (and even for that task it uses others of our libraries with a
like io-react, used in other high-load projects. (BTW pls pay
attention to how such libs are documented:
<https://docs.rs/io-reactor/0.1.2/reactor/>;).
The real RGB code is here, as it is clearly stated in the Github:
* consensus-level
- <github.com/RGB-WG/rgb-core>
- <github.com/BP-WG/bp-core>
- <github.com/LNP-BP/client_side_validation>
* integration libraries - <github.com/RGB-WG/rgb-wallet>
With the update to v0.10 a lot of code was changed, so most of tests
got outdated and were thrown out. Yes, we need more time and
effort to re-do them -- but even taking that into the account,
the core of RGB is covered to much greater extent than the estimations
you have provided:
* client-side-validation library - 80%:
<https://app.codecov.io/gh/LNP-BP/client_side_validation>;
* RGB Core - ~25% (due to significant re-write in v0.10)
<https://app.codecov.io/gh/RGB-WG/rgb-core>;
Finally, the most of code coverage happens due to the used
strict types system [4], which does compilation-type verification
of all data serialization, deserialization and semantic type system.
I.e. with its taken into account, the actual code coverage exceeds
2/3 (>60%). I provide more explanations at the end of the letter.
> /tmp/lnp-node$ git grep -i '\<test\>' | wc -l
> 4
>
> ~/repos/rust-lightning$ git grep -i '\<test\>' | wc -l
> 2008
> ~/repos/cln$ git grep -i '\<test\>' | wc -l
> 1459
> ~/repos/lnd$ git grep -i '\<test\>' | wc -l
> 3547
> ~/repos/eclair$ git grep -i '\<test\>' | wc -l
> 2576
Well, that's a strange way to estimate the code coverage by counting
lines with the word "test". I usually use code coverage reports, like
those coming from codecov.com -- since neither count of unit tests nor
code lines in them are a valid metric to see how the code is
test-covered.
Unlike all those implementations, the node repository doesn't contain
any lightning network business logic or protocols - it is just a "shell"
providing I/O, networking and thread management, not requiring much
testing. Instead, one should look into the actual LNP codebase
implementing BOLT standards, which is in
<https://github.com/LNP-BP/lnp-core>;. And as one might see from test
coverage reports [2] it has 40% of all code covered in tests - which
is not huge, yes, but (see next reply)...
> I realize those are all projects by larger teams than that which works
> on RGB, but a difference of three orders of magnitude is very surprising
> to me. Do you have out-of-tree testing or am I missing something else?
... as I pointed above, there is no "three orders of magnitude" difference
if the comparison is done correctly. Of course mainstream implementation
developed for more than 5 years, with millions invested by large companies/
non-profits and full teams of devs will have more test coverage than
an implementation created at my own personal expense - and I do not
understand what is surprising here :) I wrote about this implementation
to attract more interest from the community devs who may be interested
in joining to work/use an independent lightning re-implementation with
more open architecture allowing much larger protocol-level customizations
than any other mainstream implementation out there.
> As your replies to previous reviewers also mentioned that they should
> view your Youtube videos, I also tried that.
I had never replied to previous _reviewers_ like that. For instance, the
protocol was reviewed by Peter Todd and Federico Tenga, as well as by
other devs from the community, and I am sure they can confirm that the
communications were complete and I was providing all the required answers
and comments. If you are talking about Ruben Somsen, he was never
introduced to me as a reviewer, nor wrote to me anything other than
several sarcastic tweets -- and I rather see him as an internet troll,
who, notwithstanding verbal communications and explanations I
had provided him over phone call, continues to spread miss-information
positioning himself as "RGB expert" -- while still demonstrating absence
of desire of reading any recent updates we had even when provided links.
> I focused on the ones
> discussing LNP, as LN is something I know fairly well,
But this discussion is an offtopic to RGB and this letter. I am fine to
have it, but let's move it to a separate thread.
> I skimmed them quite fast, but I couldn't find any demos where you
> progressed beyond using LNP to open a channel with another node. E.g.,
> they seemed to stop at the same point as this demo:
> https://github.com/LNP-WG/lnp-node/blob/c402decc9ff5b557a9e3d542f74e2fd6ed856742/doc/demo-alpha.4/README.md
The current LNP Node version is v0.9, so v0.4-alpha is extremely outdated,
as well a doc there. For sure the doc needs an update. These are more
recent demos of node operations, with a remote CLN node as of end of 2021:
* https://youtu.be/DoNfKLLjzsY
* https://youtu.be/L6JlIQXbl6Y
> My understanding of the basic goal of RGB from years ago was that it
> would allow ordinary users to define new assets on Bitcoin in a way that
> would allow those assets to be transferred over LN. As far as I can
> tell, it doesn't do that yet, not even in a way that's accessible to a
> power user such as myself.
I do not know how to compare the power of users, but, for instance it
was successfully demonstrated by an independent Bitfinex team using BDK
and LDK last month on the stage of Lightning Tuscany Summit. Of course
there yet no fancy UI all around, but absence of UI says nothing about
technology or its readiness. In fact, several teams are working on the
first UI apps using RGB, so this thing will change - and it is not a
task for a non-profit protocol-level research organization, but rather
for commercial ventures.
> Even for that original goal, there are
> several problems outstanding---problems which will likely require
> significant research and experimentation to overcome, e.g.[2].
> Instead of tackling those problems and building upon existing wallet and
> LN libraries, I see an ambitious effort at reimplementation and massive
> scope creep.
Sorry for not meeting your expectations about how I should use my time,
funds and funds of those who support such developments :)
The real part of the story is that 80% of the time we were working
on solving the original scope you pointed out. It just appeared that
there were much more hidden stones (than was expected back in 2018)
in doing the idea of "assets over Lightning":
* the original single-use-seals didn't worked for the world of multiple
assets which may co-exists, which required development of a set of
quite complex protocols (still being probably most complex part
of RGB);
* the original pay-to-contract commitment scheme did work badly on
practice, being poorly inter-operable with existing lightning
implementations (which took a year of work on LNP node in total,
as I mentioned above) and leading to hard problems in wallet
integration and compatibility with all existing frameworks. We were
solving that problem until Taproot had arrived - and then we switched
to Taproot-based commitment scheme. It took another half of year
to update the whole code base - but the reward was compatibility
with BDK, LDK and other existing tools in the ecosystem, so your
comment on "Instead of tackling those problems and building upon
existing wallet and LN libraries" is not valid;
* as a part of the above, I had to significantly contribute to
taproot implementation and maintenance of rust-bitcoin project,
being the most active project contributor during the course of
2019-2022 [3] (="building upon existing wallet libraries);
* a lot of time was spent on privacy-related improvements, since none
of the original authors nor sponsors had seen any value about putting
"colored coins on lightning" without worriyng about the privacy;
and only after that comes some "fancy stuff", like doing a dedicated
virtual machine and rich data type system -- which primary features
were not "Turing completeness" but ability of formal verification and
compiler-time safety guarantees. For instance, the developed type
system [4] allows not just to "compile" any rust data type into a
smart contract, but also to prove that all data types used in
consensus code do not mutate between releases. If bitcoin had that
type system, there would be no "unnoticed" hardforks or risks of them
in a future -- and I think not implementing such system in doing a
new consensus protocol would be a poor decision. And the last, but
not least, I do not want to be in such situation
<https://twitter.com/fiatjaf/status/1647976362374844416?s=61&t=H4U6Q30N4eanvS4GjyAt4g>;
preferring to spend a year more on a proper design instead of
"shipping earlier". What is acceptable for networking protocols
can't be acceptable for consensus-level protocols, which, unlike
even blockchain consensus can't have softforks.
----
Overall, I am quite negatively impressed with the amount of
misinterpretations and estimates which are simply wrong - and I know
that they are being floated around working like fake news.
Unfortunately writing answers to such claims takes a lot of time -
and takes it from doing other stuff like writing docs, tests or
integrations. I understand that this letter will be read by many,
thus I did address some of most common misconceptions. I am doing
that on bitcoin-dev mail list, but not elsewhere, since if I were
doing that on Twitter I would never have time for anything else.
[1]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-April/021558.html
[2]: https://app.codecov.io/gh/LNP-WG/lnp-core
[3]: https://github.com/rust-bitcoin/rust-bitcoin/graphs/contributors?from=2019-01-11&to=2022-04-11&type=c
[4]: https://www.strict-types.org
🗒️ Summary of this message: The author thanks the recipient for their analysis and addresses small questions. They clarify the status of RGB implementations and mention documentation efforts.
📝 Original message:Hi David,
Thank you for taking time on doing analysis and writing comments.
I will address all small questions in this reply -- with a follow-up
e-mail dedicated to the technical question from your letter
regarding the problem of "on-publishable conditional statements seemingly
being insecure in multiparty protocols", which requires longer write-up.
> FYI: the RGB-WG organization page links to a repository whose latest
> release is 0.9 and whose latest commit is titled, "Release v.0.9.1", see
> https://github.com/RGB-WG/rgb-node/
Thank you for spotting; we hadn't updated the readme. Now its fixed.
> > Nevertheless, in 2021 we were able to present both RGB powered with a
> > Turing-complete virtual machine (AluVM) [2] and RGB had became
> > operational on
> > Lightning Network [3] using the LNP Node - a complete rust
> > re-implementation of
> > the Lightning protocol made by me at the Association [4].
>
> Could you clarify the status of these implementations?
The status of LNP implementation is "experimental": it is an instrument
for testing new protocols in the scope of the Lightning network, and
its main feature is in having lightweight and modular architecture.
LNP implementation is not anymore required for using RGB on LN - as
Federico already mentioned in his reply, Bitfinex team was able to
independently integrate RGB with existing LDK codebase [1] (I even
didn't know about that before they announced), and there is a WIP on
integrating RGB through CLN extensions. So the question of LNP node
readiness/completeness is unrelated to the question of RGB readiness,
features or technical properties; in my previous e-mail I just had
pointed out that at some (past) point in time we had to work on LN
re-implementation due to some restrictions we had back those days --
like with pay-to-contract commitments which were impossible to
implement in the existing nodes due to architecture limitations --
but with the change to taproot-based commitments in late 2021-
early 2022 this is not true anymore.
> While trying to
> learn about RGB, I noticed that you don't have much completed
> documentation. Previous reviewers also mentioned this and I saw that
> you suggested them to read the code or view your videos.
A lot of documentation was written and is being written. For instance,
if you look at the foundational crates we have in RGB, they are
well documented, containing more docs than the code itself, like in
<https://github.com/LNP-BP/client_side_validation/blob/master/single_use_seals/src/lib.rs>;
Also, we have a number of websites tracking the RGB docs, listed in
<https://rgb.tech/docs/>; and <https://rgb.tech/learn/>; - literally
dozens of them. So your information is outdated.
Of course, much more need to be written - but again, for a small team
of community & self-fundend non-profit with the budget comparable to
a coffee shop. I think we are doing all what we can. If community
needs more docs -- it is welcome to provide more funding or hands in
writing them.
> When reading your code for your LN implementation (LNP), I noticed it
> seemed to be missing a lot of things present in other LN implementations
> I regularly review. For example, I can't find where it supports
> creating or parsing onions, which seems to be a fundamental requirement
> for using LN.
It is there, literally for years, where it should be - in P2P protocol,
BOLT4, as directory and file names suggest:
<https://github.com/LNP-WG/lnp-core/blob/master/lnp2p/src/bolt/bolt4.rs>;
It is based on our other library which provides encryption:
<https://docs.rs/internet2/0.9.0/internet2/presentation/sphinx/index.html>;
> In trying to figure out how it works, I also noticed that
> I couldn't find either unit tests or integration tests---indeed several
> of your applications seem to almost entirely lack the string "test".
> For example, here are LNP-node and RGB-node compared to the four LN
> implementations I regularly monitor:
>
> /tmp/rgb-node$ git grep -i '\<test\>' | wc -l
> 7
RGB Node is not a part of the current RGB release -- starting from
v0.10 RGB do not require a background service. The node is still useful
in server-side environments - but 100% of mobile and most of desktop
users and wallet devs would never need to touch it; thus the update
of the node to v0.10 will come later. Anyway, there is no reason of
doing its extensive test coverage since it its role to be a wrapper
around RGB libraries (existing in other repositories) which contain
100% of RGB consensus and applied business logic. Node just manages
threads and file I/O - not the stuff which is test-covered first of
all (and even for that task it uses others of our libraries with a
like io-react, used in other high-load projects. (BTW pls pay
attention to how such libs are documented:
<https://docs.rs/io-reactor/0.1.2/reactor/>;).
The real RGB code is here, as it is clearly stated in the Github:
* consensus-level
- <github.com/RGB-WG/rgb-core>
- <github.com/BP-WG/bp-core>
- <github.com/LNP-BP/client_side_validation>
* integration libraries - <github.com/RGB-WG/rgb-wallet>
With the update to v0.10 a lot of code was changed, so most of tests
got outdated and were thrown out. Yes, we need more time and
effort to re-do them -- but even taking that into the account,
the core of RGB is covered to much greater extent than the estimations
you have provided:
* client-side-validation library - 80%:
<https://app.codecov.io/gh/LNP-BP/client_side_validation>;
* RGB Core - ~25% (due to significant re-write in v0.10)
<https://app.codecov.io/gh/RGB-WG/rgb-core>;
Finally, the most of code coverage happens due to the used
strict types system [4], which does compilation-type verification
of all data serialization, deserialization and semantic type system.
I.e. with its taken into account, the actual code coverage exceeds
2/3 (>60%). I provide more explanations at the end of the letter.
> /tmp/lnp-node$ git grep -i '\<test\>' | wc -l
> 4
>
> ~/repos/rust-lightning$ git grep -i '\<test\>' | wc -l
> 2008
> ~/repos/cln$ git grep -i '\<test\>' | wc -l
> 1459
> ~/repos/lnd$ git grep -i '\<test\>' | wc -l
> 3547
> ~/repos/eclair$ git grep -i '\<test\>' | wc -l
> 2576
Well, that's a strange way to estimate the code coverage by counting
lines with the word "test". I usually use code coverage reports, like
those coming from codecov.com -- since neither count of unit tests nor
code lines in them are a valid metric to see how the code is
test-covered.
Unlike all those implementations, the node repository doesn't contain
any lightning network business logic or protocols - it is just a "shell"
providing I/O, networking and thread management, not requiring much
testing. Instead, one should look into the actual LNP codebase
implementing BOLT standards, which is in
<https://github.com/LNP-BP/lnp-core>;. And as one might see from test
coverage reports [2] it has 40% of all code covered in tests - which
is not huge, yes, but (see next reply)...
> I realize those are all projects by larger teams than that which works
> on RGB, but a difference of three orders of magnitude is very surprising
> to me. Do you have out-of-tree testing or am I missing something else?
... as I pointed above, there is no "three orders of magnitude" difference
if the comparison is done correctly. Of course mainstream implementation
developed for more than 5 years, with millions invested by large companies/
non-profits and full teams of devs will have more test coverage than
an implementation created at my own personal expense - and I do not
understand what is surprising here :) I wrote about this implementation
to attract more interest from the community devs who may be interested
in joining to work/use an independent lightning re-implementation with
more open architecture allowing much larger protocol-level customizations
than any other mainstream implementation out there.
> As your replies to previous reviewers also mentioned that they should
> view your Youtube videos, I also tried that.
I had never replied to previous _reviewers_ like that. For instance, the
protocol was reviewed by Peter Todd and Federico Tenga, as well as by
other devs from the community, and I am sure they can confirm that the
communications were complete and I was providing all the required answers
and comments. If you are talking about Ruben Somsen, he was never
introduced to me as a reviewer, nor wrote to me anything other than
several sarcastic tweets -- and I rather see him as an internet troll,
who, notwithstanding verbal communications and explanations I
had provided him over phone call, continues to spread miss-information
positioning himself as "RGB expert" -- while still demonstrating absence
of desire of reading any recent updates we had even when provided links.
> I focused on the ones
> discussing LNP, as LN is something I know fairly well,
But this discussion is an offtopic to RGB and this letter. I am fine to
have it, but let's move it to a separate thread.
> I skimmed them quite fast, but I couldn't find any demos where you
> progressed beyond using LNP to open a channel with another node. E.g.,
> they seemed to stop at the same point as this demo:
> https://github.com/LNP-WG/lnp-node/blob/c402decc9ff5b557a9e3d542f74e2fd6ed856742/doc/demo-alpha.4/README.md
The current LNP Node version is v0.9, so v0.4-alpha is extremely outdated,
as well a doc there. For sure the doc needs an update. These are more
recent demos of node operations, with a remote CLN node as of end of 2021:
* https://youtu.be/DoNfKLLjzsY
* https://youtu.be/L6JlIQXbl6Y
> My understanding of the basic goal of RGB from years ago was that it
> would allow ordinary users to define new assets on Bitcoin in a way that
> would allow those assets to be transferred over LN. As far as I can
> tell, it doesn't do that yet, not even in a way that's accessible to a
> power user such as myself.
I do not know how to compare the power of users, but, for instance it
was successfully demonstrated by an independent Bitfinex team using BDK
and LDK last month on the stage of Lightning Tuscany Summit. Of course
there yet no fancy UI all around, but absence of UI says nothing about
technology or its readiness. In fact, several teams are working on the
first UI apps using RGB, so this thing will change - and it is not a
task for a non-profit protocol-level research organization, but rather
for commercial ventures.
> Even for that original goal, there are
> several problems outstanding---problems which will likely require
> significant research and experimentation to overcome, e.g.[2].
> Instead of tackling those problems and building upon existing wallet and
> LN libraries, I see an ambitious effort at reimplementation and massive
> scope creep.
Sorry for not meeting your expectations about how I should use my time,
funds and funds of those who support such developments :)
The real part of the story is that 80% of the time we were working
on solving the original scope you pointed out. It just appeared that
there were much more hidden stones (than was expected back in 2018)
in doing the idea of "assets over Lightning":
* the original single-use-seals didn't worked for the world of multiple
assets which may co-exists, which required development of a set of
quite complex protocols (still being probably most complex part
of RGB);
* the original pay-to-contract commitment scheme did work badly on
practice, being poorly inter-operable with existing lightning
implementations (which took a year of work on LNP node in total,
as I mentioned above) and leading to hard problems in wallet
integration and compatibility with all existing frameworks. We were
solving that problem until Taproot had arrived - and then we switched
to Taproot-based commitment scheme. It took another half of year
to update the whole code base - but the reward was compatibility
with BDK, LDK and other existing tools in the ecosystem, so your
comment on "Instead of tackling those problems and building upon
existing wallet and LN libraries" is not valid;
* as a part of the above, I had to significantly contribute to
taproot implementation and maintenance of rust-bitcoin project,
being the most active project contributor during the course of
2019-2022 [3] (="building upon existing wallet libraries);
* a lot of time was spent on privacy-related improvements, since none
of the original authors nor sponsors had seen any value about putting
"colored coins on lightning" without worriyng about the privacy;
and only after that comes some "fancy stuff", like doing a dedicated
virtual machine and rich data type system -- which primary features
were not "Turing completeness" but ability of formal verification and
compiler-time safety guarantees. For instance, the developed type
system [4] allows not just to "compile" any rust data type into a
smart contract, but also to prove that all data types used in
consensus code do not mutate between releases. If bitcoin had that
type system, there would be no "unnoticed" hardforks or risks of them
in a future -- and I think not implementing such system in doing a
new consensus protocol would be a poor decision. And the last, but
not least, I do not want to be in such situation
<https://twitter.com/fiatjaf/status/1647976362374844416?s=61&t=H4U6Q30N4eanvS4GjyAt4g>;
preferring to spend a year more on a proper design instead of
"shipping earlier". What is acceptable for networking protocols
can't be acceptable for consensus-level protocols, which, unlike
even blockchain consensus can't have softforks.
----
Overall, I am quite negatively impressed with the amount of
misinterpretations and estimates which are simply wrong - and I know
that they are being floated around working like fake news.
Unfortunately writing answers to such claims takes a lot of time -
and takes it from doing other stuff like writing docs, tests or
integrations. I understand that this letter will be read by many,
thus I did address some of most common misconceptions. I am doing
that on bitcoin-dev mail list, but not elsewhere, since if I were
doing that on Twitter I would never have time for anything else.
[1]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-April/021558.html
[2]: https://app.codecov.io/gh/LNP-WG/lnp-core
[3]: https://github.com/rust-bitcoin/rust-bitcoin/graphs/contributors?from=2019-01-11&to=2022-04-11&type=c
[4]: https://www.strict-types.org