Rusty Russell [ARCHIVE] on Nostr: š Original date posted:2015-09-24 š Original message: [ Pieter Wuille Cc'd for ...
š
Original date posted:2015-09-24
š Original message:
[ Pieter Wuille Cc'd for pubkey recovery, search for "recovered" ]
Mats Jerratsch <matsjj at gmail.com> writes:
>> Indeed. Random selection helps, here, but analysis will be interesting.
> How have you ended up with a number of beacons you need? 12 seems so
> low, I canāt imagine so few nodes to support all transacting for even
> 10 minutes..
As we keep the last 100 sets of beacons, the load is spread a little.
And remember that the network rebalances; too many transfers in one
direction and it becomes cheaper to send funds in the other direction.
It's unclear how bad it will be.
> And itās very difficult to estimate how much bitcoin they
> might need for that timeframe, as some people might want to transact
> only few satoshi, while others send multiple bitcoin.
True. I assume if we build it they will come. But what will the come
with?
> We can abstract
> the network towards two nodes at each end and N beacon nodes in the
> middle doing all the transacting. The two nodes at either side are
> separate networks then, which need to figure out their own routing.
Indeed, but it's efficient.
> Furthermore, I donāt really see the big advantage in this scheme. We
> still need to find a route towards these beacons, and often one might
> not be interested in the cheapest one, but would want to add more
> privacy. I think propagating the node-map is pretty efficient in
> generalā¦
It's not the node map (which is pretty static; we've even discussed
making the on-chain anchor transactions be the map), it's the continuous
fee updates.
>> Short term I'm thinking we'll have an IRC channel (a-la early bitcoin)
>> and everyone will advertise their channels there. This is design for
>> the next, more ambitious phase.
>
> Hm I think this would be a wasted effort. While there are probably good libraries to do the job out there, it would still need some implementation. As soon as we can solve the node discovery (and if we want to go with a central server, we might just as well hard-code some reliable nodes into the code for now), we can communicate over the network..
As you note below, we can use IRC for the node discovery too. If we
just have an IRC channel where you send all your routes, we get both.
That lets us design something real, while we work on getting the rest
stable.
>>> Furthermore, we can also include
>>> some byte for a reputation / web-of-trust system. It is completely
>>> optional, if you are able to figure out how to route the payment
>>> across your nodes you are not forced to use this system, and I don't
>>> think it will have severe privacy problems.
>>
>> I think reputation systems will become an overlay, if the basic system
>> proves vulnerable. It's nicer if we don't have to though, as reputation
>> is both hard to encode in normal behaviour, and deeply centralizing.
>
> Yea centralisation is an issue with reputation, we might want to have it rather limited. Have it some way, such that there isnāt much difference between a big node that has been around for long and a relatively new node, as long as both are
> (1) low latency
> (2) highly available
> (3) honest.
(1) can be measured, (2) can be intuited over time, but I have no idea
how to measure (3) :(
>>> Is there any plan how we handle connections currently? I am thinking
>>> about sending the current IP address with this update, such that nodes
>>> can connect to each other after one broke down again, and it would
>>> further support dynamic IPs as well. Could be possible to extend this
>>> to hidden service notation, although this would need some more bytes
>>> to store.
>>
>> So, I'm assuming the basic network comms is organized along channel
>> lines; you have to communicate to the other end of the channel anyway,
>> so that makes sense.
>>
>> This doesn't solve "how do I connect to the network" or "how do I start
>> a channel with this particular node", though. For that, I've floated
>> the idea of reusing the bittorrent DHT, which has extensions to allow
>> just this.
>
> Yea exactly, we can squeeze these information in there. The packages
> are very small. If we end up in a situation where they impede the
> payment transactions, we might want to have some priority mechanism..
Yes, and it's proven and robust in practice. It's also possible
that by adding nodes to the BT DHT we help robustness for everyone.
> The āhow to connect to the networkā can be solved with basic peer
> discovery schemes, like IRC or stable-nodes. Having received the
> topology of the mesh, you can decide which nodes you want to create a
> channel with (maybe we should add a flag somewhere, whether this node
> wants to open a channel with strangers and under which conditions). He
> can directly connect to that node and open a channel with him, or am I
> missing something?
To start, I was thinking you establish channels with 5 random nodes.
Later, we'll probably want something more sophisticated, like 2 nearby
nodes (low ping time), and 3 random nodes. Close nodes make sense
because payments also cluster along geographic lines. Random nodes make
sense for network robustness. Over time, you might open a channel to a
node you end up using regularly; there's heaps of fun heuritics to be
developed here!
I assumed that advertising an address would be optional; so if you don't
want incoming connections, don't advertise.
>> You can squeze some more bytes out of you want:
>> 1) Signature should be 64 bytes (never DER encode).
>> 2) Pubkey can be hashed bitcoin-address style, and recovered from sig.
>
> You can recover the pubkey from the hash and the sig? Why are we putting the pubkey in the scriptSig then? ;)
Because crypto is hard :(
TBH I only learned a few months ago that you can do this.
It helps if you have the (two-bit) recovery id, but you can brute force
it AFAICT. You then check if the pubkey matches the hash you're given.
>> If we're using protobufs, they're extensible, so you can add reputation
>> later if we need it.
>>
>> You also want two signatures, I think: one from each side. Though it
>> makes sense to have a separate teardown message which only needs one
>> sig.
>
> Yea was thinking about this as well, I thought as both nodes will propagate their own message, other nodes can puzzle it out, whether the channel has been confirmed by both parties. It saves one signature too..
Or, in other words, they propagate almost twice as much data :)
>> Unclear what reputation means, here?
>
> I was thinking about storing reputation for some time for all
> nodes. Iām not sure this is a good idea anymore, as it may lead to
> some amount of centralization.. It would be beneficial in this case to
> have the reputation data along with the signature only, to be
> efficient. That way you can always recover the reputation data of all
> nodes of the network for a new node, as you can send along all
> signatures..
You need to flesh out how reputation is built and destroyed. Does
someone with sufficient reputation complain about someone else?
Every time I tried to build a reputation system my head hurt :(
>> BTW I've not been separating nodes and clients in my head-design. Of
>> course, some nodes might not ever generate payments, and some nodes
>> might never route others' payments, but it increases privacy vastly if
>> they *can*, so I'm trying to think about them that way.
>
> For me it is more a connection issue. Nodes should have a 99+% uptime
> and allow for direct incoming connections. Still have to figure it
> out, but of course as a node provider you can always issue a payment
> directly (I think for any outsider, there is no difference between the
> node issuing the payment or some client connected to that node)
Yes, and eventually we end up bridging to other networks, so you really
can't tell.
A weird property is that it's only mildly bad to be unreachable (stuff
just bounced back for re-routing), as long as you don't go unreachable
with HTLCs outstanding. The latter will cause people to drop channels
to you.
>> What do you think about the idea of using the anchor transactions in the
>> blockchain as a map of the network? You could use an OP_RETURN to stash
>> two pubkeysx, that is the two node IDs.
>
> Hm isnāt this difficult to prove? Anyone could have put those pubkeys
> in there, so unless we further put a signature in there, this
> information holds little value I think..
Yeah, good point! We can't fit key+sig in 80 bytes, either. Need to do
something less dumb if we go that route...
> I like the idea in general,
> as it would make fake-nodes more difficult, but I think it is
> difficult to implement in the bitcoin blockchain.
I think I'm uncomfortable because I've tried to err on the side of
releasing as little info as possible.
> Furthermore, how do
> you plan to have the scriptPubKey for the anchor tx then? Ultimately
> it is still P2SH, so no one is really able to recover any data from
> it..
We could actually make this one non-P2SH, since multisig is standard.
That would imply that the address there is the node id.
> As a plus, I am currently working on some simulation project for
> lightning. A lot of the metrics are very difficult to estimate (let
> alone calculate), and often we want to depend on random selection. We
> will be able to deduct a lot of properties from it, and be able to
> determine, whether we need something like the reputation system at
> all, and how many dishonest nodes it takes for some attacks, and how
> efficient routing is, all the good stuff...
Great! BTW my mental goalpost is 1M nodes. More than that is still
interesting, but probably only realistic once we have more/real data.
> Cheers,
> Mats
>
> PS.: This was the last time I opened a new thread incidentally -.- Just disabled the digest optionā¦ ;) Sorry for the inconvenience...
Heh, that's fine. You get a free pass when you've got such
information-dense mails :)
Thanks!
Rusty.
š Original message:
[ Pieter Wuille Cc'd for pubkey recovery, search for "recovered" ]
Mats Jerratsch <matsjj at gmail.com> writes:
>> Indeed. Random selection helps, here, but analysis will be interesting.
> How have you ended up with a number of beacons you need? 12 seems so
> low, I canāt imagine so few nodes to support all transacting for even
> 10 minutes..
As we keep the last 100 sets of beacons, the load is spread a little.
And remember that the network rebalances; too many transfers in one
direction and it becomes cheaper to send funds in the other direction.
It's unclear how bad it will be.
> And itās very difficult to estimate how much bitcoin they
> might need for that timeframe, as some people might want to transact
> only few satoshi, while others send multiple bitcoin.
True. I assume if we build it they will come. But what will the come
with?
> We can abstract
> the network towards two nodes at each end and N beacon nodes in the
> middle doing all the transacting. The two nodes at either side are
> separate networks then, which need to figure out their own routing.
Indeed, but it's efficient.
> Furthermore, I donāt really see the big advantage in this scheme. We
> still need to find a route towards these beacons, and often one might
> not be interested in the cheapest one, but would want to add more
> privacy. I think propagating the node-map is pretty efficient in
> generalā¦
It's not the node map (which is pretty static; we've even discussed
making the on-chain anchor transactions be the map), it's the continuous
fee updates.
>> Short term I'm thinking we'll have an IRC channel (a-la early bitcoin)
>> and everyone will advertise their channels there. This is design for
>> the next, more ambitious phase.
>
> Hm I think this would be a wasted effort. While there are probably good libraries to do the job out there, it would still need some implementation. As soon as we can solve the node discovery (and if we want to go with a central server, we might just as well hard-code some reliable nodes into the code for now), we can communicate over the network..
As you note below, we can use IRC for the node discovery too. If we
just have an IRC channel where you send all your routes, we get both.
That lets us design something real, while we work on getting the rest
stable.
>>> Furthermore, we can also include
>>> some byte for a reputation / web-of-trust system. It is completely
>>> optional, if you are able to figure out how to route the payment
>>> across your nodes you are not forced to use this system, and I don't
>>> think it will have severe privacy problems.
>>
>> I think reputation systems will become an overlay, if the basic system
>> proves vulnerable. It's nicer if we don't have to though, as reputation
>> is both hard to encode in normal behaviour, and deeply centralizing.
>
> Yea centralisation is an issue with reputation, we might want to have it rather limited. Have it some way, such that there isnāt much difference between a big node that has been around for long and a relatively new node, as long as both are
> (1) low latency
> (2) highly available
> (3) honest.
(1) can be measured, (2) can be intuited over time, but I have no idea
how to measure (3) :(
>>> Is there any plan how we handle connections currently? I am thinking
>>> about sending the current IP address with this update, such that nodes
>>> can connect to each other after one broke down again, and it would
>>> further support dynamic IPs as well. Could be possible to extend this
>>> to hidden service notation, although this would need some more bytes
>>> to store.
>>
>> So, I'm assuming the basic network comms is organized along channel
>> lines; you have to communicate to the other end of the channel anyway,
>> so that makes sense.
>>
>> This doesn't solve "how do I connect to the network" or "how do I start
>> a channel with this particular node", though. For that, I've floated
>> the idea of reusing the bittorrent DHT, which has extensions to allow
>> just this.
>
> Yea exactly, we can squeeze these information in there. The packages
> are very small. If we end up in a situation where they impede the
> payment transactions, we might want to have some priority mechanism..
Yes, and it's proven and robust in practice. It's also possible
that by adding nodes to the BT DHT we help robustness for everyone.
> The āhow to connect to the networkā can be solved with basic peer
> discovery schemes, like IRC or stable-nodes. Having received the
> topology of the mesh, you can decide which nodes you want to create a
> channel with (maybe we should add a flag somewhere, whether this node
> wants to open a channel with strangers and under which conditions). He
> can directly connect to that node and open a channel with him, or am I
> missing something?
To start, I was thinking you establish channels with 5 random nodes.
Later, we'll probably want something more sophisticated, like 2 nearby
nodes (low ping time), and 3 random nodes. Close nodes make sense
because payments also cluster along geographic lines. Random nodes make
sense for network robustness. Over time, you might open a channel to a
node you end up using regularly; there's heaps of fun heuritics to be
developed here!
I assumed that advertising an address would be optional; so if you don't
want incoming connections, don't advertise.
>> You can squeze some more bytes out of you want:
>> 1) Signature should be 64 bytes (never DER encode).
>> 2) Pubkey can be hashed bitcoin-address style, and recovered from sig.
>
> You can recover the pubkey from the hash and the sig? Why are we putting the pubkey in the scriptSig then? ;)
Because crypto is hard :(
TBH I only learned a few months ago that you can do this.
It helps if you have the (two-bit) recovery id, but you can brute force
it AFAICT. You then check if the pubkey matches the hash you're given.
>> If we're using protobufs, they're extensible, so you can add reputation
>> later if we need it.
>>
>> You also want two signatures, I think: one from each side. Though it
>> makes sense to have a separate teardown message which only needs one
>> sig.
>
> Yea was thinking about this as well, I thought as both nodes will propagate their own message, other nodes can puzzle it out, whether the channel has been confirmed by both parties. It saves one signature too..
Or, in other words, they propagate almost twice as much data :)
>> Unclear what reputation means, here?
>
> I was thinking about storing reputation for some time for all
> nodes. Iām not sure this is a good idea anymore, as it may lead to
> some amount of centralization.. It would be beneficial in this case to
> have the reputation data along with the signature only, to be
> efficient. That way you can always recover the reputation data of all
> nodes of the network for a new node, as you can send along all
> signatures..
You need to flesh out how reputation is built and destroyed. Does
someone with sufficient reputation complain about someone else?
Every time I tried to build a reputation system my head hurt :(
>> BTW I've not been separating nodes and clients in my head-design. Of
>> course, some nodes might not ever generate payments, and some nodes
>> might never route others' payments, but it increases privacy vastly if
>> they *can*, so I'm trying to think about them that way.
>
> For me it is more a connection issue. Nodes should have a 99+% uptime
> and allow for direct incoming connections. Still have to figure it
> out, but of course as a node provider you can always issue a payment
> directly (I think for any outsider, there is no difference between the
> node issuing the payment or some client connected to that node)
Yes, and eventually we end up bridging to other networks, so you really
can't tell.
A weird property is that it's only mildly bad to be unreachable (stuff
just bounced back for re-routing), as long as you don't go unreachable
with HTLCs outstanding. The latter will cause people to drop channels
to you.
>> What do you think about the idea of using the anchor transactions in the
>> blockchain as a map of the network? You could use an OP_RETURN to stash
>> two pubkeysx, that is the two node IDs.
>
> Hm isnāt this difficult to prove? Anyone could have put those pubkeys
> in there, so unless we further put a signature in there, this
> information holds little value I think..
Yeah, good point! We can't fit key+sig in 80 bytes, either. Need to do
something less dumb if we go that route...
> I like the idea in general,
> as it would make fake-nodes more difficult, but I think it is
> difficult to implement in the bitcoin blockchain.
I think I'm uncomfortable because I've tried to err on the side of
releasing as little info as possible.
> Furthermore, how do
> you plan to have the scriptPubKey for the anchor tx then? Ultimately
> it is still P2SH, so no one is really able to recover any data from
> it..
We could actually make this one non-P2SH, since multisig is standard.
That would imply that the address there is the node id.
> As a plus, I am currently working on some simulation project for
> lightning. A lot of the metrics are very difficult to estimate (let
> alone calculate), and often we want to depend on random selection. We
> will be able to deduct a lot of properties from it, and be able to
> determine, whether we need something like the reputation system at
> all, and how many dishonest nodes it takes for some attacks, and how
> efficient routing is, all the good stuff...
Great! BTW my mental goalpost is 1M nodes. More than that is still
interesting, but probably only realistic once we have more/real data.
> Cheers,
> Mats
>
> PS.: This was the last time I opened a new thread incidentally -.- Just disabled the digest optionā¦ ;) Sorry for the inconvenience...
Heh, that's fine. You get a free pass when you've got such
information-dense mails :)
Thanks!
Rusty.