Ruben Somsen [ARCHIVE] on Nostr: 📅 Original date posted:2019-04-21 📝 Original message:Hi ZmnSCPxj, Allow me to ...
📅 Original date posted:2019-04-21
📝 Original message:Hi ZmnSCPxj,
Allow me to reply to your post in mixed order (fraud proofs first):
>But peers can be set up to allow you to hear of all chains while denying you proof of the invalidity of some UTXO.
I don't believe this is fundamentally different. In either scenario
you end up on the wrong chain if all your peers are lying to you. One
happens by omission of a fraud proof, while the other happens by
omission of a valid longest chain.
>This is precisely the "data unavailability claim" that shot down the previous fraud proofs
The "data unavailability" issue I was referring to, and which I
believe is the reason why fraud proofs were abandoned, is the
following:
- Alice downloads a block with her full node, but the block is
incomplete (e.g. a transaction is missing).
- Alice reports this to Bob's SPV fraud proof client, who verifies
this by requesting the transaction from the network.
- If Bob can't download it, he rejects the block.
- If Bob can download it, either Alice was malicious, or a miner was
temporarily withholding the data.
- Since Bob can't be certain Alice was being malicious, Bob can't ban
her, which results in a DoS vector where SPV fraud proof clients can
be forced to download all blocks.
We circumvent the data unavailability problem here completely, since
we are only questioning the validity of blocks which are involved in a
fork (expensive and/or rare), and we are simply always downloading
them in full.
If my arguments above hold up, we can use fraud proof commitments as
described in segwit BIP141 [0] instead of UTXO set commitments, which
seems like the more elegant way to achieve the desired outcome.
>Perhaps in combination with BIP157/158 it may be possible, if the filters contain UTXO spends and a BIP158 filter was committed to on-chain. Then a proof of absence could be done by revealing all the BIP158 filters from the UTXO creation to the block being validated, as well as the blocks whose BIP158 filters matched the UTXO and revealing that no, they actually do not spend the UTXO.
Yes, I mentioned something similar to Laolu, but it does seem
computationally expensive to run every input in a block through the
filter of every past block. The fact that BIP157/158 can function
without commitments is also why I suspected we may not necessarily
need UTXO set commitments.
>UTXO sets can only be validated by actually running the entire blockchain, i.e. fullnoding.
It seems to me you can validate uncommitted UTXO sets by comparing
them. Download and compare UTXO set hashes from multiple peers. If
they disagree on a certain block, download that block and the relevant
merkle path(s) from the previous block's UTXO set, and then verify who
is right. Ban the peer who lied. Note that unlike fraud proofs, it is
not possible to lie by omission, but it does assume one of your peers
is honest. Of course this does nothing to dispute your earlier point
that this may not be all that efficient (e.g. full nodes keeping
merkle paths of all prior states).
>What BIP157 does is summarize data that is within a block, thus validating them can be done simply by downloading the block in question.
>UTXO sets summarize data in the entire blockchain, hence proper validation requires downloading the entire blockchain.
Thus it cannot be a comparison point.
It's still possible to lie by omission. Let's say a miner spends some
coins in block N, and spends the exact same coins again in block N+1,
making block N+1 invalid. If the filter for block N is maliciously
constructed, you won't notice the spend in block N, causing you to
think block N+1 is valid. In short, you're still relying on one of
your peers to give you a correct filter. If all your peers lie, you
can always be deceived.
>Tangentially, we cannot just magically commit to anything on the blockchain. [...] if you are adding new information to be committed, that may increase the resource usage of fullnodes. [...] This is probably still better than BIP37 but we should still be aware the additional load on fullnodes.
I agree with all this.
To summarize, this is my current understanding of our options for
enabling light clients to verify a single block in isolation:
1. UTXO set commitments (complex, more resource usage to full nodes)
2. BIP157/158 commitments (expensive for clients to check all filters
to get exclusion proofs)
3. BIP141 fraud proof commitments (assumes fraud proofs will be passed
on to the SPV client)
The debate is still open on whether the options above can be done
without actually committing them into blocks via a soft fork. My
current hunch is "yes" for 1 and 2, and "no" for 3, which would be
unfortunate, because 3 currently seems to me like the more elegant
solution.
-- Ruben Somsen
[0] https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#Compact_fraud_proof_for_SPV_nodes
>UTXO sets can only be validated by actually running the entire blockchain, i.e. fullnoding.
>What BIP157 does is summarize data that is within a block, thus validating them can be done simply by downloading the block in question.
UTXO sets summarize data in the entire blockchain, hence proper
validation requires downloading the entire blockchain.
Thus it cannot be a comparison point.
> > This makes no sense
> > or you trust that every peer you have is not omitting the proof.
>
> It's the latter, you trust every peer you have is not omitting the
> proof. It requires one honest peer. The reason this is acceptable is
> because you're already making that assumption. If none of your peers
> are honest, you have no guarantee of hearing about the chain with the
> most PoW.
But peers can be set up to allow you to hear of all chains while
denying you proof of the invalidity of some UTXO.
This is precisely the "data unavailability claim" that shot down the
previous fraud proofs (i.e. absence of proof is not proof of absence,
and proof of UTXO validity was defined by proof of absence of any
intervening spend of the UTXO).
Perhaps in combination with BIP157/158 it may be possible, if the
filters contain UTXO spends and a BIP158 filter was committed to
on-chain.
Then a proof of absence could be done by revealing all the BIP158
filters from the UTXO creation to the block being validated, as well
as the blocks whose BIP158 filters matched the UTXO and revealing that
no, they actually do not spend the UTXO.
--
Tangentially, we cannot just magically commit to anything on the blockchain.
Header blocks commit to block data and commit to some other header block.
All those header blocks and the block data need to be stored and
transmitted over the network somehow, even though they are "only"
being committed to.
Thus, if you are adding new information to be committed, that may
increase the resource usage of fullnodes.
So if UTXO set commitments, or utreexo commitments, or BIP158 filter
digests, etc. are committed to in the coinbase, they have to be stored
somehow in fullnodes the entire UUTXO set, or the actual utreexo
structure, or the actual BIP158 filter, etc. at each block.
Otherwise it would be pointless to store those commitments since it
would not be possible to somehow acquire the data being committed to
after-the-fact.
This is probably still better than BIP37 but we should still be aware
the additional load on fullnodes.
Regards,
ZmnSCPxj
On Sat, Apr 20, 2019 at 6:45 AM ZmnSCPxj <ZmnSCPxj at protonmail.com> wrote:
>
> Good morning Ruben,
>
> > Hi ZmnSCPxj,
> >
> > > There is no safe way to use UTXO sets without identifying who is telling you those sets are valid, or making it expensive to lie
> > > The first option requires trust and is weaker than SPV, the second requires committing to a proof-of-work
> >
> > Olaoluwa Osuntokun's BIP157 manages to function without a commitment:
> > "If the client receives conflicting filter headers from different
> > peers for any block and filter type, it SHOULD interrogate them to
> > determine which is faulty."
> >
> > I am wondering if the same logic can be applied to UTXO sets or the
> > fraud proofs I just described.
>
> UTXO sets can only be validated by actually running the entire blockchain, i.e. fullnoding.
>
> What BIP157 does is summarize data that is within a block, thus validating them can be done simply by downloading the block in question.
>
> UTXO sets summarize data in the entire blockchain, hence proper validation requires downloading the entire blockchain.
> Thus it cannot be a comparison point.
>
>
> > > This makes no sense
> > > or you trust that every peer you have is not omitting the proof.
> >
> > It's the latter, you trust every peer you have is not omitting the
> > proof. It requires one honest peer. The reason this is acceptable is
> > because you're already making that assumption. If none of your peers
> > are honest, you have no guarantee of hearing about the chain with the
> > most PoW.
>
> But peers can be set up to allow you to hear of all chains while denying you proof of the invalidity of some UTXO.
> This is precisely the "data unavailability claim" that shot down the previous fraud proofs (i.e. absence of proof is not proof of absence, and proof of UTXO validity was defined by proof of absence of any intervening spend of the UTXO).
>
> Perhaps in combination with BIP157/158 it may be possible, if the filters contain UTXO spends and a BIP158 filter was committed to on-chain.
> Then a proof of absence could be done by revealing all the BIP158 filters from the UTXO creation to the block being validated, as well as the blocks whose BIP158 filters matched the UTXO and revealing that no, they actually do not spend the UTXO.
>
> --
>
> Tangentially, we cannot just magically commit to anything on the blockchain.
> Header blocks commit to block data and commit to some other header block.
> All those header blocks and the block data need to be stored and transmitted over the network somehow, even though they are "only" being committed to.
> Thus, if you are adding new information to be committed, that may increase the resource usage of fullnodes.
>
> So if UTXO set commitments, or utreexo commitments, or BIP158 filter digests, etc. are committed to in the coinbase, they have to be stored somehow in fullnodes the entire UUTXO set, or the actual utreexo structure, or the actual BIP158 filter, etc. at each block.
> Otherwise it would be pointless to store those commitments since it would not be possible to somehow acquire the data being committed to after-the-fact.
>
> This is probably still better than BIP37 but we should still be aware the additional load on fullnodes.
>
> Regards,
> ZmnSCPxj
📝 Original message:Hi ZmnSCPxj,
Allow me to reply to your post in mixed order (fraud proofs first):
>But peers can be set up to allow you to hear of all chains while denying you proof of the invalidity of some UTXO.
I don't believe this is fundamentally different. In either scenario
you end up on the wrong chain if all your peers are lying to you. One
happens by omission of a fraud proof, while the other happens by
omission of a valid longest chain.
>This is precisely the "data unavailability claim" that shot down the previous fraud proofs
The "data unavailability" issue I was referring to, and which I
believe is the reason why fraud proofs were abandoned, is the
following:
- Alice downloads a block with her full node, but the block is
incomplete (e.g. a transaction is missing).
- Alice reports this to Bob's SPV fraud proof client, who verifies
this by requesting the transaction from the network.
- If Bob can't download it, he rejects the block.
- If Bob can download it, either Alice was malicious, or a miner was
temporarily withholding the data.
- Since Bob can't be certain Alice was being malicious, Bob can't ban
her, which results in a DoS vector where SPV fraud proof clients can
be forced to download all blocks.
We circumvent the data unavailability problem here completely, since
we are only questioning the validity of blocks which are involved in a
fork (expensive and/or rare), and we are simply always downloading
them in full.
If my arguments above hold up, we can use fraud proof commitments as
described in segwit BIP141 [0] instead of UTXO set commitments, which
seems like the more elegant way to achieve the desired outcome.
>Perhaps in combination with BIP157/158 it may be possible, if the filters contain UTXO spends and a BIP158 filter was committed to on-chain. Then a proof of absence could be done by revealing all the BIP158 filters from the UTXO creation to the block being validated, as well as the blocks whose BIP158 filters matched the UTXO and revealing that no, they actually do not spend the UTXO.
Yes, I mentioned something similar to Laolu, but it does seem
computationally expensive to run every input in a block through the
filter of every past block. The fact that BIP157/158 can function
without commitments is also why I suspected we may not necessarily
need UTXO set commitments.
>UTXO sets can only be validated by actually running the entire blockchain, i.e. fullnoding.
It seems to me you can validate uncommitted UTXO sets by comparing
them. Download and compare UTXO set hashes from multiple peers. If
they disagree on a certain block, download that block and the relevant
merkle path(s) from the previous block's UTXO set, and then verify who
is right. Ban the peer who lied. Note that unlike fraud proofs, it is
not possible to lie by omission, but it does assume one of your peers
is honest. Of course this does nothing to dispute your earlier point
that this may not be all that efficient (e.g. full nodes keeping
merkle paths of all prior states).
>What BIP157 does is summarize data that is within a block, thus validating them can be done simply by downloading the block in question.
>UTXO sets summarize data in the entire blockchain, hence proper validation requires downloading the entire blockchain.
Thus it cannot be a comparison point.
It's still possible to lie by omission. Let's say a miner spends some
coins in block N, and spends the exact same coins again in block N+1,
making block N+1 invalid. If the filter for block N is maliciously
constructed, you won't notice the spend in block N, causing you to
think block N+1 is valid. In short, you're still relying on one of
your peers to give you a correct filter. If all your peers lie, you
can always be deceived.
>Tangentially, we cannot just magically commit to anything on the blockchain. [...] if you are adding new information to be committed, that may increase the resource usage of fullnodes. [...] This is probably still better than BIP37 but we should still be aware the additional load on fullnodes.
I agree with all this.
To summarize, this is my current understanding of our options for
enabling light clients to verify a single block in isolation:
1. UTXO set commitments (complex, more resource usage to full nodes)
2. BIP157/158 commitments (expensive for clients to check all filters
to get exclusion proofs)
3. BIP141 fraud proof commitments (assumes fraud proofs will be passed
on to the SPV client)
The debate is still open on whether the options above can be done
without actually committing them into blocks via a soft fork. My
current hunch is "yes" for 1 and 2, and "no" for 3, which would be
unfortunate, because 3 currently seems to me like the more elegant
solution.
-- Ruben Somsen
[0] https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#Compact_fraud_proof_for_SPV_nodes
>UTXO sets can only be validated by actually running the entire blockchain, i.e. fullnoding.
>What BIP157 does is summarize data that is within a block, thus validating them can be done simply by downloading the block in question.
UTXO sets summarize data in the entire blockchain, hence proper
validation requires downloading the entire blockchain.
Thus it cannot be a comparison point.
> > This makes no sense
> > or you trust that every peer you have is not omitting the proof.
>
> It's the latter, you trust every peer you have is not omitting the
> proof. It requires one honest peer. The reason this is acceptable is
> because you're already making that assumption. If none of your peers
> are honest, you have no guarantee of hearing about the chain with the
> most PoW.
But peers can be set up to allow you to hear of all chains while
denying you proof of the invalidity of some UTXO.
This is precisely the "data unavailability claim" that shot down the
previous fraud proofs (i.e. absence of proof is not proof of absence,
and proof of UTXO validity was defined by proof of absence of any
intervening spend of the UTXO).
Perhaps in combination with BIP157/158 it may be possible, if the
filters contain UTXO spends and a BIP158 filter was committed to
on-chain.
Then a proof of absence could be done by revealing all the BIP158
filters from the UTXO creation to the block being validated, as well
as the blocks whose BIP158 filters matched the UTXO and revealing that
no, they actually do not spend the UTXO.
--
Tangentially, we cannot just magically commit to anything on the blockchain.
Header blocks commit to block data and commit to some other header block.
All those header blocks and the block data need to be stored and
transmitted over the network somehow, even though they are "only"
being committed to.
Thus, if you are adding new information to be committed, that may
increase the resource usage of fullnodes.
So if UTXO set commitments, or utreexo commitments, or BIP158 filter
digests, etc. are committed to in the coinbase, they have to be stored
somehow in fullnodes the entire UUTXO set, or the actual utreexo
structure, or the actual BIP158 filter, etc. at each block.
Otherwise it would be pointless to store those commitments since it
would not be possible to somehow acquire the data being committed to
after-the-fact.
This is probably still better than BIP37 but we should still be aware
the additional load on fullnodes.
Regards,
ZmnSCPxj
On Sat, Apr 20, 2019 at 6:45 AM ZmnSCPxj <ZmnSCPxj at protonmail.com> wrote:
>
> Good morning Ruben,
>
> > Hi ZmnSCPxj,
> >
> > > There is no safe way to use UTXO sets without identifying who is telling you those sets are valid, or making it expensive to lie
> > > The first option requires trust and is weaker than SPV, the second requires committing to a proof-of-work
> >
> > Olaoluwa Osuntokun's BIP157 manages to function without a commitment:
> > "If the client receives conflicting filter headers from different
> > peers for any block and filter type, it SHOULD interrogate them to
> > determine which is faulty."
> >
> > I am wondering if the same logic can be applied to UTXO sets or the
> > fraud proofs I just described.
>
> UTXO sets can only be validated by actually running the entire blockchain, i.e. fullnoding.
>
> What BIP157 does is summarize data that is within a block, thus validating them can be done simply by downloading the block in question.
>
> UTXO sets summarize data in the entire blockchain, hence proper validation requires downloading the entire blockchain.
> Thus it cannot be a comparison point.
>
>
> > > This makes no sense
> > > or you trust that every peer you have is not omitting the proof.
> >
> > It's the latter, you trust every peer you have is not omitting the
> > proof. It requires one honest peer. The reason this is acceptable is
> > because you're already making that assumption. If none of your peers
> > are honest, you have no guarantee of hearing about the chain with the
> > most PoW.
>
> But peers can be set up to allow you to hear of all chains while denying you proof of the invalidity of some UTXO.
> This is precisely the "data unavailability claim" that shot down the previous fraud proofs (i.e. absence of proof is not proof of absence, and proof of UTXO validity was defined by proof of absence of any intervening spend of the UTXO).
>
> Perhaps in combination with BIP157/158 it may be possible, if the filters contain UTXO spends and a BIP158 filter was committed to on-chain.
> Then a proof of absence could be done by revealing all the BIP158 filters from the UTXO creation to the block being validated, as well as the blocks whose BIP158 filters matched the UTXO and revealing that no, they actually do not spend the UTXO.
>
> --
>
> Tangentially, we cannot just magically commit to anything on the blockchain.
> Header blocks commit to block data and commit to some other header block.
> All those header blocks and the block data need to be stored and transmitted over the network somehow, even though they are "only" being committed to.
> Thus, if you are adding new information to be committed, that may increase the resource usage of fullnodes.
>
> So if UTXO set commitments, or utreexo commitments, or BIP158 filter digests, etc. are committed to in the coinbase, they have to be stored somehow in fullnodes the entire UUTXO set, or the actual utreexo structure, or the actual BIP158 filter, etc. at each block.
> Otherwise it would be pointless to store those commitments since it would not be possible to somehow acquire the data being committed to after-the-fact.
>
> This is probably still better than BIP37 but we should still be aware the additional load on fullnodes.
>
> Regards,
> ZmnSCPxj