Pieter Wuille [ARCHIVE] on Nostr: 📅 Original date posted:2023-02-17 🗒️ Summary of this message: A proposal to ...
📅 Original date posted:2023-02-17
🗒️ Summary of this message: A proposal to use short IDs to minimize bandwidth in Bitcoin transactions has been suggested, with short ID 0 reserved for long commands. This would reduce complexity and eliminate the need for variable-length encoding.
📝 Original message:On Friday, February 17th, 2023 at 10:51 AM, Anthony Towns via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> I think it's probably less complex to close some of the doors?
> 2) are short ids available/meaningful to send prior to VERACK being
> completed?
Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for *re*-negotiating.
> Here's another approach:
>
> idea: we use short ids to minimise bandwidth, and don't care about
> bandwidth for long ids
>
> implementation:
> short id 0 is reserved for long commands. when received, we
> decode the first 12 bytes of the payload and treat them
> exactly the same as a v1 p2p message (trailing 0-bytes, etc)
> (if there's not 12 bytes of payload, it's just treated as an
> invalid command and dropped)
>
> short ids 1-255 are available for use as aliases of particular
> long commands
>
> (That's exactly compatible with p2p v1, and also avoids the temptation
> to try to choose short command names rather than descriptive ones -- the
> 0-padding to 12 bytes prevents you from saving any bandwidth that way;
> but that's what we have short ids for anyway)
I like this idea. It avoids the variable-length encoding question and related complexity entirely for things where we admittedly don't care about the bandwidth impact anyway.
It may also have another (rather weak) advantage, in that it may reduce how much information a passive observe may learn about application level features (sendheaders, sendaddrv2, ...) from the packet size sent (which would otherwise depend on command lengths), even when decoys are not in use, if no short commands are included for these messages.
> > - We remove 1 byte allocations for messages that are sent at most once per connection per direction
>
> I think this leaves 32 commands that get short ids initially:
>
> misc: ADDR, ADDRV2, BLOCK, FEEFILTER, GETBLOCKS, GETDATA, GETHEADERS,
> HEADERS, INV, NOTFOUND, PING, PONG, TX
> bip 35/37: FILTERADD, FILTERCLEAR, FILTERLOAD, MEMPOOL, MERKLEBLOCK
> bip 152: BLOCKTXN, CMPCTBLOCK, GETBLOCKTXN
> bip 157: CFCHECKPT, CFHEADERS, CFILTER, GETCFCHCKPT, GETCFHEADERS,
> GETCFILTERS
> bip 330: RECONCILDIFF, REQRECON, REQSKETCHEXT, SENDCMPCT, SKETCH
Sounds right.
> which drops:
>
> VERSION, VERACK, GETADDR, SENDADDRV2, SENDHEADERS, SENDTXRCNCL,
> WTXIDRELAY
Indeed.
> compared to bip 324 currently.
>
> I think the things missing from the current list (and not currently in
> use by bitcoin core) are:
>
> bip 61: REJECT
> bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO
Do you feel REJECT should be included?
> > - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern.
>
> I agree that's fine, though I expect that we'll probably want to do it
> not long after bip 331 is ready for merge (or some other p2p improvement
> comes along)...
I do prefer that as well; it feels like the transport layer shouldn't be aware of the different command names that exist, but this is very much just an implementation issue.
Perhaps a possibility is having the transport layer translate short-command-number-N to the 12-byte command "\x00\x00..." + byte(N), and hand that to the application layer, which could then do the mapping?
Cheers,
--
Pieter
🗒️ Summary of this message: A proposal to use short IDs to minimize bandwidth in Bitcoin transactions has been suggested, with short ID 0 reserved for long commands. This would reduce complexity and eliminate the need for variable-length encoding.
📝 Original message:On Friday, February 17th, 2023 at 10:51 AM, Anthony Towns via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> I think it's probably less complex to close some of the doors?
> 2) are short ids available/meaningful to send prior to VERACK being
> completed?
Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for *re*-negotiating.
> Here's another approach:
>
> idea: we use short ids to minimise bandwidth, and don't care about
> bandwidth for long ids
>
> implementation:
> short id 0 is reserved for long commands. when received, we
> decode the first 12 bytes of the payload and treat them
> exactly the same as a v1 p2p message (trailing 0-bytes, etc)
> (if there's not 12 bytes of payload, it's just treated as an
> invalid command and dropped)
>
> short ids 1-255 are available for use as aliases of particular
> long commands
>
> (That's exactly compatible with p2p v1, and also avoids the temptation
> to try to choose short command names rather than descriptive ones -- the
> 0-padding to 12 bytes prevents you from saving any bandwidth that way;
> but that's what we have short ids for anyway)
I like this idea. It avoids the variable-length encoding question and related complexity entirely for things where we admittedly don't care about the bandwidth impact anyway.
It may also have another (rather weak) advantage, in that it may reduce how much information a passive observe may learn about application level features (sendheaders, sendaddrv2, ...) from the packet size sent (which would otherwise depend on command lengths), even when decoys are not in use, if no short commands are included for these messages.
> > - We remove 1 byte allocations for messages that are sent at most once per connection per direction
>
> I think this leaves 32 commands that get short ids initially:
>
> misc: ADDR, ADDRV2, BLOCK, FEEFILTER, GETBLOCKS, GETDATA, GETHEADERS,
> HEADERS, INV, NOTFOUND, PING, PONG, TX
> bip 35/37: FILTERADD, FILTERCLEAR, FILTERLOAD, MEMPOOL, MERKLEBLOCK
> bip 152: BLOCKTXN, CMPCTBLOCK, GETBLOCKTXN
> bip 157: CFCHECKPT, CFHEADERS, CFILTER, GETCFCHCKPT, GETCFHEADERS,
> GETCFILTERS
> bip 330: RECONCILDIFF, REQRECON, REQSKETCHEXT, SENDCMPCT, SKETCH
Sounds right.
> which drops:
>
> VERSION, VERACK, GETADDR, SENDADDRV2, SENDHEADERS, SENDTXRCNCL,
> WTXIDRELAY
Indeed.
> compared to bip 324 currently.
>
> I think the things missing from the current list (and not currently in
> use by bitcoin core) are:
>
> bip 61: REJECT
> bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO
Do you feel REJECT should be included?
> > - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern.
>
> I agree that's fine, though I expect that we'll probably want to do it
> not long after bip 331 is ready for merge (or some other p2p improvement
> comes along)...
I do prefer that as well; it feels like the transport layer shouldn't be aware of the different command names that exist, but this is very much just an implementation issue.
Perhaps a possibility is having the transport layer translate short-command-number-N to the 12-byte command "\x00\x00..." + byte(N), and hand that to the application layer, which could then do the mapping?
Cheers,
--
Pieter