Multipool Admin [ARCHIVE] on Nostr: 📅 Original date posted:2015-10-19 📝 Original message:My nodes are continuously ...
📅 Original date posted:2015-10-19
📝 Original message:My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.
The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:
> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj at gmail.com>
> wrote:
>
> This is *most likely* the mempool, but is just not reported correctly.
>
>
> I did some testing with PR #6410's better mempool reporting. The improved
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
> The excess memory usage that I'm seeing is on the order of 100x higher than
> the mempool "bytes": value. As such, I think it's unlikely that this is the
> mempool, or at least not normal/correct mempool behavior.
>
> Another user (admin at multipool.us) reported 35 GB of RSS usage. I'm
> guessing his bitcoind has been running longer than any of mine. His server
> definitely has more RAM. I don't know which email list he is subscribed to
> (probably XT), so I'm sharing it with both lists to make sure you're all
> aware of how big an issue this can be.
>
> In the meantime you can mitigate the mempool growth by setting
> `-mintxfee`, see
>
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
>
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
> enough to exclude essentially all of the of the 14700-14800 byte flood
> transactions. My nodes' mempools only contain about one or two blocks'
> worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
> Core bitcoind process (commit d78a880) overnight with no mining connected
> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
> getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable,
> and record the results more carefully. Expect a follow-up from me in a day
> or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880== definitely lost: 0 bytes in 0 blocks
> ==6880== indirectly lost: 0 bytes in 0 blocks
> ==6880== possibly lost: 288 bytes in 1 blocks
> ==6880== still reachable: 10,552 bytes in 39 blocks
> ==6880== suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778== definitely lost: 0 bytes in 0 blocks
> ==6778== indirectly lost: 0 bytes in 0 blocks
> ==6778== possibly lost: 320 bytes in 1 blocks
> ==6778== still reachable: 10,080 bytes in 32 blocks
> ==6778== suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the
> multi-GB memory usage after running for a few days, but I could be missing
> it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll
> have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar
> fallout. While it's probably too late for this thread, I'll make sure to
> post in a manner that keeps the threads clearly separate in the future
> (e.g. different subject lines).
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151019/e7eae196/attachment.html>;
📝 Original message:My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.
The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:
> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj at gmail.com>
> wrote:
>
> This is *most likely* the mempool, but is just not reported correctly.
>
>
> I did some testing with PR #6410's better mempool reporting. The improved
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
> The excess memory usage that I'm seeing is on the order of 100x higher than
> the mempool "bytes": value. As such, I think it's unlikely that this is the
> mempool, or at least not normal/correct mempool behavior.
>
> Another user (admin at multipool.us) reported 35 GB of RSS usage. I'm
> guessing his bitcoind has been running longer than any of mine. His server
> definitely has more RAM. I don't know which email list he is subscribed to
> (probably XT), so I'm sharing it with both lists to make sure you're all
> aware of how big an issue this can be.
>
> In the meantime you can mitigate the mempool growth by setting
> `-mintxfee`, see
>
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
>
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
> enough to exclude essentially all of the of the 14700-14800 byte flood
> transactions. My nodes' mempools only contain about one or two blocks'
> worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
> Core bitcoind process (commit d78a880) overnight with no mining connected
> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
> getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable,
> and record the results more carefully. Expect a follow-up from me in a day
> or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880== definitely lost: 0 bytes in 0 blocks
> ==6880== indirectly lost: 0 bytes in 0 blocks
> ==6880== possibly lost: 288 bytes in 1 blocks
> ==6880== still reachable: 10,552 bytes in 39 blocks
> ==6880== suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778== definitely lost: 0 bytes in 0 blocks
> ==6778== indirectly lost: 0 bytes in 0 blocks
> ==6778== possibly lost: 320 bytes in 1 blocks
> ==6778== still reachable: 10,080 bytes in 32 blocks
> ==6778== suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the
> multi-GB memory usage after running for a few days, but I could be missing
> it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll
> have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar
> fallout. While it's probably too late for this thread, I'll make sure to
> post in a manner that keeps the threads clearly separate in the future
> (e.g. different subject lines).
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151019/e7eae196/attachment.html>;