mikedilger on Nostr: GOSSIP USERS: I have made available an experimental `lmdb` branch of gossip on ...
GOSSIP USERS: I have made available an experimental `lmdb` branch of gossip on github. You don't have to use this, you can stay on a release, on master, or on unstable. But if you want to give it a go, here are a few things to know about it:
1) In many ways it is currently SLOWER than SQLite because I'm not using indexes well yet. This is especially true for the inbox. Startup seems quite a bit faster.
2) It will not modify your SQLite data. So trying this branch won't break your current data and you can go back to a stable branch and be right back where you left off.
3) The database format may still change, without migrations, so you might need to (in the future) wipe your lmdb/ directory and reimport from SQLite, losing any changes and new events that went into lmdb. This will only be for a little while longer. PLEASE don't delete your SQLite data if you want to preserve your current setup and event history.
4) When you first try it, it will import the data. This may take a couple of minutes.
5) You'll have a new directory in your gossip config area called 'lmdb/'. One of those files (data.mdb) appears to be 128 GB large. Don't worry, it is actually much smaller. Run a 'du' command against it to see how big it actually is.
6) You can prune your database. It currently goes back 180 days (I think it used to be 7 days or something really short). BUT you will notice that the lmdb/ directory disk usage won't change. In order to actually shrink the lmdb/ directory you will need to use the command line tool 'mdb_copy -c' while gossip is not running. RTFM.
7) One downside is you can't fire up an sqlite command and peek at or mess with the data. I may write some tools to help in this regard.
8) If anything that works on unstable or master doesn't work right on the LMDB branch (not speed, but function), then please open an issue on github so I can know about it and work to resolve it.
We got rid of about 4400 lines of code. We added back about 3500 new lines of code, but about 1100 of those new lines are the code required to import from legacy sqlite, baggage that we will have to carry with us. So ignoring the import code, the new LMDB code is about half the size of the old SQLite code.
Among that, we got rid of a lot of thread spawning, async polling, and type conversions.
We also got rid of a lot of caching code that tried to keep an in-memory cache in sync (somewhat) with the SQLite data so we didn't have to go back to SQLite in situations where we couldn't afford to wait for it. Now we just go to LMDB and presume it will be fast enough. A lot of that caching code was badly designed and buggy, causing (for example) some older thread replies to not show up even though SQLite held the event. No caches means no loading and nothing getting out of sync.
1) In many ways it is currently SLOWER than SQLite because I'm not using indexes well yet. This is especially true for the inbox. Startup seems quite a bit faster.
2) It will not modify your SQLite data. So trying this branch won't break your current data and you can go back to a stable branch and be right back where you left off.
3) The database format may still change, without migrations, so you might need to (in the future) wipe your lmdb/ directory and reimport from SQLite, losing any changes and new events that went into lmdb. This will only be for a little while longer. PLEASE don't delete your SQLite data if you want to preserve your current setup and event history.
4) When you first try it, it will import the data. This may take a couple of minutes.
5) You'll have a new directory in your gossip config area called 'lmdb/'. One of those files (data.mdb) appears to be 128 GB large. Don't worry, it is actually much smaller. Run a 'du' command against it to see how big it actually is.
6) You can prune your database. It currently goes back 180 days (I think it used to be 7 days or something really short). BUT you will notice that the lmdb/ directory disk usage won't change. In order to actually shrink the lmdb/ directory you will need to use the command line tool 'mdb_copy -c' while gossip is not running. RTFM.
7) One downside is you can't fire up an sqlite command and peek at or mess with the data. I may write some tools to help in this regard.
8) If anything that works on unstable or master doesn't work right on the LMDB branch (not speed, but function), then please open an issue on github so I can know about it and work to resolve it.
We got rid of about 4400 lines of code. We added back about 3500 new lines of code, but about 1100 of those new lines are the code required to import from legacy sqlite, baggage that we will have to carry with us. So ignoring the import code, the new LMDB code is about half the size of the old SQLite code.
Among that, we got rid of a lot of thread spawning, async polling, and type conversions.
We also got rid of a lot of caching code that tried to keep an in-memory cache in sync (somewhat) with the SQLite data so we didn't have to go back to SQLite in situations where we couldn't afford to wait for it. Now we just go to LMDB and presume it will be fast enough. A lot of that caching code was badly designed and buggy, causing (for example) some older thread replies to not show up even though SQLite held the event. No caches means no loading and nothing getting out of sync.