Anthony Accioly on Nostr: Hi fiatjaf, yes, that’s the one. At 2:00 am “compact” somehow became ...
Hi fiatjaf, yes, that’s the one. At 2:00 am “compact” somehow became “compress” in my head. But you’re right, other than making certain “immortal” events deletable on LMDB as per your original intent above, for the end user, it’s essentially a major database-wide deduplication (which is exactly what I was looking for with BadgerDB).
You’re also right that nuking the database and reimporting old notes has the same effect. This is what I’m suggesting for Haven users for now. Unfortunately, Haven can’t import its own backups (yet), but users can always reimport some of their old notes from other relays or temporarily use a second instance of Haven to do this. I was just considering a Haven-specific --compact flag for completeness (e.g., so users don’t lose private notes that aren’t currently reimported) and to save them the trouble of doing this manually.
Either way, awesome work. Many thanks! Haven is absolutely flying with the new Khatru engine. I even tested this with an old LMDB database backup I keep around for testing purposes. Compacting the database cleared out over 2 million duplicsted events from a database containing only around 1k short notes. It’s impressive how clients continuously spam lists, sets, etc. Now I know there was much more to it than just the Amethyst kind 10002 write loop bug.
You’re also right that nuking the database and reimporting old notes has the same effect. This is what I’m suggesting for Haven users for now. Unfortunately, Haven can’t import its own backups (yet), but users can always reimport some of their old notes from other relays or temporarily use a second instance of Haven to do this. I was just considering a Haven-specific --compact flag for completeness (e.g., so users don’t lose private notes that aren’t currently reimported) and to save them the trouble of doing this manually.
Either way, awesome work. Many thanks! Haven is absolutely flying with the new Khatru engine. I even tested this with an old LMDB database backup I keep around for testing purposes. Compacting the database cleared out over 2 million duplicsted events from a database containing only around 1k short notes. It’s impressive how clients continuously spam lists, sets, etc. Now I know there was much more to it than just the Amethyst kind 10002 write loop bug.