satonomics on Nostr: Fjall was nice but too slow, tried canopydb but it's a bit too raw and harder to use; ...
Fjall was nice but too slow, tried canopydb but it's a bit too raw and harder to use; so I'm gonna stick with Sanakirja which is the devil I know and can manage
I also created a new crate `storable_vec` which is, as the name suggests, a list that can be saved on disk
It's push only for now (as it's all that I needed) and very fast, small and simple. It only stores the raw list for now, there is not even a header yet, so disk usage is incredibly efficient.
As many store use some kind of index as keys (height/txindex/txoutindex) I was able to migrate many of them to storable_vec (only 6 out of ~30 will use sanakirja) and the speed gain AND storage reduction is very noticeable
There is a lot more work left before the v0.6 is done but it will be the biggest update yet
#Bitcoin is hope 🕊️
I also created a new crate `storable_vec` which is, as the name suggests, a list that can be saved on disk
It's push only for now (as it's all that I needed) and very fast, small and simple. It only stores the raw list for now, there is not even a header yet, so disk usage is incredibly efficient.
As many store use some kind of index as keys (height/txindex/txoutindex) I was able to migrate many of them to storable_vec (only 6 out of ~30 will use sanakirja) and the speed gain AND storage reduction is very noticeable
There is a lot more work left before the v0.6 is done but it will be the biggest update yet
#Bitcoin is hope 🕊️
quoting nevent1q…tjcaIt's been a while since I posted an update
So for anyone wondering I'm in the middle of R&D and trying out two major tasks:
- Fjall the key value store instead of Sanakirja
- Extracting and improving the parsing part (think creating a database from the blocks) from the "parser" crate (or folder, in the v0.5 codebase) which would end up being only a "computer" now (and thus a more organized codebase with 4 major parts: iterator, parser, computer and server)
Both have the same goal which is massively improve the self-hosting experience
The latter is going well I think and is looking like a good base for a mempool.space like database/explorer/whatever (which was kind of already needed and there but in the right shape)
The reason for doing is because up to the v0.5 the databases are stateful, by that I mean that when you compute a block you can't go back as the databases have the state at a current height, it's to save disk space since sanakirja is very demanding on this side but had the disadvantage is the need to recompute the databases again each time we need to start from the beginning (because of a new dataset for example) which is a lot (like a LOT) of disk writes (and time !)
So the goal is to have an append only database instead which is pretty much one and done like the chain itself (and no I can't just use the chain, it's too slow)
The former on the other hand, isn't going as well as I hoped, reads and writes are okay, the space needed is absolutely great the database is tiny but the RAM usage is quite high (12-13 GB at block 400 000 which is not great) with fjall. I'll do more research and see what I can do with fjall but also trying out sanakirja with the new code and maybe even manually with mmaps by hand if none of them satisfy the requirements.
Again the reason for this is to limit disk usage as it's currently very very high and this takes priority over anything else
Overall I'm happy with my progress, you can check the current state of the extracted parser (named "bitbase") here: https://github.com/kibo-money/kibo/tree/v0.6-fjall/src/crates/bitbase
Please note that I might change to branch's name so the link might break at some point
nevent1q…hfvz