satonomics on Nostr: It's been a while since I posted an update So for anyone wondering I'm in the middle ...
It's been a while since I posted an update
So for anyone wondering I'm in the middle of R&D and trying out two major tasks:
- Fjall the key value store instead of Sanakirja
- Extracting and improving the parsing part (think creating a database from the blocks) from the "parser" crate (or folder, in the v0.5 codebase) which would end up being only a "computer" now (and thus a more organized codebase with 4 major parts: iterator, parser, computer and server)
Both have the same goal which is massively improve the self-hosting experience
The latter is going well I think and is looking like a good base for a mempool.space like database/explorer/whatever (which was kind of already needed and there but in the right shape)
The reason for doing is because up to the v0.5 the databases are stateful, by that I mean that when you compute a block you can't go back as the databases have the state at a current height, it's to save disk space since sanakirja is very demanding on this side but had the disadvantage is the need to recompute the databases again each time we need to start from the beginning (because of a new dataset for example) which is a lot (like a LOT) of disk writes (and time !)
So the goal is to have an append only database instead which is pretty much one and done like the chain itself (and no I can't just use the chain, it's too slow)
The former on the other hand, isn't going as well as I hoped, reads and writes are okay, the space needed is absolutely great the database is tiny but the RAM usage is quite high (12-13 GB at block 400 000 which is not great) with fjall. I'll do more research and see what I can do with fjall but also trying out sanakirja with the new code and maybe even manually with mmaps by hand if none of them satisfy the requirements.
Again the reason for this is to limit disk usage as it's currently very very high and this takes priority over anything else
Overall I'm happy with my progress, you can check the current state of the extracted parser (named "bitbase") here: https://github.com/kibo-money/kibo/tree/v0.6-fjall/src/crates/bitbase
Please note that I might change to branch's name so the link might break at some point
So for anyone wondering I'm in the middle of R&D and trying out two major tasks:
- Fjall the key value store instead of Sanakirja
- Extracting and improving the parsing part (think creating a database from the blocks) from the "parser" crate (or folder, in the v0.5 codebase) which would end up being only a "computer" now (and thus a more organized codebase with 4 major parts: iterator, parser, computer and server)
Both have the same goal which is massively improve the self-hosting experience
The latter is going well I think and is looking like a good base for a mempool.space like database/explorer/whatever (which was kind of already needed and there but in the right shape)
The reason for doing is because up to the v0.5 the databases are stateful, by that I mean that when you compute a block you can't go back as the databases have the state at a current height, it's to save disk space since sanakirja is very demanding on this side but had the disadvantage is the need to recompute the databases again each time we need to start from the beginning (because of a new dataset for example) which is a lot (like a LOT) of disk writes (and time !)
So the goal is to have an append only database instead which is pretty much one and done like the chain itself (and no I can't just use the chain, it's too slow)
The former on the other hand, isn't going as well as I hoped, reads and writes are okay, the space needed is absolutely great the database is tiny but the RAM usage is quite high (12-13 GB at block 400 000 which is not great) with fjall. I'll do more research and see what I can do with fjall but also trying out sanakirja with the new code and maybe even manually with mmaps by hand if none of them satisfy the requirements.
Again the reason for this is to limit disk usage as it's currently very very high and this takes priority over anything else
Overall I'm happy with my progress, you can check the current state of the extracted parser (named "bitbase") here: https://github.com/kibo-money/kibo/tree/v0.6-fjall/src/crates/bitbase
Please note that I might change to branch's name so the link might break at some point
quoting nevent1q…hfvzFjall (storage) looks very promising on the disk space side and write speed, not sure about reads as I’m still benchmarking it
If reads are at worst slightly slower, I’ll start replacing Sanakirja by this thing
It could be a huge improvement for all self hosters
Links:
https://github.com/fjall-rs/fjall
https://docs.rs/sanakirja