mleku on Nostr: success! realy now has a pubkey index substitution scheme that compresses the amount ...
success!
realy now has a pubkey index substitution scheme that compresses the amount of data required to store pubkeys in the event pubkey field and p tags
this is comparing the test #realy on my pc running the new code with the pubkey index, vs the production wss://mleku.realy.lol relay currently:
root@mleku:~# du -h -d1 /home/mleku/.config/realy
151M /home/mleku/.config/realy
mleku@ziox:~$ du -h -d1 ~/.config/realy
106M /home/mleku/.config/realy
i exported the data from mleku.realy.lol and then imported it into test.realy.lol
realy includes a spider that searches for the "directory" events of all of the users included in the whitelist via my configured "owners" list which is just me, it is some 11k npubs who are the complete set of everyone who everyone i follow, follows, i follow around 100 npubs, and they follow however many whatever
the spider actively rescans once an hour a couple of randomly selected relays chosen out of the list of relays that are discovered in the relay lists it has stored, and it has been doing that like clockwork every hour for about 3 weeks now, i think, somewhere around that, so, that database includes a LOT of follow lists, at least 11000 of them for sure
those events can typically range from around 30kb to 500kb, as each npub costs 64 bytes of storage
i have just cut that down to around 6-12 bytes per npub reference, and it just slashed the relay's database storage cost by around 30%
it was worth it
i think it has probably slightly reduced retrieval latency, and maybe increased memory use a little by adding the pubkey index, but with that extra 30% space that i can load up with follow lists, this will be the premiere WoT data relay
realy now has a pubkey index substitution scheme that compresses the amount of data required to store pubkeys in the event pubkey field and p tags
this is comparing the test #realy on my pc running the new code with the pubkey index, vs the production wss://mleku.realy.lol relay currently:
root@mleku:~# du -h -d1 /home/mleku/.config/realy
151M /home/mleku/.config/realy
mleku@ziox:~$ du -h -d1 ~/.config/realy
106M /home/mleku/.config/realy
i exported the data from mleku.realy.lol and then imported it into test.realy.lol
realy includes a spider that searches for the "directory" events of all of the users included in the whitelist via my configured "owners" list which is just me, it is some 11k npubs who are the complete set of everyone who everyone i follow, follows, i follow around 100 npubs, and they follow however many whatever
the spider actively rescans once an hour a couple of randomly selected relays chosen out of the list of relays that are discovered in the relay lists it has stored, and it has been doing that like clockwork every hour for about 3 weeks now, i think, somewhere around that, so, that database includes a LOT of follow lists, at least 11000 of them for sure
those events can typically range from around 30kb to 500kb, as each npub costs 64 bytes of storage
i have just cut that down to around 6-12 bytes per npub reference, and it just slashed the relay's database storage cost by around 30%
it was worth it
i think it has probably slightly reduced retrieval latency, and maybe increased memory use a little by adding the pubkey index, but with that extra 30% space that i can load up with follow lists, this will be the premiere WoT data relay