Vitor Pamplona on Nostr: Devs, has anybody built a memory-efficient Event Hint indexer? Basically, I want a ...
Devs, has anybody built a memory-efficient Event Hint indexer?
Basically, I want a separate in memory cache where I store all event IDs the app has seen and all relays they can be found on, merging receipts "seen on" with the hints in all tags.
The idea is to decouple event loading from the information in a particular event. Instead of using just the hint provided by the event, the app should check the hints for all other events received.
For instance, we could do a Bloom filter for each relay to avoid storing event ids in memory (and in disk). If I need the relays for a given event ID, I just check the presence in all Bloom filters from the lowest to the highest ping.
We could use the same for pubkey hints and address hints as well.
My only concern is the computational expense of doing multiple hashing rounds for each add and contains operation for every hint in each event the app receives. 🤔
Basically, I want a separate in memory cache where I store all event IDs the app has seen and all relays they can be found on, merging receipts "seen on" with the hints in all tags.
The idea is to decouple event loading from the information in a particular event. Instead of using just the hint provided by the event, the app should check the hints for all other events received.
For instance, we could do a Bloom filter for each relay to avoid storing event ids in memory (and in disk). If I need the relays for a given event ID, I just check the presence in all Bloom filters from the lowest to the highest ping.
We could use the same for pubkey hints and address hints as well.
My only concern is the computational expense of doing multiple hashing rounds for each add and contains operation for every hint in each event the app receives. 🤔