david on Nostr: Neo4j is definitely resource hungry. But it is designed to handle very large ...
Neo4j is definitely resource hungry.
But it is designed to handle very large datasets, with real-world examples in the hundreds of billions of nodes. My understanding is that there is no theoretical upper limit to the number of nodes and edges other than those imposed by hardware.
I envision two applications for neo4j and nostr: 1) central services like nostr.band and 2) personalized WoT relays, with personal relays using intelligent Knowledge Graph based WoT to curate the local dataset so it doesn’t get too unwieldy and expensive.
So what would it take to put the entire nostr.band dataset onto a neo4j graph? Perhaps Jay [MOVED HERE] (npub10mt…27hf) you can help us do a back of the envelope calculation in terms of RAM and other requirements to support this database. Suppose we were to host it on AWS. How much would it cost?
I imagine we would start with 3 node types: NostrUser, NostrEvent, and NostrRelay.
Then add relationships:
user to user: FOLLOW, MUTE, REPORT, etc
user to event: AUTHOR
Various event to event relationships: IS_A_REPLY_TO, etc.
Various other relationships and node types as we turn it into a bona fide Knowledge Graph.
How big is your current dataset? Do you filter anything out or keep every event that you come across?
But it is designed to handle very large datasets, with real-world examples in the hundreds of billions of nodes. My understanding is that there is no theoretical upper limit to the number of nodes and edges other than those imposed by hardware.
I envision two applications for neo4j and nostr: 1) central services like nostr.band and 2) personalized WoT relays, with personal relays using intelligent Knowledge Graph based WoT to curate the local dataset so it doesn’t get too unwieldy and expensive.
So what would it take to put the entire nostr.band dataset onto a neo4j graph? Perhaps Jay [MOVED HERE] (npub10mt…27hf) you can help us do a back of the envelope calculation in terms of RAM and other requirements to support this database. Suppose we were to host it on AWS. How much would it cost?
I imagine we would start with 3 node types: NostrUser, NostrEvent, and NostrRelay.
Then add relationships:
user to user: FOLLOW, MUTE, REPORT, etc
user to event: AUTHOR
Various event to event relationships: IS_A_REPLY_TO, etc.
Various other relationships and node types as we turn it into a bona fide Knowledge Graph.
How big is your current dataset? Do you filter anything out or keep every event that you come across?