A boltdb backend for fiatjaf's relayer
@fiatjaf (nprofile…pawm) ’s relayer is a go library for making custom relays. I’ve been investigating its capabilities in the past few weeks and I like how customizable it is. I’m not a nostr expert, but I think that the ability to make relays tuned to specific purposes is important for nostr to succeed (e.g. I want a relay to allow people to run competitive lnproxy relays without needing to expose their clearnet IP addresses).
In testing, I’ve found the performance of the different storage backends available for relayer lacking, but I’ve been working on a speedier solution based on boltdb! With that in mind, last week I asked nostr:
nevent1q…ykj8
Unfortunately none of the suggestions were exactly what I needed but I was able to set up some benchmarks in go and the results are looking great:
BenchmarkSQLite3QueryEvents/IDs-8 30 42339015 ns/op
BenchmarkSQLite3QueryEvents/Authors-8 36 37080938 ns/op
BenchmarkSQLite3QueryEvents/Tags-8 31 41998150 ns/op
BenchmarkSQLite3QueryEvents/Kinds-8 18 69550451 ns/op
BenchmarkSQLite3QueryEvents/Authors,Kinds-8 48 41373654 ns/op
BenchmarkSQLite3QueryEvents/Since-8 9 129129112 ns/op
BenchmarkSQLite3QueryEvents/Authors,Since-8 40 47585709 ns/op
BenchmarkSQLite3QueryEvents/Authors,Kinds,Since-8 18 64724085 ns/op
BenchmarkSQLite3QueryEvents/Tags,Kinds-8 36 49993809 ns/op
BenchmarkSQLite3QueryEvents/Tags,Authors-8 34 31319155 ns/op
BenchmarkBoltQueryEvents/IDs-8 1591 756493 ns/op
BenchmarkBoltQueryEvents/Authors-8 255 4652186 ns/op
BenchmarkBoltQueryEvents/Tags-8 264 4566247 ns/op
BenchmarkBoltQueryEvents/Kinds-8 250 4772767 ns/op
BenchmarkBoltQueryEvents/Authors,Kinds-8 170 6893723 ns/op
BenchmarkBoltQueryEvents/Since-8 277 4473123 ns/op
BenchmarkBoltQueryEvents/Authors,Since-8 243 4738151 ns/op
BenchmarkBoltQueryEvents/Authors,Kinds,Since-8 163 7289618 ns/op
BenchmarkBoltQueryEvents/Tags,Kinds-8 190 6502720 ns/op
BenchmarkBoltQueryEvents/Tags,Authors-8 705 1755724 ns/op
I’ll try to release the code sometime this weekend, but I wanted to share that my boltdb solution is much faster than sqlite for simple queries and even beats sql on fairly complex queries.
Published at
2023-08-11 02:02:00Event JSON
{
"id": "e6be5cfc529b1eea5dd8c3d050be46b1c1d83695b21692f303188df3f0b67139",
"pubkey": "e25a8b2051022a08f97d267d4b99ddfc500a0bfe149a5f671e46f72e9ea36ec9",
"created_at": 1691719320,
"kind": 30023,
"tags": [
[
"d",
"1691718362241"
],
[
"title",
"A boltdb backend for fiatjaf's relayer"
],
[
"summary",
"A boltdb backend for github.com/fiatjaf/relayer speeds up queries by \u003e 5X"
],
[
"published_at",
"1691719320"
],
[
"p",
"3bf0c63fcb93463407af97a5e5ee64fa883d107ef9e558472c4eb9aaaefa459d"
],
[
"e",
"e189d740d51b55874b62e5f0ad5c8803fa33dd9df248497b2e602f4826ad084f"
]
],
"content": "@nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gpp4mhxue69uhkummn9ekx7mqpr4mhxue69uhkummnw3ez6ur4vgh8wetvd3hhyer9wghxuet5qyt8wumn8ghj7un9d3shjtnwdaehgu3wvfskueqpz9mhxue69uhkummnw3ezumr49e4k2grpawm 's [relayer](https://github.com/fiatjaf/relayer) is a go library for making custom relays. I've been investigating its capabilities in the past few weeks and I like how customizable it is. I'm not a nostr expert, but I think that the ability to make relays tuned to specific purposes is important for nostr to succeed (e.g. I want a relay to allow people to run competitive lnproxy relays without needing to expose their clearnet IP addresses).\n\nIn testing, I've found the performance of the different storage backends available for relayer lacking, but I've been working on a speedier solution based on boltdb! With that in mind, last week I asked nostr:\n\nnostr:nevent1qqswrzwhgr23k4v8fd3wtu9dtjyq873nmkwlyjzf0vhxqt6gy6kssncprpmhxue69uhkummnw3ezuendwsh8w6t69e3xj730qy2hwumn8ghj7un9d3shjtnyv9kh2uewd9hj7qg5waehxw309askummw9e3k7mtsw46x2u30qy2hwumn8ghj7un9d3shjtnwdaehgu3wvfnj77dykj8\n\nUnfortunately none of the suggestions were exactly what I needed but I was able to set up some benchmarks in go and the results are looking great:\n```\nBenchmarkSQLite3QueryEvents/IDs-8 \t 30\t 42339015 ns/op\nBenchmarkSQLite3QueryEvents/Authors-8 \t 36\t 37080938 ns/op\nBenchmarkSQLite3QueryEvents/Tags-8 \t 31\t 41998150 ns/op\nBenchmarkSQLite3QueryEvents/Kinds-8 \t 18\t 69550451 ns/op\nBenchmarkSQLite3QueryEvents/Authors,Kinds-8 \t 48\t 41373654 ns/op\nBenchmarkSQLite3QueryEvents/Since-8 \t 9\t 129129112 ns/op\nBenchmarkSQLite3QueryEvents/Authors,Since-8 \t 40\t 47585709 ns/op\nBenchmarkSQLite3QueryEvents/Authors,Kinds,Since-8 \t 18\t 64724085 ns/op\nBenchmarkSQLite3QueryEvents/Tags,Kinds-8 \t 36\t 49993809 ns/op\nBenchmarkSQLite3QueryEvents/Tags,Authors-8 \t 34\t 31319155 ns/op\nBenchmarkBoltQueryEvents/IDs-8 \t 1591\t 756493 ns/op\nBenchmarkBoltQueryEvents/Authors-8 \t 255\t 4652186 ns/op\nBenchmarkBoltQueryEvents/Tags-8 \t 264\t 4566247 ns/op\nBenchmarkBoltQueryEvents/Kinds-8 \t 250\t 4772767 ns/op\nBenchmarkBoltQueryEvents/Authors,Kinds-8 \t 170\t 6893723 ns/op\nBenchmarkBoltQueryEvents/Since-8 \t 277\t 4473123 ns/op\nBenchmarkBoltQueryEvents/Authors,Since-8 \t 243\t 4738151 ns/op\nBenchmarkBoltQueryEvents/Authors,Kinds,Since-8 \t 163\t 7289618 ns/op\nBenchmarkBoltQueryEvents/Tags,Kinds-8 \t 190\t 6502720 ns/op\nBenchmarkBoltQueryEvents/Tags,Authors-8 \t 705\t 1755724 ns/op\n```\n\nI'll try to release the code sometime this weekend, but I wanted to share that my boltdb solution is much faster than sqlite for simple queries and even beats sql on fairly complex queries.\n\n",
"sig": "4f6c004978e2424503a587f4b3839082ebe8baf3af3210695da89c456cdbfe3d26adbcfa7973149d135d731f255e358f584e5de27d3f4652a1dcdc5ee299247a"
}