Alex on Nostr: I want to tell you why #Ditto was slow and how we fixed it. There were 2 issues: ...
I want to tell you why #Ditto was slow and how we fixed it.
There were 2 issues: Postgres performance and nip05 lookups.
Postgres performance was caused by profile lookups, eg:
{ kinds: [0], authors: [a, b, c, d, ...] }
The issue was ORDER BY clauses on those queries. This caused Postgres to do a "sequence scan" instead of using the correct index:
"nostr_events_replaceable_idx" UNIQUE, btree (kind, pubkey) WHERE kind >= 10000 AND kind < 20000 OR (kind = ANY (ARRAY[0, 3]))
I actually fixed this already last year, and then reverted it in the last few months due to a bad decision. 🤦 See: https://gitlab.com/soapbox-pub/nostrify/-/commit/6a51c3f8a73c8a9f852a46d57119afdd4bc25b53 and https://habla.news/u/alex@gleasonator.dev/nostr-filters-limit
Logging query errors exposed this. This is why logging matters, people! Now I am switching Ditto to use JSON logging and incorporating Loki + Promtail so can easily track these problems in the future.
As for nip05 lookups - Ditto actually does nip05 verification in realtime as feeds are being rendered. This is obviously very bad. It does have an in-memory LRU cache, but restarting Ditto resets it. As a result, feeds can take up to 3 seconds to load (that's the timeout I set for fetching nip05). And if fetching fails, usernames are not displayed correctly.
To solve this (with a bandaid solution), I configured squid (HTTP proxy) to aggressively cache nip05 responses with this:
refresh_pattern /\.well-known/nostr\.json 3600 0% 259200
I also had to configure Ditto and squid with self-signed SSL certs so it can mitm and actually cache:
http_proxy="http://squid.lan:3128";
https_proxy="http://squid.lan:3128";
DENO_CERT="/etc/ssl/certs/squid.crt"
My TODO is to cache nip05 verifications in Postgres, to solve the root problem for self-hosters and increase performance even more.
tl;dr Ditto is fast now.
There were 2 issues: Postgres performance and nip05 lookups.
Postgres performance was caused by profile lookups, eg:
{ kinds: [0], authors: [a, b, c, d, ...] }
The issue was ORDER BY clauses on those queries. This caused Postgres to do a "sequence scan" instead of using the correct index:
"nostr_events_replaceable_idx" UNIQUE, btree (kind, pubkey) WHERE kind >= 10000 AND kind < 20000 OR (kind = ANY (ARRAY[0, 3]))
I actually fixed this already last year, and then reverted it in the last few months due to a bad decision. 🤦 See: https://gitlab.com/soapbox-pub/nostrify/-/commit/6a51c3f8a73c8a9f852a46d57119afdd4bc25b53 and https://habla.news/u/alex@gleasonator.dev/nostr-filters-limit
Logging query errors exposed this. This is why logging matters, people! Now I am switching Ditto to use JSON logging and incorporating Loki + Promtail so can easily track these problems in the future.
As for nip05 lookups - Ditto actually does nip05 verification in realtime as feeds are being rendered. This is obviously very bad. It does have an in-memory LRU cache, but restarting Ditto resets it. As a result, feeds can take up to 3 seconds to load (that's the timeout I set for fetching nip05). And if fetching fails, usernames are not displayed correctly.
To solve this (with a bandaid solution), I configured squid (HTTP proxy) to aggressively cache nip05 responses with this:
refresh_pattern /\.well-known/nostr\.json 3600 0% 259200
I also had to configure Ditto and squid with self-signed SSL certs so it can mitm and actually cache:
http_proxy="http://squid.lan:3128";
https_proxy="http://squid.lan:3128";
DENO_CERT="/etc/ssl/certs/squid.crt"
My TODO is to cache nip05 verifications in Postgres, to solve the root problem for self-hosters and increase performance even more.
tl;dr Ditto is fast now.