mleku on Nostr: #gm realy managed not to die after 55gb of data at an eventual average speed of ...
#gm
realy managed not to die after 55gb of data at an eventual average speed of 1.2mb/s throughput
today i will make the import function batch like 10k events at a time, so it doesn't get bogged down with so many little writes
after stopping it, it eventually gets to compacting the tables and cpu goes to 100%
i'm sure the tables compact better when they are written in a batch
it's also making me pause to think about how maybe it would be better to keep the files on a filesystem, and then you don't even need this export step, it's just a matter of copying those files... ah but too complexity...
well, here goes, batching it is...
outside it's rainy and windy like crazy, i woke up and it looks still very dark but i see all these cars driving around the streets, i'm like, oh, it must be past 7am already, yup
well, here goes
realy managed not to die after 55gb of data at an eventual average speed of 1.2mb/s throughput
today i will make the import function batch like 10k events at a time, so it doesn't get bogged down with so many little writes
after stopping it, it eventually gets to compacting the tables and cpu goes to 100%
i'm sure the tables compact better when they are written in a batch
it's also making me pause to think about how maybe it would be better to keep the files on a filesystem, and then you don't even need this export step, it's just a matter of copying those files... ah but too complexity...
well, here goes, batching it is...
outside it's rainy and windy like crazy, i woke up and it looks still very dark but i see all these cars driving around the streets, i'm like, oh, it must be past 7am already, yup
well, here goes