bert hubert 🇺🇦🇪🇺 on Nostr: Continuing the DuckDB insane speed. CSV File with ~15 million rows. Takes 0.5 seconds ...
Continuing the DuckDB insane speed. CSV File with ~15 million rows. Takes 0.5 seconds to wc -l. DuckDB imports it in 12 seconds. And then you can summarize that whole file .. in 0.3 seconds. And (not shown), if you use the select * from 'data.csv' syntax (so no import), it does the analysis + read in 5 seconds. Flat out INSANE.
Published at
2023-11-26 20:20:17Event JSON
{
"id": "cda4071b5ae515ddd6b3efe5db4af1c765750a6d3fe81aeda3610426db9fede9",
"pubkey": "b2a729af72f53eaf7a64944a3b4fcc1a5c4e4476852b80513bf405183c484187",
"created_at": 1701030017,
"kind": 1,
"tags": [
[
"proxy",
"https://fosstodon.org/users/bert_hubert/statuses/111478703235782201",
"activitypub"
]
],
"content": "Continuing the DuckDB insane speed. CSV File with ~15 million rows. Takes 0.5 seconds to wc -l. DuckDB imports it in 12 seconds. And then you can summarize that whole file .. in 0.3 seconds. And (not shown), if you use the select * from 'data.csv' syntax (so no import), it does the analysis + read in 5 seconds. Flat out INSANE.\n\nhttps://cdn.fosstodon.org/media_attachments/files/111/478/694/307/412/599/original/f650c4d42067017a.png",
"sig": "439691a9bf36bad9fc5433e24b556187e3b904f4004146d4667a2dd57589095814c7c47b570f86f65cecb1a8b9866fd473698ce9cf95703289e750159f83c72b"
}
