Thorwegian ❄️ on Nostr: feels a bit wasteful for every Mastodon instance to keep a copy of the same file. ...
feels a bit wasteful for every Mastodon instance to keep a copy of the same file. some redundancy and decentralisation is good, but i wouldn't mind splitting the bill for storage with a few other instances.
how about an S3 proxy that deduplicates files to save space on copies? just store by hash instead of file path and have a key-value map so it looks like a file system to the Mastodon instance. one such map per instance connected to the shared storage pool.
you might even design it so it moves files away from shared storage and to instance-specific storage if only one instance needs that file, so you're only sharing the cost of shared files.
#mastoadmin
how about an S3 proxy that deduplicates files to save space on copies? just store by hash instead of file path and have a key-value map so it looks like a file system to the Mastodon instance. one such map per instance connected to the shared storage pool.
you might even design it so it moves files away from shared storage and to instance-specific storage if only one instance needs that file, so you're only sharing the cost of shared files.
#mastoadmin