Stefano Marinelli on Nostr: Just received an email: "Hey, I'm out of (Next)cloud space! How's that possible? I ...
Just received an email: "Hey, I'm out of (Next)cloud space! How's that possible? I have a 1TB limit, but I don't think I have that many files!"
It's a client's Linux server, using an ext4 file system. I'm authorized to check it out.
I find a roughly 30GB folder "Archive 2013" in the 2014 directory – but there's also a 2013 folder with the same files.
Digging deeper, the 2015 folder contains "Archive 2014," with all files from 2014, including the 2013 archive.
And so on.
Without a deduplicating file system, space just adds up. I inform the client and suggest keeping only one archive copy per year, without duplicating it in subsequent years. I recommend having the current year (e.g., 2024) and, at a parent directory, all archive directories. “Oh no, then to find a file from 2013, I have to go back (!!!), while this way I just go 'forward,' year by year, until I reach the right directory. Can't we increase the limits? Clouds should have no limits, I've never seen Amazon say 'you have a limit of'.
I tried suggesting switching to a different file system, like ZFS (with deduplication) or BTRFS (supports it with external tools), but they're actually considering asking me to move the data to S3 for infinite growth.
I'll try to explain that, in my opinion, it makes no sense to spend hundreds of euros a year more (which can be significant for them) just to keep a copy of a copy of a copy of the archive. We'll see...
#Linux #CloudStorage #FileSystem #DataManagement #DataDuplication #ZFS #BTRFS #ITChallenges
It's a client's Linux server, using an ext4 file system. I'm authorized to check it out.
I find a roughly 30GB folder "Archive 2013" in the 2014 directory – but there's also a 2013 folder with the same files.
Digging deeper, the 2015 folder contains "Archive 2014," with all files from 2014, including the 2013 archive.
And so on.
Without a deduplicating file system, space just adds up. I inform the client and suggest keeping only one archive copy per year, without duplicating it in subsequent years. I recommend having the current year (e.g., 2024) and, at a parent directory, all archive directories. “Oh no, then to find a file from 2013, I have to go back (!!!), while this way I just go 'forward,' year by year, until I reach the right directory. Can't we increase the limits? Clouds should have no limits, I've never seen Amazon say 'you have a limit of'.
I tried suggesting switching to a different file system, like ZFS (with deduplication) or BTRFS (supports it with external tools), but they're actually considering asking me to move the data to S3 for infinite growth.
I'll try to explain that, in my opinion, it makes no sense to spend hundreds of euros a year more (which can be significant for them) just to keep a copy of a copy of a copy of the archive. We'll see...
#Linux #CloudStorage #FileSystem #DataManagement #DataDuplication #ZFS #BTRFS #ITChallenges