Answer by ximik for Tips on efficiently storing 25TB+ worth million files in...
@Ankur and @Porch.I strongly agree with the necessity to compress these logs.@jetI think the simpler scheme is better - thus httpd for end user is close to ideal.And backend could be any.My opinion -...
View ArticleAnswer by jet for Tips on efficiently storing 25TB+ worth million files in...
export these folders via NFSmount them on a single machine with apache running (under document root) as a treeuse zip to compress them- good compress ratio, zip can be opened from all OSeslist files in...
View ArticleAnswer by Hubert Kario for Tips on efficiently storing 25TB+ worth million...
lessfs is a deduplicating, compressing file system. While it won't solve whole problem, it may be worth a look to look at as a backend.
View ArticleAnswer by Ryan for Tips on efficiently storing 25TB+ worth million files in...
First, log files can be compressed at really high ratios. I find my log files compress at a 10:1 ratio. If they compress to even a 5:1 ratio, that's only 5GB, or 20% of your storage capacity.Given that...
View ArticleAnswer by Porch for Tips on efficiently storing 25TB+ worth million files in...
You ever think about compressing the log files? Then do something on the frontend to uncompress them before serving them to the end user. Maybe a CGI script of sorts.
View ArticleAnswer by Scott McClenning for Tips on efficiently storing 25TB+ worth...
I'm not a distributed file system ninja, but after consolidating as many drives I can into as few machines as I can, I would try using iSCSI to connect the bulk of the machines to one main machine....
View ArticleTips on efficiently storing 25TB+ worth million files in filesystem
Say you are confronted with 25 TB worth uncompressed log files and you have at your disposal an array of 20 commodity boxes with collective free storage capacity of 25 TB.How would you store these ?....
View Article
