well I lost everything on my server

[Curious.George]

Originally Posted by sofTest

Nope, one do not design storage systems based on information on need for storage capacity alone. But you wouldn't know that. But I'm not going to waste more time on this. Based on what you have proven not knowing above, and other wild claims, I'm sorry to say that I'm leaning more toward you trolling than credible.

(sigh) Sad little person. You've obviously not considered the consequences of your decisions. Fear not, life will teach you these lessons! You'll remember this discussion and be even angrier at me when you realize the prophecy!

Ask yourself how you're going to increase your storage capacity: copy everything off the ZFS onto ???; tear down your vdevs, add another drive and then rebuild everything and copy everything back? Or, replace drives, one at a time in each vdev with larger drives, resilver and then then let the pool expand? Discard the old (now too small) drives? Add a second vdev (3 more drives) -- if the hardware can support it? Add a second box with a different configuration (components of which will be useless as spares for the first box; and vice versa).

You may not have 100T now. But, once you start getting used to having "unlimited storage" you'll discover that you use it, fast. With ZFS/RAID, all of the data is spinning at any given time. So, it's all AT RISK. A careless keystroke, a catastrophic hardware failure, a bug in the software, etc.

(So, just keep duplicates of EVERY box that you have as insurance! )

I add new drives just by plugging them in (unplugging something that isn't CURRENTLY being used). I keep 1, 2 or 30 copies of files as I see fit. They can reside on the same volumes or on N different ones -- in any combination. They can have different names and reside in different directories across volumes -- beyond even the capacity of symlinks to map equivalences!

I can mail a medium to a client and he can read/write it at his leisure. When it comes back to me, I can see what (if anything) has been changed. I can store any portion of my archive offsite on as many volumes as I see fit (including just one). A client can mail me a volume and I can import it to my archive "as is" -- so, when he refers to "/projects/2010/summer/proposals/blah" I'll be able to directly and unambiguously access it.

I can access it via Solaris, Windows (XP, Vista, 7, 8, 10 -- even 95/98 with USB support), Linux, MacOSX -- or even a thin client with just 100MB of RAM (and no display!). I can upgrade any of these OS's and never put more than one volume at risk (to prove the integrity of the OS upgrade).

I can browse my entire repository's namespace without mounting ANY volumes. I can create a new volume drawing files from any number of volumes. I can create volumes on R/O media. I can do this on a laptop without AC mains being available.

I can FREELY incorporate volumes of different types, technologies and interfaces (SCA/FC-AL/SCSI/SATA/PATA/floppy/memory card/etc.). I can export an NFS/SMB share and treat it as part of the archive -- verifying its integrity or automatically "reinitializing it" (mark it as corrupt) with the "backup originals" from the rest of the archive.

I don't need special tools to do any of them (other than some SQL scripts and the ability to mount volumes in whatever way the host OS desires). So, I don't have to worry about updates to those tools "breaking things" or putting the entire archive at risk.

You can do all of those things with ZFS, right?

In your professional capacity , address the "hypothetical" client with 100T of storage doubling every 3-4 years. How should he ensure he can access old tax records (in the UNLIKELY event of an audit), old sales histories (when proposing a new product), old development costs (when budgeting the development effort for that new product), etc. Be sure to tell him what sort of equipment he should expect to buy and what sort of staff (and labor costs) he should be prepared to encounter in order to maintain that data. Prepare a budget of the hardware maintenance costs for the foreseeable future (how long before those drives fail? How much money and time to replace?

I'll gladly present your recommendations to MY clients -- as EXPERT ADVICE. So, be sure to reflect all the costs/benefits in your proposal.

Or, are you a one-trick pony? Would you have advocated building an addition on the corporate office to hold more FILE CABINETS to keep ALL of the companies records "accessible" -- despite the age or value thereof -- instead of moving old stuff to "secondary storage"? Sure sounds like it!

To mimic your comments, based on how evasive you have been over real issues (you haven't even disclosed the size of your array, number of objects it contains, the hardware its running on, your costs to implement/maintain, etc.) and other wild claims, I'm sorry to say that I'm leaning more toward you being a simple ZFS fanboi rather than credible.

[or, having sorely limited experience "in the wild"... ever work for a company with 10 employees? 100? 1000? 100000? If you have, you'll know that the way problems are solved varies greatly, as do the resources available to throw at problems! Hint: I've worked at or for all of the above and had to come up with solutions that "made sense" to each of them]

I'll respond when you've proposed a solution that I'll gladly pass by real clients for their unbiased opinions. I'm sure the big companies with systems in place will be EAGER (not!) to adopt your solution. As will the companies with no IT departments.

[sofTest]

Originally Posted by Curious.George

(sigh) Sad little person. You've obviously not considered the consequences of your decisions. Fear not, life will teach you these lessons! You'll remember this discussion and be even angrier at me when you realize the prophecy!

Ask yourself how you're going to increase your storage capacity: copy everything off the ZFS onto ???; tear down your vdevs, add another drive and then rebuild everything and copy everything back? Or, replace drives, one at a time in each vdev with larger drives, resilver and then then let the pool expand? Discard the old (now too small) drives? Add a second vdev (3 more drives) -- if the hardware can support it? Add a second box with a different configuration (components of which will be useless as spares for the first box; and vice versa).

You may not have 100T now. But, once you start getting used to having "unlimited storage" you'll discover that you use it, fast. With ZFS/RAID, all of the data is spinning at any given time. So, it's all AT RISK. A careless keystroke, a catastrophic hardware failure, a bug in the software, etc.

(So, just keep duplicates of EVERY box that you have as insurance! )

I add new drives just by plugging them in (unplugging something that isn't CURRENTLY being used). I keep 1, 2 or 30 copies of files as I see fit. They can reside on the same volumes or on N different ones -- in any combination. They can have different names and reside in different directories across volumes -- beyond even the capacity of symlinks to map equivalences!

I can mail a medium to a client and he can read/write it at his leisure. When it comes back to me, I can see what (if anything) has been changed. I can store any portion of my archive offsite on as many volumes as I see fit (including just one). A client can mail me a volume and I can import it to my archive "as is" -- so, when he refers to "/projects/2010/summer/proposals/blah" I'll be able to directly and unambiguously access it.

I can access it via Solaris, Windows (XP, Vista, 7, 8, 10 -- even 95/98 with USB support), Linux, MacOSX -- or even a thin client with just 100MB of RAM (and no display!). I can upgrade any of these OS's and never put more than one volume at risk (to prove the integrity of the OS upgrade).

I can browse my entire repository's namespace without mounting ANY volumes. I can create a new volume drawing files from any number of volumes. I can create volumes on R/O media. I can do this on a laptop without AC mains being available.

I can FREELY incorporate volumes of different types, technologies and interfaces (SCA/FC-AL/SCSI/SATA/PATA/floppy/memory card/etc.). I can export an NFS/SMB share and treat it as part of the archive -- verifying its integrity or automatically "reinitializing it" (mark it as corrupt) with the "backup originals" from the rest of the archive.

I don't need special tools to do any of them (other than some SQL scripts and the ability to mount volumes in whatever way the host OS desires). So, I don't have to worry about updates to those tools "breaking things" or putting the entire archive at risk.

You can do all of those things with ZFS, right?

In your professional capacity , address the "hypothetical" client with 100T of storage doubling every 3-4 years. How should he ensure he can access old tax records (in the UNLIKELY event of an audit), old sales histories (when proposing a new product), old development costs (when budgeting the development effort for that new product), etc. Be sure to tell him what sort of equipment he should expect to buy and what sort of staff (and labor costs) he should be prepared to encounter in order to maintain that data. Prepare a budget of the hardware maintenance costs for the foreseeable future (how long before those drives fail? How much money and time to replace?

I'll gladly present your recommendations to MY clients -- as EXPERT ADVICE. So, be sure to reflect all the costs/benefits in your proposal.

Or, are you a one-trick pony? Would you have advocated building an addition on the corporate office to hold more FILE CABINETS to keep ALL of the companies records "accessible" -- despite the age or value thereof -- instead of moving old stuff to "secondary storage"? Sure sounds like it!

To mimic your comments, based on how evasive you have been over real issues (you haven't even disclosed the size of your array, number of objects it contains, the hardware its running on, your costs to implement/maintain, etc.) and other wild claims, I'm sorry to say that I'm leaning more toward you being a simple ZFS fanboi rather than credible.

[or, having sorely limited experience "in the wild"... ever work for a company with 10 employees? 100? 1000? 100000? If you have, you'll know that the way problems are solved varies greatly, as do the resources available to throw at problems! Hint: I've worked at or for all of the above and had to come up with solutions that "made sense" to each of them]

I'll respond when you've proposed a solution that I'll gladly pass by real clients for their unbiased opinions. I'm sure the big companies with systems in place will be EAGER (not!) to adopt your solution. As will the companies with no IT departments.

Yupp, trolling.^

[Curious.George]

Originally Posted by sofTest

Yupp, trolling.^

^Yup, incompetent.

(amazing how when forced to stand by your words, you run and hide... I guess even YOU don't believe what you're saying!)

Bye, bye, little person!

[sofTest]

Originally Posted by Curious.George

^Yup, incompetent.

(amazing how when forced to stand by your words, you run and hide... I guess even YOU don't believe what you're saying!)

Bye, bye, little person!

Forced to stand by my words? My only claim from the start has been that it is not usually not hard to move a software RAID. I have not solicited people, not even you, to use any kind of storage solution. You on the other hand has repeatedly and boringly droned on about your magnificent shelf stores, under ways making several incompetent claims about ZFS. Then you are trying to draw me out on "designing" a storage system, despite I clearly has stated before that your data, your problem, not interested. Then you want information about my system, despite I clearly said before, none of your business. You sir, are simply a repetitive boring troll, and I'm simply to old to play.

[momaka]

So back to the original topic in hand...

Uranium, does that mean the entire electrolytic capacitor data sheet database is gone too? :\
I hope not, and it would be a shame if it is. But I guess these things happen.

Good thing I kind of keep my own "database" of cap data sheets.

Similar to Curious.George's, I forgo the whole RAID/ZFS stuff and just keep copies onto random PCs, spare HDDs, Flash drivers, and DVDs/CDs (but much MUCH less elaborate and more manual/not automated compared to his, if it can even be compared ).

I'm actually pretty messy about my backups. Some files I have backed up in over 6 places (again, different PCs with different HDDs/interfaces and OSes, CDs, and flash drives), and some files I have only in one or two backup HDDs. It all depends how important the files are to me. Things like my pictures collection is well backed up: it exists on my main PC, a backup (portable) HDD, my laptop (which is also portable, sort of ), and another PC in my uncle's house in an entirely different continent. I also have partial backups of that collection in various PCs I've used over the years before that (partial, because at some point these PCs were my main PCs, and that means at some point they were the "original" copy/backup before I retired them).

So all in all, even a major wipe out will still likely leave me with *most* (if not all) of my data intact. Because of the messy way I've backed up things, of course, retrieving some files back will be a pain in the butt, if I happen to need them again. But nothing lasts forever, so whatever!

[Uranium-235]

the cap db is hosted on a 1und1 account (I say und because it's a german company :P)

I might get another TB drive in there to do monthly backups to. I've been reading on commands to spindown/spinup drives, so I might make a script to spinup (which I think can be done just by mounting), backup, mark spindown time as short, and unmount it

or get an Esata -> Sata bracket, and put the drive in an enclosure with an esata port and hotswap it, power off, so it's not exposed to possible power fluxuations, and I can keep the backup schedule on my time