Announcement

Collapse
No announcement yet.

well I lost everything on my server

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #21
    Re: well I lost everything on my server

    Originally posted by Curious.George View Post
    But you then need ZFS to access the medium. What do you do if the box(es) that support it are down?

    The same problem applies to the various RAID technologies. When your RAID hardware (or system) dies, how do you access (or recover) the contents of those volumes?
    Actually not difficult if we are talking about HD's and software RAID. My NAS is a HP Microserver with 5 WD Red's running software ZFS-raid (RAIDZ) under Nas4free. The whole Nas4free is installed on a bootable USB-pen. If I need to move my NAS to new hardware, I pull the drives and the USB-pen, connect it up to new hardware and boot. Should the USB-pen be bad, I keep a copy of the config file elsewhere.

    Due to space restrictions and drive sizes at the time, I elected to have only RAID-Z1 to achieve my storage needs. RAID-Z2 is possible, but not recommended on a 5 disk setup. It's possible to use up to RAID-Z3 with ZFS, were up to 3 drives can go bad, and still save the pool.
    Last edited by sofTest; 07-26-2018, 07:25 PM.
    ------------
    Be a mensch

    Comment


      #22
      Re: well I lost everything on my server

      Originally posted by sofTest View Post
      Actually not difficult if we are talking about HD's and software RAID. My NAS is a HP Microserver with 5 WD Red's running software ZFS-raid (RAIDZ) under Nas4free.
      Now, your HP Microserver dies -- power supply catches fire.

      You pull your 5 drives. And, discover that the only other box you have on hand AT THIS MOMENT will only support 4 drives. Or, 2. Ooops!

      To truly protect your data investment, you need a spare HP Microserver lying around, idle. (or, the ability to MAKE it idle while you access your files).

      If your array uses an advanced RAID controller (so-called "hardware RAID"), then you also need to have one of those controllers on hand (and whatever box it runs inside).

      If you buy a RAID appliance, then you're faced with having to sort out how that vendor chose to implement the technology in that device. (you're not going to pull the drives from a Synology box (that has died) and throw them into a QNAP box.

      If you only have one array, keeping a spare isn't that hard (though it represents a big cost penalty unless you can put it into use -- and live with NO spare :> ). Once you end up with many arrays, you either have to tie yourself down to one make/model/implementation or deal with having lots of different spares on hand (as you can't know which will crap out).

      [I can have upwards of 70 disks spinning at any given time. And, I can "recover" from ALL of that hosting hardware failing, simultaneously, by dragging out a laptop and an external, single-drive dock. No magic OS to install cuz the volumes aren't "special" in any way -- I can even read them under Windows]

      Comment


        #23
        Re: well I lost everything on my server

        Originally posted by Curious.George View Post
        Now, your HP Microserver dies -- power supply catches fire.

        You pull your 5 drives. And, discover that the only other box you have on hand AT THIS MOMENT will only support 4 drives. Or, 2. Ooops!

        To truly protect your data investment, you need a spare HP Microserver lying around, idle. (or, the ability to MAKE it idle while you access your files).
        Everyone should have a strategy on how to get access to their data according to urgency. The most important thing is to make sure that important data is likely retrievable and intact at all, even if it takes a day or a week.

        If your array uses an advanced RAID controller (so-called "hardware RAID"), then you also need to have one of those controllers on hand (and whatever box it runs inside).

        If you buy a RAID appliance, then you're faced with having to sort out how that vendor chose to implement the technology in that device. (you're not going to pull the drives from a Synology box (that has died) and throw them into a QNAP box.
        Yes, that's why I went for a software RAID, and explicitly talked about software RAID...

        If you only have one array, keeping a spare isn't that hard (though it represents a big cost penalty unless you can put it into use -- and live with NO spare :> ). Once you end up with many arrays, you either have to tie yourself down to one make/model/implementation or deal with having lots of different spares on hand (as you can't know which will crap out).

        [I can have upwards of 70 disks spinning at any given time. And, I can "recover" from ALL of that hosting hardware failing, simultaneously, by dragging out a laptop and an external, single-drive dock. No magic OS to install cuz the volumes aren't "special" in any way -- I can even read them under Windows]
        You haven't researched this have you? If I wanted to, I could put my drives in USB docks, use simple USB-SATA cables or USB-enclosures (mixed any way I want) and hook them up to a laptop, to use your "strategy". I could even have some on SATA on a MB and some on USB. If I wanted to, I could have run my NAS entirely on external USB-drives. With todays SS-USB, it probably wouldn't be much of a speed penalty either.

        If your juggling and keeping track of disks, WORM's, tapes or whatever suits you, fine. I certainly have no problem with it. My argument about it's not difficult moving a software RAID, is still valid.
        ------------
        Be a mensch

        Comment


          #24
          Re: well I lost everything on my server

          Originally posted by sofTest View Post
          You haven't researched this have you? If I wanted to, I could put my drives in USB docks, use simple USB-SATA cables or USB-enclosures (mixed any way I want) and hook them up to a laptop, to use your "strategy". I could even have some on SATA on a MB and some on USB. If I wanted to, I could have run my NAS entirely on external USB-drives. With todays SS-USB, it probably wouldn't be much of a speed penalty either.
          Now you've got a bunch of docks/enclosures to keep on a shelf -- because you need to be able to mount all drives in the array at a given time -- just to give you a compatible hardware interface. Then, have to ensure the drives are probed in the correct order (so the RAID software doesn't think you've mixed them up). And, you're just dealing with 5 drives!

          Now, manage ~100T of data like that -- which set of drives go together? How were they configured? What's on them? Which physical case holds the first drive in the array? The second?? etc.

          By contrast, I can take a drive to a neighbor, the local library or any other computer and read its contents -- without having to boot from a thumb drive/CD/DVD and hope the hardware in that computer is "compatible" with that software.

          If your juggling and keeping track of disks, WORM's, tapes or whatever suits you, fine. I certainly have no problem with it. My argument about it's not difficult moving a software RAID, is still valid.
          Yet people always seem to have problems when their feet are in the fire.

          Throw one of your drives away. Today (cuz you presumably are READY for a catastrophic failure TODAY!) Wait a week so you know it's in a landfill, somewhere. Now, try to recover (if you're confident that you CAN do this, then this shouldn't be an issue for you and you shouldn't even break into a sweat over it!)

          [I invited a colleague to "borrow" one of my drives on a recent visit to prove that I could recreate it's contents (though not its sector-wise image) while he had it in his possession. If I couldn't, the drive (and its contents) would be his to keep!]

          Repeat the exercise with a new version of the RAID software (or, try FreeNAS instead). I.e., imagine having built an array -- perhaps a BACKUP of your array -- today. Placed it on the shelf. Then, in 2020, dragging it off the shelf (because your main array was vandalized or stolen) and try to resurrect it. Do you remember what version of the software you used? Are you sure the 2020 version's changes are 100% compatible?

          "FOSS" doesn't mean "bug free" any more than COTS means "buggy"!

          With an archive as large as mine, 98% of it sits dormant on a shelf for very long periods of time. When I start a new project, I build a "guest disk" with whatever I need, culled from the archive. I have media that were written almost a decade ago by various different OS's. All I care about is the format of the BASIC filesystem(s) used, not additional structures layered atop that (by the RAID software). The 720K 3" floppies (FAT12) for my Unisite are handled just as easily as the 4TB (NTFS) SATA disks and the 146GB SCA (FFS) drives.

          [I've owned -- and retired -- perhaps a dozen RAID/NAS appliances over the years. Each with its own set of headaches as each vendor/model chose to implement things slightly differently from model to model or vendor to vendor. Bare drives and shelfs are the only approach that work reliably for really large data stores managed without an IT budget or staff.]

          Comment


            #25
            Re: well I lost everything on my server

            Originally posted by Curious.George View Post
            [Snipp] Bare drives and shelfs are the only approach that work reliably for really large data stores managed without an IT budget or staff.]
            Until your building burns down, hit by a meteor, or similar. Or your entire city get destroyed by some other freak of nature. You are talking about far fetched scenarios, don't you?

            Your don't get it do you? I don't care about your backup strategy, or whatever you call it.

            I don't even talk about any backup strategy, not even mine. I'm not talking about long time offline data storage either. A NAS alone will never be a backup strategy, but might be part of a backup strategy.

            Not important for this discussion, but for your info. The order of the disks are not important when moving ZFS. You can always import a pool, whatever order of the disks. ZFS uses vdevs and pools, not arrays. You really do need to update.

            My only, and ONLY, point is that it is usually not that difficult to transplant a software RAID, if you happen to set one up. Help me out here folks, am I so unclear?
            ------------
            Be a mensch

            Comment


              #26
              Re: well I lost everything on my server

              Originally posted by sofTest View Post
              My only, and ONLY, point is that it is usually not that difficult to transplant a software RAID, if you happen to set one up. Help me out here folks, am I so unclear?
              As a friend who operates a large data center for a multinational bank is so fond of pointing out:

              "There are two types of RAID (or any similar technology) users: those who HAVE experienced data loss -- and those who WILL"

              Hint: the ONLY time I came close to losing data was when an OS upgrade introduced a bug into the SCSI disk driver that caused the disks attached to the controller to be scrambled. So much for "free" software...

              Comment


                #27
                Re: well I lost everything on my server

                Originally posted by Curious.George View Post
                As a friend who operates a large data center for a multinational bank is so fond of pointing out:

                "There are two types of RAID (or any similar technology) users: those who HAVE experienced data loss -- and those who WILL"
                Sure, right. Have I said that RAID is a warranty against data loss? Again, RAID is no substitute for good backup routines, of which I have said nothing. RAID only reduces the risk for data loss. But I know one thing, any data centres for multinational banks does not operate shelves with bare drives.

                Hint: the ONLY time I came close to losing data was when an OS upgrade introduced a bug into the SCSI disk driver that caused the disks attached to the controller to be scrambled. So much for "free" software...
                Hint: No software, paid or free, gives warranties against data loss. I you know any, please list them, and I'll be sure to stay away from them. They would be as dishonest as their warranties.

                BTW, many of the biggest data centres in the world run "free" software.
                ------------
                Be a mensch

                Comment


                  #28
                  Re: well I lost everything on my server

                  ^ As does the supercomputer that the NWS uses. And Greenhouse Data Centers. And the Echostar uplink center. And Walmart. And even Microsoft in the Azure DC over here.
                  Things I've fixed: anything from semis to crappy Chinese $2 radios, and now an IoT Dildo....

                  "Dude, this is Wyoming, i hopped on and sent 'er. No fucking around." -- Me

                  Excuse me while i do something dangerous


                  You must have a sad, sad boring life if you hate on people harmlessly enjoying life with an animal costume.

                  Sometimes you need to break shit to fix it.... Thats why my lawnmower doesn't have a deadman switch or engine brake anymore

                  Follow the white rabbit.

                  Comment


                    #29
                    Re: well I lost everything on my server

                    Originally posted by goontron View Post
                    ^ As does the supercomputer that the NWS uses. And Greenhouse Data Centers. And the Echostar uplink center. And Walmart. And even Microsoft in the Azure DC over here.
                    And they all have paid staff working to keep that software running properly. How many guys in your (personal) IT department? :>

                    [I suspect I'm one of the few folks, here, who has actually done any kernel hacking or contributed to any FOSS projects -- having run NetBSD since 0.8 and FreeBSD since 0.9 (with the 5" floppies to prove it!)]

                    Comment


                      #30
                      Re: well I lost everything on my server

                      Originally posted by Curious.George View Post
                      And they all have paid staff working to keep that software running properly. How many guys in your (personal) IT department? :>

                      [I suspect I'm one of the few folks, here, who has actually done any kernel hacking or contributed to any FOSS projects -- having run NetBSD since 0.8 and FreeBSD since 0.9 (with the 5" floppies to prove it!)]
                      Sure, if you don't feel competent, or are unable or unwilling to acquire the required competence, you should stay away. The knowledge is out there for the grab for those interested. Today you can also pay for free software training, as for paid software. It's not like paid software doesn't require competence. Some even have paid and free versions of their software, often with main difference access to support.

                      As for FreeBSD, Nas4free is based on that. From above about ZFS, you have obviously not staid current/updated. ZFS has been part of FreeBSD since v7 and NetBSD since 2009.
                      ------------
                      Be a mensch

                      Comment


                        #31
                        Re: well I lost everything on my server

                        Originally posted by Curious.George View Post
                        And they all have paid staff working to keep that software running properly. How many guys in your (personal) IT department? :>

                        [I suspect I'm one of the few folks, here, who has actually done any kernel hacking or contributed to any FOSS projects -- having run NetBSD since 0.8 and FreeBSD since 0.9 (with the 5" floppies to prove it!)]
                        Well, considering i am employed by one of the quoted companies I AM a support personnel.... And that companies name doesn't start with a "W".

                        BTW, you might want to hop off that horse before you fall, i got a feeling you don't know how to ride.
                        Last edited by goontron; 07-28-2018, 05:46 PM.
                        Things I've fixed: anything from semis to crappy Chinese $2 radios, and now an IoT Dildo....

                        "Dude, this is Wyoming, i hopped on and sent 'er. No fucking around." -- Me

                        Excuse me while i do something dangerous


                        You must have a sad, sad boring life if you hate on people harmlessly enjoying life with an animal costume.

                        Sometimes you need to break shit to fix it.... Thats why my lawnmower doesn't have a deadman switch or engine brake anymore

                        Follow the white rabbit.

                        Comment


                          #32
                          Re: well I lost everything on my server

                          Originally posted by sofTest View Post
                          As for FreeBSD, Nas4free is based on that. From above about ZFS, you have obviously not staid current/updated. ZFS has been part of FreeBSD since v7 and NetBSD since 2009.
                          I ran ZFS under Slowaris more than a decade ago when it was still a part of Sun's offerings. It simply will not address the size of my archive (without me NEEDLESSLY having all that rust spinning).

                          I really want to keep the source code, schematics, board layouts, component datasheets/manuals, toolchains, etc. for every project I've ever designed. And my tax records. And my music archive. And...

                          Do I want them to reside on media that is accessible any time of day or night? Nope. So, why have them spinning, needlessly? Should I buy 30 NASs and leave them all powered off until needed (configured for whatever level of redundancy I deem appropriate for those particular files)?

                          Do you have all your CD/DVD media in individual drives so you can access them at a moments notice? (I doubt it) OTOH, I suspect you want to be able to access the data stored on those media WHEN you decide you have a need for it. What measures have you taken to ensure that it remains accessible?

                          I've come up with a solution that lets me:
                          • know what I have
                          • where its located
                          • verify its integrity
                          • locate redundant copies in case of corruption


                          And, it costs me only -- and exactly -- the price of the media required to store those copies. I have no special hardware or software requirements to recover (read) any of that data. Or, alter it. And, use zero power to run that "system".

                          Comment


                            #33
                            Re: well I lost everything on my server

                            is the data not recoverable ?

                            Comment


                              #34
                              Re: well I lost everything on my server

                              Originally posted by goontron View Post
                              Well, considering i am employed by one of the quoted companies I AM a support personnel.... And that companies name doesn't start with a "W".
                              So, the size of your IT department is exactly 1. And the time you spend on "support" comes at the expense of the other activities in which you'd probably prefer to be engaged.

                              I would much rather be designing a new piece of medical equipment or a slot machine or a process control system than attending to a "storage mechanism". OTOH, I rely on that storage mechanism to enable me to design new pieces of medical equipment, slot machines, process control systems, etc. So, it becomes an issue of how to maximize the effectiveness of ANY solution while minimizing its ongoing costs of maintenance.

                              BTW, you might want to hop off that horse before you fall, i got a feeling you don't know how to ride.
                              You'd best hope I'm a great rider because many of the devices I've designed have a significant impact on your health and life! :> Wouldn't want you to get misdiagnosed as HIV positive; or, have your car stop in the middle of the highway; or, ...

                              Comment


                                #35
                                Re: well I lost everything on my server

                                kids kids kids.....lets all be friends!
                                <--- Badcaps.net Founder

                                Badcaps.net Services:

                                Motherboard Repair Services

                                ----------------------------------------------
                                Badcaps.net Forum Members Folding Team
                                http://folding.stanford.edu/
                                Team : 49813
                                Join in!!
                                Team Stats

                                Comment


                                  #36
                                  Re: well I lost everything on my server

                                  Originally posted by Curious.George View Post


                                  You'd best hope I'm a great rider because many of the devices I've designed have a significant impact on your health and life! ... or, have your car stop in the middle of the highway; or, ...
                                  Jokes on you. It already has! The computer locked up and the engine went WOT until it blew. I hope you never worked on the computers for the '06 Prius.

                                  EDIT: And before you say "throttle issue", it didn't go into neutral, the ODO was unresponsive, and i was able to pull the wiring loom off on the throttle. Still ran away.
                                  Last edited by goontron; 07-28-2018, 08:16 PM.
                                  Things I've fixed: anything from semis to crappy Chinese $2 radios, and now an IoT Dildo....

                                  "Dude, this is Wyoming, i hopped on and sent 'er. No fucking around." -- Me

                                  Excuse me while i do something dangerous


                                  You must have a sad, sad boring life if you hate on people harmlessly enjoying life with an animal costume.

                                  Sometimes you need to break shit to fix it.... Thats why my lawnmower doesn't have a deadman switch or engine brake anymore

                                  Follow the white rabbit.

                                  Comment


                                    #37
                                    Re: well I lost everything on my server

                                    Originally posted by Curious.George View Post
                                    I ran ZFS under Slowaris more than a decade ago when it was still a part of Sun's offerings. It simply will not address the size of my archive (without me NEEDLESSLY having all that rust spinning).
                                    ZFS from more than a decade ago is not relevant. A lot has happened with it since then, and features are still being implemented. From what you have proven above, your level of knowledge of ZFS is likely being able to spell ZFS. No one have suggested that you should use any other storage solution than you have today. Your data, your time, your problem.

                                    I really want to keep the source code, schematics, board layouts, component datasheets/manuals, toolchains, etc. for every project I've ever designed. And my tax records. And my music archive. And...

                                    Do I want them to reside on media that is accessible any time of day or night? Nope. So, why have them spinning, needlessly? Should I buy 30 NASs and leave them all powered off until needed (configured for whatever level of redundancy I deem appropriate for those particular files)?

                                    Do you have all your CD/DVD media in individual drives so you can access them at a moments notice? (I doubt it) OTOH, I suspect you want to be able to access the data stored on those media WHEN you decide you have a need for it. What measures have you taken to ensure that it remains accessible?

                                    I've come up with a solution that lets me:
                                    • know what I have
                                    • where its located
                                    • verify its integrity
                                    • locate redundant copies in case of corruption


                                    And, it costs me only -- and exactly -- the price of the media required to store those copies. I have no special hardware or software requirements to recover (read) any of that data. Or, alter it. And, use zero power to run that "system".
                                    Really? However many times you keep repeating yourself, I am not interested in how you organize your storage needs. Your data (hopefully), your time, your problem, not interested. I will however say as much that your scheme is not for everyone, and if I proposed it in any sort of professional capacity, I wouldn't keep that capacity long.

                                    As for what personal data I keep online and offline; my data, my time, my problem, none of your business.
                                    ------------
                                    Be a mensch

                                    Comment


                                      #38
                                      Re: well I lost everything on my server

                                      Originally posted by sofTest View Post
                                      Really? However many times you keep repeating yourself, I am not interested in how you organize your storage needs. Your data (hopefully), your time, your problem, not interested. I will however say as much that your scheme is not for everyone, and if I proposed it in any sort of professional capacity, I wouldn't keep that capacity long.
                                      It's not intended for work in a professional setting where folks have staff dedicated to IT! It's intended for folks with HUGE archives (do you have at least 20T in your pool?) and who want not to spend time rebuilding arrays and wondering if the latest update will break something and render their data inaccessible.

                                      I'll be delighted to hear how YOU solve this problem when your dataset approaches 100T. Heck, you can even fantasize and SUGGEST your solution, today! In your "professional capacity", that is...

                                      I wouldn't keep my "professional capacity" long if I couldn't reproduce the entire development environment for any of the products that I've created years after their initial release. Does the latest version of Cadence recognize PCB layouts created from OrCAD 9? 7?? Can I modify the design of the FPGA that I designed under the original Xilinx tools with the latest Vivado suite? Will the latest version of gcc handle the extensions that the Whitesmith compiler supported?

                                      If not, those products can't be maintained or evolved ("We'd like to do a version 2 of that product you designed for us several years ago"). Of course, if you've got a full-time staff on hand, they can keep upgrading your tools and porting old designs forward -- all reflected in your "overhead".

                                      Perhaps you've never been in a position where your client is genuinely concerned about the consequences -- to HIS organization -- of your being hit by a bus. You're likely pretty easily replaced. My files/designs/data aren't. Being able to prove that to clients goes a long way to reassuring them -- and justifying the rates that I charge!

                                      Comment


                                        #39
                                        Re: well I lost everything on my server

                                        Originally posted by goontron View Post
                                        Jokes on you. It already has! The computer locked up and the engine went WOT until it blew. I hope you never worked on the computers for the '06 Prius.

                                        EDIT: And before you say "throttle issue", it didn't go into neutral, the ODO was unresponsive, and i was able to pull the wiring loom off on the throttle. Still ran away.
                                        Then you've seen the consequences of a NOT great rider! I've never had a design come back to me after it was released. And, as my contracts include a "bugs fixed free" clause -- with no limits on time since original release -- the client always has a strong incentive to throw any such problems back to me.

                                        Comment


                                          #40
                                          Re: well I lost everything on my server

                                          Originally posted by Curious.George View Post
                                          It's not intended for work in a professional setting where folks have staff dedicated to IT! It's intended for folks with HUGE archives (do you have at least 20T in your pool?) and who want not to spend time rebuilding arrays and wondering if the latest update will break something and render their data inaccessible.

                                          I'll be delighted to hear how YOU solve this problem when your dataset approaches 100T. Heck, you can even fantasize and SUGGEST your solution, today! In your "professional capacity", that is... [snip]
                                          Nope, one do not design storage systems based on information on need for storage capacity alone. But you wouldn't know that. But I'm not going to waste more time on this. Based on what you have proven not knowing above, and other wild claims, I'm sorry to say that I'm leaning more toward you trolling than credible.
                                          Last edited by sofTest; 07-29-2018, 10:26 AM.
                                          ------------
                                          Be a mensch

                                          Comment

                                          Working...
                                          X