My Latest 8-core Project

[Topcat]

I have a source that contacts me usually once or twice a year and trades me a goodie box of various computer boards and hardware for some caps....well he calls and says there's a box headed my way. I asked what was in it, and he emailed me the list... Mostly a bunch of Dell GX745SFF boards in need of recap....but in every bunch from these guys, theres always a gem, and here it is:

http://www.supermicro.com/products/m...00/X7DWA-N.cfm

Its brand new, sealed in the retail package.....they couldn't test it as they don't have any processors for it...so they didn't even open it. AWW YEAHS!!! Looks like the Frankenserver's innards will be retired soon.

I got the goodie box including this board for about $50.00 worth of caps!!

[c_hegge]

WOW. I wish I got scores like that more often.

[Uranium-235]

right now the xeon 5150 771 seems to have dropped a few hundred bucks

http://www.amazon.com/Intel-1333MHz-.../dp/B000I1OFK2

[Topcat]

^
I actually thought about that. I'm not sure if that case can take an eATX board though...I'd have to see. The problem after that, I'd have no real use for it after that...

Originally Posted by Uranium-235

right now the xeon 5150 771 seems to have dropped a few hundred bucks

http://www.amazon.com/Intel-1333MHz-.../dp/B000I1OFK2

These are the CPU's I was looking at, seems like the best bang for the buck.

http://www.ebay.com/itm/320842242142

$42 bucks a pop with a 12mb cache....cant beat that with a stick!! I still have to hunt some DDR2 ECC. I have like 4gb worth of it to test with, but I'd probably run it with 32gb....the board maxes at 64gb, but good God in heaven, what would I need 64gb of RAM for (or even 32 for that matter)...

[ratdude747]

Originally Posted by Topcat

^
I actually thought about that. I'm not sure if that case can take an eATX board though...I'd have to see. The problem after that, I'd have no real use for it after that...

True EATX cases are hard to find for cheap...

if you find an old roomy case with a non-centered floppy /hdd or an old white gateway case you can dremel it into a hacked EATX case... I'me doing that to my modded gateway so I have a proper home for my massive EATX dual PIII board. To do this I will have to trim more off the HDD cage (it will only hold 2 HDDs instead of 3).

The board's old case was a poorly modded old ATX case with the lower 3 cdrom bays cut out... there may be pics of it around here from a few years back.

[ratdude747]

Nice... its a supermicro too...

I have a suggestion for the old innards of the frankenserver: put them in that massive server case you recently cleared out (where my X5DAL came from) and make another system...

[gg1978]

Nice board. Don't forget ya need FB-DIMM's for it though, i had to order some for the Dell Server at work once.

[Topcat]

Scored a pair of these for it today.... $140 shipped for a pair....couldn't even find them that cheap on fleabay!

http://www.cpu-world.com/sspec/SL/SLANR.html

I'm a big fan of the "E" series (low power consumption) Xeons....good overclockers, run cooler, less stressful on the motherboard and PSU....just a little pricier...

[shovenose]

That's gonna be super fast!!

[Topcat]

Seems I've hit my first obstacle. Yes, this board does require fully buffered memory....I knew that already....but with the CPU's @ 1600MHz FSB, to run the memory full speed, I need FB/ECC PC6400 (800MHz) DDR2....good luck finding that!! It seems to be nearly non-existent...and when I do find it, bend over!! I can get DDR2 FB/ECC PC5300 (667MHz) for next to nothing (like 16gb of it for $100 bucks).... The CPU's will of course run at full speed...but the RAM won't unless I force it (BIOS actually has an option for that)....so this should be interesting.

[mariushm]

http://cpubenchmark.net/cpu_list.php

140$ [Dual CPU] Intel Xeon E5472 @ 3.00GHz 9561 Passmark points

250$ AMD FX-8150 Eight-Core 8243 Passmark points
320$ Intel Core i7-2600K @ 3.40GHz 9092 Passmark points

Hmmm... even with the free motherboard, is it really worth anymore?

The 2 cpus and the fbdimm memory will eat loads of power - won't you get charged extra by the datacenter for it?

Something like a FX8150 + $60-80 mb + cheapo ddr3 seems more reasonable, and you'd get almost the same performance.

Well now since you already bought the cpus you're already committed i guess...

[Topcat]

Originally Posted by mariushm

http://cpubenchmark.net/cpu_list.php

140$ [Dual CPU] Intel Xeon E5472 @ 3.00GHz 9561 Passmark points

250$ AMD FX-8150 Eight-Core 8243 Passmark points
320$ Intel Core i7-2600K @ 3.40GHz 9092 Passmark points

Hmmm... even with the free motherboard, is it really worth anymore?

The 2 cpus and the fbdimm memory will eat loads of power - won't you get charged extra by the datacenter for it?

Something like a FX8150 + $60-80 mb + cheapo ddr3 seems more reasonable, and you'd get almost the same performance.

Well now since you already bought the cpus you're already committed i guess...

Ohh absolutely. I already have the case and all the extras. At the end of teh day, I'll have less in it this entire system core (CPU's, motherboard, RAM) the price of the I7 CPU alone @ $320 in your above statement....and this thing outscores it according to your info source....That's a pretty nice deal to me.... By the time I bought a good motherboard and all the supporting hardware for the I7, I'll be out a hell of a lot more than $320! These are E series....they consume about the same as the I7....of course that's x2, since there's a pair, so combined, its 160W at full load.... I can't believe you even asked this... Supermicro (bulletproof motherboard) compared to some 80-dollar cheapo? Come on now!! In 5 years, this thing will still be cranking along and not quitting anytime soon! A cheapo board in 5yrs will probably be dead and gone....or in need of a recap...heh...I wonder if I know anyone who could do that!

If I were worried about paying my electric bill, I wouldn't be buying/building this kind of stuff...

[Uranium-235]

yeah and completely different uses. server vs gaming desktop

[Topcat]

^
Indeed. I dont game. I build high end workstations and servers.....I dont build gaming rigs.

FWIW, my main rig was built Feb 2009. To build the exact same rig today would cost me 500 bucks more than it cost me back then. The processors went up in price (e5520 nehalems), the HDD's went up, and the barebones went up... the only piece that dropped was the memory, and it wasn't by much.

[mariushm]

In the past, it may have been a big difference between server motherboards and regular desktop systems. Server grade hardware was more reliable, it came with expensive stuff integrated such as SCSI controllers, had better heatsinks because the manufacturer couldn't rely on the datacenter to keep the rack temperature low...

Nowadays, even the gaming or regular cheap boards come with polymer caps, the have good enough heatsinks because most people overclock their systems, the boards have good SATA 6 gbps controllers integrated and video... basically the only benefit server motherboards have nowadays is support for ECC RAM and maybe SAS. Even ECC RAM is not that special, as some AMD chipsets support it on regular motherboards.


If you were to build the server now from scratch, it would not be worth it.
1. You'd have to get a computer case that supports EATX or whatever format the motherboard has.
2. Processors would be much more expensive than the 140$ you paid for the 2 cpus now.
3. You'd have to buy cpu coolers as they don't normally come with the processors.
4. You'd need a special power supply that would have TWO x 8 pin cpu cables - so that's a 650+ w power supply.

In the end, you'd have an expensive system that usually consumes more than 1A of power, which is the usual limit for a 1-2U server in a datacenter.

For the same price you'd pay if all parts were new, you'd probably be able to make two systems with the AMD FX8150 for example, and keep the second as spare machine or have the second as a failover.

Even with one system, you'd have 90% of the performance and about 95% of the reliability of your normal system, yet with about 60% of the power consumption, which would normally save you money in the datacenter if you were to colocate it.

There are hosting companies that do work with this kind of systems - Hetzner in Germany for example uses thousands of such systems and they found them just as reliable as Supermicro and Dell servers... see http://www.hetzner.de/en/hosting/pro...oduktmatrix-ex
... and they're using these boards for various offerings:

All EQ and EX models (MSI X58 Pro-E, Asus P8H67-M Pro, Asus P8BWS)

DS Servers with the following motherboards:

MSI K9AG Neo2 (MS-7368)
MSI K8T Neo2 - F (MS-7032)
MSI K8MM3-V (MS-7181)
MSI KA780G (MS-7551)
MSI K9NBPM2-FID (MS-7252)

And let's be honest here... even if the motherboard would die in 3 years, it's a 60-100$ part, and the server is hosting a forum and maybe some other things. You won't lose millions if the forum will be down for the time you can go to a retail store to buy a motherboard and replace or order one overnight from Newegg. But you'll definitely have a harder time fixing this server if the Supermicro board dies.

I didn't criticize that you do this build, because i realize having the motherboard free changes things. You're also lucky that you found those processors for a bargain... I even agree it's okay to spend some money and be at a loss, just because you see this as a project and a fun/interesting thing to do.

[Topcat]

Originally Posted by mariushm

And let's be honest here... even if the motherboard would die in 3 years, it's a 60-100$ part, and the server is hosting a forum and maybe some other things. You won't lose millions if the forum will be down for the time you can go to a retail store to buy a motherboard and replace or order one overnight from Newegg. But you'll definitely have a harder time fixing this server if the Supermicro board dies.

I'm an owner in the datacenter I have my servers in....so thats not an issue anyway. I dont put server application/loads on gaming type systems, that is always a disaster....we've done a few colo's like that...they never end well. I am 600 miles away from my datacenter....so I'm not gambling that some gaming board is going to remain stable/stay working. I don't cheap out on server hardware, there's just too much riding on it. It's a LOT easier to migrate the data from a functioning old server to a freshly built new server (can do it all online).....but disaster recovery on some crap gaming type board that failed?! I'll pass. Would I lose money if the forum goes down, no.....but when the store goes down with it, my cash flow is completely stopped. That sir, is not an option....and why my servers are never built on low-end gamer boards....I wouldn't even build a server from a 'high-end' gaming board, as even that wouldn't equal the stability and reliability of a true server board.

If that Supermicro dies before its 15th birthday, I'd be amazed. I've NEVER taken a supermicro serverboard out of service because it was malfunctioning....I've always replaced them just because they're getting old and slow....and this one is a long way from that.

Originally Posted by mariushm

I didn't criticize that you do this build, because i realize having the motherboard free changes things. You're also lucky that you found those processors for a bargain... I even agree it's okay to spend some money and be at a loss, just because you see this as a project and a fun/interesting thing to do.

Be at a loss? How ya figure? I could sell the thing for much more than I have in it.... You completely lost me now.... Hell, even on Ebay, this system would bring atleast a grand.....I won't have half that in it.... I actually have a mission for this....but you're completely confusing me.... Who buys/builds IT grade computers for use and plans on making money on the tail end?! When I take a server out of service, I usually GIVE them away....because they've made me a fortune while they were in service, which is what they're supposed to do... The server that runs this site has made me an untold fortune in revenue, but the hardware is essentially worthless today....the one I'm building will do the same.... Enterprise systems don't value out the way a home/gaming system does....so your comparisons just aren't valid, sorry.

[ratdude747]

Originally Posted by Topcat

When I take a server out of service, I usually GIVE them away....because they've made me a fortune while they were in service, which is what they're supposed to do... The server that runs this site has made me an untold fortune in revenue, but the hardware is essentially worthless today....the one I'm building will do the same.... Enterprise systems don't value out the way a home/gaming system does....so your comparisons just aren't valid, sorry.

I can confirm that with the Mobo in my main rig (A supermicro x5dal-TG2, used to be the BCN server at one point). Thanks Topcat!

[Uranium-235]

Originally Posted by mariushm

Even ECC RAM is not that special, as some AMD chipsets support it on regular motherboards.

its the am2/3 processors themselfs that support ECC, but only asus boards have the option to enable it

One of the main things that server boards i've seen that gives advantages over regular boards is the onboard management software. Turn one off/on flash bios and some will let you even kvm it remotely.

aside from that, I put the hardware quality up there for some of them with gigabyte, some, like supermicro are even better

[mariushm]

I've built computers with server grade components, if you want I can show you the manuals and various crap i got with a 2 socket Tyan motherboard I bought.

The part you don't understand is because you're looking at everything from the perspective of someone that is owner at a datacenter, that already has a server or more there and doesn't have to care about electricity charges, about renting a single unit or a half rack or a full rack, that doesn't factor in downtime, that won't pay extra to have the server hosted there because you're just replacing the old one or you're owner so you don't care...

Look at it from the perspective of a regular person that wants to make a system and have it colocated somewhere and do the math

- how much will the parts for this system you make now would cost the person if he were to buy them from the store now?
- how much will he pay the datacenter for the rack space
- how much will he pay the datacenter for the 2A of electricity
- what happens if the server dies, what backup plans do you have

A regular user can probably make this system for... let's say $1500 ($300 case and psu + $400 mb + $400 cpus (2) + $ 100 cpu fans + $200 ram + $100 hdd) and gets a 1U system using 1.5-2A of power.

He goes to a datacenter that charges his $25 for 1U but charges $50 a month for 1A of power, so he pays $125 a month just to colocate the server there.
At the end of the year, he's out of $3000 (125$ x 12 + $1500).

What if the server has a failure for some reason? Tough, he just colocates, he doesn't have spare parts, the datacenter is 600 miles away.

Now what if he was to build a system from regular computer parts?

A 2-3U rackable case is 100$, the psu is as low as 60$ but let's say $100 (brand name quality 500w one), mb is 100$, cpu is $250, ram is $100, hdd is $100 ... the total cost is $750.
The system has 90% of the previous server grade hardware yet user only pays 50$ for 2U and 50$ for 1A of power so he pays 100$ a month (for the 2u case).

At the end of the year, he's out of 1950$ (750$ + 12x100)

With the savings, he can very easily build an identical system with the first one and pay the datacenter just 50$ extra a month to keep the second server in the rack, without any power.

If there's any failure, the user can just pay the 100$ for one hour of remote hands and tell the engineer to move the hard drives and the power cable over and when he has time, he can go to the datacenter to fix or replace the broken server.

It's not just the motherboard you're relying on, you're also relying on memory modules not going bad all of the sudden, you're also relying on the hard drive to keep spinning and not die suddenly after a few months...

To me, the second case looks much better, if I'm on a limited budget or I'm just starting my business and I'm not sure about how much income the business is gonna bring me. I'd be reluctant to rely on a single server for my business and since the datacenter is so far away, especially since you say it made you an untold fortune in revenue (though if this old one you have now did that, I would have thought you'd be eager to replace it with a better one sooner)
Not to mention that you wouldn't have just one server or all servers in one location if you make so much money out of this and it's important to be online.

When I said you're at a loss, I was thinking of how much money you throw to build this compared to how much money you'd pay usually just to lease a supermicro or a dell server directly from the manufacturers . Again, it's one point of failure while by leasing the server you'd get a year or so on site replacements and all that - it can be a much better investment of your money.

[Topcat]

^
I still disagree with that. I've had an occasional PSU or HDD failure....that's just the nature of the beast.... There are people on hand capable of chaning out a hot-swap HDD tray or PSU module. In 14 years, I've never had an in-service (live server) motherboard failure or memory failure (outside of my VP6's with bad caps in the dark ages, thankfully those were local). I had a backup system crap out on me once, was an Intel SAI2 motherboard (dual socket 603 Xeon) that just went tits up one day after 8 years of running 24/7... Swapped out with a spare on the shelf, server was live again in 20 minutes. I did a quick bench check of the board afterward, it wouldn't POST anymore...I stripped it and tossed it in the trash....it wasn't worth another minute's worth of my time.

I dont think you realize what kind of revenue these servers generate. $3000 bucks is just a petty tax writeoff in retrospect. For example, the X5DAL-TG2 that I gave to Ratdude....let me tell you the story behind those. I had 3 of them. 2 fully assembled and live in servers, the 3rd was new in the box as a hotspare....just incase. As its time to begin phasing them out, I gave one to Ratdude, the hotspare I tossed into a case and gave it to my stepdaughter, and the last one is actually still in service....you're posting to it now. All those boards were bought back in 03~04, when they were new.... I bargain hunt yes (I wait a year or so from their launch for prices to drop, etc)....but all were of top-end parts, I had probably 4000 in them all, give or take a few hundred. In their SEVEN years of FLAWLESS service, they have generated probably close to 7-figures in revenue....probably more with all the side stuff. See where I'm going with this? At the end of the day, the hardware is worthless, and the cost of buying it was very trivial. I give them away to keep them from either sitting on the junk shelf or becoming landfill. Ratdude was able to make good use of it, and I was happy to see that become of it. Quality aint cheap my friend... yes, I could have built cheaper machines.....but I can guarantee with a fair degree of confidence that they would not have given the same top-grade service that the intended hardware has given.

I have been on your side of the aisle.....I've had to build and maintain servers on a very tight budget before....but that did not mean I skimped on parts. I would simply work it so I was able to buy the high-end hardware....thinking back, there are times I think of and wonder how I ever pulled it off....I was THAT broke!! ..but I'm just one of those that won't compromise quality to save a couple hundred bucks....it will get you in the end....and in the case of this particular machine, it wouldn't be any cheaper to build the cheaper grade system....and it certainly won't perform better...

I'm not saying your method is wrong....I just see things differently and built my servers for the long-haul.... If your way works for you, by all means!!! ...and I think that's great that you found a method that you're happy with.