I always have a devil of a time figuring out what point in time is optimal for upgrading my software test-bed equipment. It seems every time I do it, a new platform or a new set of standards that vastly improve productivity emerges two weeks later. It’s exasperating and it’s only getting worse as hardware improvements seem to be coming faster and faster. Add to that software version releases coming practically on top of each other and things are getting out of control. I think we all face this problem and we need a plan to know when the time is right. If you’re managing a whole slew of desktops and a render farm, your need for good timing is magnified greatly. (PHOTO CAPTION: A shot taken as Plantec begins to build his newest, fastest content creation software test bed.)

I have a plan

If you’re in a production environment, it surely involves a pipeline. Most likely you have to do a lot of rendering of one sort or another. Rendering is always resource intensive. The other thing you probably have to deal with is expensive manpower. Top animators and VFX people are usually very expensive and the more expensive they are, the bigger impact a suboptimal pipeline becomes.

On the surface it looks like buying new equipment or upgrading what you have is just an extra expense that you’d rather avoid. But when you think about it a slow pipeline hemorrhages money almost as fast as congress. Many of the studios I’m familiar with pay upwards of one hundred grand a year for their top animators. It doesn’t take advanced math to realize that any clog in productivity is going to be massively expensive. You need your people working efficiently and not waiting around for renders. I don’t mean you’re a slave driver, but a lot of downtime is becoming ever more unnecessary.

Having a fast, powerful render farm goes without saying; and that’s a subject for a different day. Today I’m thinking about your individual work stations. You need to have the best workstations you can get and it almost doesn’t matter what they cost. You need dual widescreen real estate for efficient work flow and you need really, really fast computation and GPU accelerated simulations and video.

You say, new equipment is very expensive and it’s not in your budget. Well it should be. You should always have enough built into your overhead to regularly upgrade equipment. How often? That’s part of the plan. The answer depends on how much you pay your people. The more you pay, the more critical machine speed becomes.

Let’s meet George

George is a VFX animator whose specialty is fully photorealistic rendered environments. His work is crazy computationally intensive with atmospheres, sophisticated lighting, massive textures and millions and millions of polygons. George’s equipment was brand new state of the art a little more than a year ago. It was all purchased from a reliable dealer as a package along with 30 other workstations for co-workers. Everybody’s on the same equipment page and six months ago everybody went from 2 to 4 gigs of ram. That speeded things up a bit. It was a smart move.

Nevertheless, you find George wastes at least an hour and a half a day waiting around for sample frames to render on his desktop. He likes to pull renders often as part of his perfectionist workflow. He’s a brilliant artist and you don’t want to mess with his work style.

His animated sequences take copious amounts of CPU juice, so must be sent off to the farm for all night render sessions and he often re-renders them. Every day, George has downtime because he needs those “quick” renders to see how things are looking. Like most artists, he hates waiting. To save time he only renders a portion of critical frames.

Draft mode won’t do, because George needs to see exactly how the lighting is interacting with the atmosphere, landscapes geometry and textures. Sure, he slims down the renders as much as he can, but they still take about 20-35 minutes each frame. Unfortunately the OpenGL previews just are not good enough, but when it runs at a decent frame rate, OpenGL speeds up his scene construction immensely. When things bog down and his interface turns to jerky representational boxes. George’s productivity drops as well. Oh, and you can’t have him sending all his ‘quickie” renders to the big farm because it’s usually maxed out pumping through final shots for your client.

Let’s also assume…(a bit of a joke) that your studio is set up to work 5 days a week with a skeleton staff on the weekends to shepherd the render-farm that never sleeps. In five days, George is non-productive for only about 7.5 hours a week…if you’re lucky. Figuring overhead at 50% which is optimistically low, we can estimate that George’s down time represents nearly $750.00 in lost productivity per week. He gets two weeks of vacation, so we figure you have him 50 weeks a year for about $38,000 in lost productivity just waiting for renders to complete.

He also looses additional time because his display card is not GPU accelerated so textured geometry turns to white boxes a lot. And more time is lost because he only has one 20” monitor and things go faster with more real estate…a pair of 24” widescreen monitors would be good. Let’s modestly estimate that this brings lost productivity up to about $40,000.00 per year. That’s not including his complex software being hungry for more clock cycles (I’m thinking Vue7 Extreme here, perhaps being used inside 3ds Max.) I noticed when I upgraded from Vue6 to Vue7 Extreme; the vastly better but more complex renders brought my reasonably up to date system to its knees. I hadn’t upgraded in more than two years. That is about to change.
All this lost productivity was normal and necessary when you bought the current equipment. It was state of the art at the time and blazing fast. Upping to 4 gigs of ram helped. But are you still running 32 bit software on your 64 bit equipment because you don’t trust Vista 64? Some of you surely are. Thank the gods that Window’s 7 is looking good; and will be out by the end of the year. Vista 64 on the proper rig with plenty of memory, and the right applications can really scream. 64 bit versions use memory more effectively to reduce I/O, especially on massive jobs like George churns out.

The Rate of Upgrades is Accelerating

It’s more than a year since you set up your pipeline. Moore’s law has been chugging away. And right now things are changing very fast and that’s just going to get worse. Both Intel and AMD have been battling it out. AMD offers the most cost effective processors and Intel offers the fastest, most powerful ones, but they’re not that far apart. Sophisticated new CPU, Bridge and GPU cooling solutions allow chips to safely and reliably run at faster, more productive speeds. Add to that Seagate’s new blindingly fast Savvio 15K.2 SAS drives that are so reliable that ultrafast Raid 0 arrays make sense. They have a 6 GB/s bandwidth. That means you can stack them up to create breathtakingly fast desktop RAIDs. They may seem expensive, but the cost is in the noise. You can spend less than two grand to set up a first tier 4 drive zero array for each artist. You’ll need a second tier for media storage and a good back-up system. Each of those will run you under a hundred bucks if you want to go cheap. Even if you go with enterprise drives with half a million hours MTBF you won’t break the bank these days.

Let’s talk Intel’s new i7 CPUs. They represent a quantum leap in design and performance. With 8 virtual cores and vastly speeded up EX-58 support architecture from Intel; you’re looking at another quantum leap in productivity. Add to that, the relative ease with which they can be over-clocked and you have some serious performance options. They require new motherboards and there are some really good ones. I picked the “Ultra Durable” EX-58 Extreme board from Gigabye to experiment with. It’s loaded with clever over clocking tools. This is important because over clocking the i7 series is very different from previous CPU series from Intel. Gigabyte provides some easy automatic solutions with their motherboard. I’ll find out how good it is when I build my testbed computer next week.

By the way, Intel’s official position is that you should run the CPUs at their rated speed, but if you want to over clock, go for it at your own risk. Just watch the heat buildup on the CPU and associated support chips and be aware of the risks. They warn that reliability and stability will be sacrificed if you turbo up too high. Dan Snyder, my friend at Intel, asked me tell you that Intel thinks you should run at rated speeds in order to assure reliable operation on your pipeline.

It’s good advice, and for the most part I agree. But I’m finding that a little over clocking is possible with long term stability, especially if you invest in a good liquid cooling system. I did. I’m building my test-bed workstation and cooling it with the CoolIt Freezone Elite, which uses a unique hybrid liquid and TEC (TheromoElectric Cooling) solution that is sweet to say the least. You can use air cooling, but not the stock fins that come with the CPU if you’re going to overclock.. The Thermoright ultra 120 is your best bet there. It’s much cheaper and is almost as good as ordinary liquid cooling systems. The Elite is a monster, but it does fit inside after market boxes that have 120mm rear fans. It looks like part of an alien space ship.
I decided to go for the best, most reliable power supply I could find; and it can handle whatever GPU Accelerated display card (s) I decide on, including triple SLI and Crossfire. It’s the magnificent Corsair HX1000W. I’ve never thought of a power supply as beautiful, but this one is. It comes packed in reusable foam shell and all the cables come packed in their own cloth logo-bag with Velcro closure. All, just a sign of the loving care with which Corsair components are made.

I also went for 6 gigs of Corsair Dominator super fast memory with their Airflow dual cooling fan setup. The reason for 6 gigs is that the Gigabyte mobo loves to have at least 3 sticks of memory so it can run superfast 3 channel memory access. This way I’ll have three two gig channels. Intel’s new Nehalim bus architecture does away with the old and relatively slow Front Side Bus (FSB), to communicate with memory. They’ve replaced the FSB with QPI or Quick Path Interconnect which is far more clever and efficient. Intel probably wouldn’t admit it, but I think they were influenced by AMD’s Athalon 64 design in which they moved the memory controller to the CPU chip. The key advantage is direct CPU access to the memory. This opens up a much higher bandwidth than we could ever see with the old FSB model. This combined with new triple channel DDR3 memory architecture, and increased cache sizes speeds up content creation applications significantly; or so I’m told. The new X58 chipset is completely new design with native support for multiple SLI and Crossfire display configurations.

The Gigabyte mobo handles NVIDIA triple SLI video accelerated setups as well as Crossfire. But I’ve asked Nvida to let me try out a single Quadro CX card that is optimize not only for 3D content creation, but also for Adobe Creative Suite applications. I use Photoshop and After Effects and Sound Board; so this will be interesting. The Quadro CX features a 1.5 GB frame buffer and has 192 stream processors online, making it a true Tesla gpgpu mini super-computer as well as a Tesla Graphics display card. I’ll write more about this amazing card in my next column, in which I’ll actually build and tell you about that new really fast software testbed. Later I intend to try out various software packages on this rig to see if my productivity really is increased.

Speaking of Supercomputers

If your company does a lot simulation vfx such as fluid dynamics, complex physics etc that are math intensive, you might want to look into Nvidia’s Tesla/CUDA system. It’s a card system that plugs right into the Gigabyte motherboard (and others,) giving you up to 4 teraflops of pure number crunching power. I understand Cray is using them. Each card looks like one of their typical display cards, but it’s not. There are no display outputs. Its sole purpose is the offloading of computationally intensive tasks from your CPU.

Not only do you get supercomputer bragging rights, but one setup can blow the doors off your competition. The only drawback is that you’ll have to adapt your proprietary code to their CUDA parallel computing architecture. But one Tesla card will give you performance approximately 250 times that of a normal workstation. The Tesla C1060 with nearly a teraflop of throughput, costs less than $1800.00 at Tiger Direct. Conversion to CUDA will take some doing, but whatever it costs, the savings in high end manpower is likely to be well worth the investment. This could be the secret weapon that puts you into the big time.

Time to build a new upgrade path

Hopefully I’m being clear that with the drop in hardware prices and vast increase in performance you need to consider instituting a regular hardware upgrade schedule. Just keeping on top of the latest developments has become a complex job. For most small studios in the 20-30 person range and beyond, it would probably pay to have a fulltime person whose job is just to stay on top of the technology and suggest upgrade pathways on a regular basis. Then this person should be in charge of working weekends to keep individual workstations screaming as well as upgrading the big render farm. You don’t want your workstation upgraded during the week. A good plan is to have a store of match computers that you upgrade in the shop. You then take a few minutes to swap out hard drives with George for a quick workstation swap. The fifteen minutes lost, will be gained back quickly as George enjoys his snappy new work interface.

With such a person on staff, you can institute a continuous upgrade program in which your tech guy sets up criteria for judging when a part of each workstation has evolved far enough to make a difference. Then he should begin upgrading those particular parts.
Time to become a farmer

This brings me to another point. If you are renting a render farm, perhaps it’s time to build your own. If you’re not going super-expensive professionally built enterprise render farm yet, you can build your entire pipeline on a shoestring these days. I picked the Gigabyte mobo because it’s supposedly built to enterprise standards with heavy copper power layers, solid state capacitors and more. We shall see.

I’d even suggest a complete hardware assessment every 6-12 months. For example early next year we’ll see multiprocessor mobos for the i7 chips coming out and those chips will have improved specs. Intel tells me that Moore’s law saying computer component density doubles every 18 months is still in effect. But computers are not getting faster (they’ve actually run into negative quantum effects already) but they are getting more productive through increased numbers of cores and faster support architecture and ever smaller components. For example, the Intel i7 series is at 45 nm component size and by the end of this year we may see the first 32 nm processors. Reportedly Intel has invested about 7 billion in three plants to produce these new chips. In fact, by first quarter 2010 Intel expects to release a new 32 nm CPU architecture that will feature integrated GPUs. Will they be as good as the Nvidia and ATI gpu’s? That remains to be seen. But it’s another reason to evaluate the technology.

AMD is not far behind. Thank goodness for this battle of the CPU giants. I want them both to thrive because they drive each other to great heights. AMD has spun off “The Foundry Company” to produce their 32 nm processors and I suspect they’ll start shipping about mid 2010.
The whole way we do computing is changing rapidly. Programming languages and procedures will have to adapt to the new hardware and processing options coming online. Those VFX and animation houses like R&H who use a lot of in-house software will have a huge leg up if they’re smart and steadily adapt their proprietary applications to the newest, fastest hardware. I happen to know, the folks at R&H are smart and are on top of things.

Don’t upgrade everything!    

Your hardware doesn’t age or become obsolete at the same rate. If you do it right, core parts like power supplies and cases and cooling systems are really well made and not so subject to systemic design change. Clearly some pipeline parts will outlast others in functionality. You could go with cheap, fast whole systems with expensive GPU display cards, and then replace the entire system (save the display cards) regularly. But I’m not recommending it.

A good technical person will be able to design a cost effective maintenance and upgrade plan based on what components have most impact on the throughput of each workstation. That would be the speed sensitive bits. Consider doing your own in-house builds and upgrades on a regular basis. Which route you take depends on the size of your operation, but in many cases it will pay you to have one or more hardware people upgrading workstations.

This way you can invest in really solid core support equipment and just change out the CPU, Display card, memory, and mobos regularly. For example power supply designs change very slowly, and the exquisite ones made by Corsair are things of beauty; built with pride from no-skimp components they’re rated to last 100,000 MBTF. If you don’t know what MBTF is, you need to. One excellent power supply can last through a half dozen or more mobo upgrades. Put that supply in a solid case with excellent ventilation and you have another component that will last for several generations. Another place to invest your money that will last through several mobos is efficient liquid CPU cooling. It keeps things stable and allows you to over-clock, increasing efficiency if don’t mind the risks involved. There are many CPU cooling solutions from air heat exchangers to the more efficient liquid cooling systems such as those supplied by CoolIt.

The CoolIt company offers several innovative and cost effective hybrid solutions that use both a liquid cooling system and Pelletier/TEC (Thermo Electric Cooling) blocks which actually can refrigerate the cooling water well below ambient during idle. I find their liquid cooling solutions to be innovative, nicely designed and solidly built. Featuring zero maintenance, they are very reasonably priced compared to the competition. CoolIt gets the heat outside the case which the passive finned air coolers can’t do. But passive fins don’t fail very often…complex liquid systems can. CoolIt active cooling systems tend not to fail. Controlling processor heat is the secret to getting max performance in over-clocking systems which run very hot due to increased power dissipation. More on this next time.

It’s a matter of paying attention

It’s not a matter of me telling you what to do so much as me beating you on the head while you’re working your ass off and saying: “Take the time to look at this stuff so you can make your company more cost effective and competitive.” This economy demands it. We all get so caught up in rush jobs, mini-crises, and company politics that we forget to use our heads. Sometimes we sweat the small stuff and ignore the big glaring black hole sucking up our resources.

You know your studio best. You know what you do and how much you pay your artists and what your deadlines are. Oh…wait, which brings up another little side effect of having super-fast hardware: deadlines — making them. Novelty? I think not. You can vastly enhance your competitiveness in the market when you have a reputation for quality on-time delivery. If you’re structured appropriately; you could even come in on-time and under budget. Okay…sorry, I didn’t mean to joke about such a thing. To wrap, pay attention and use your head to get ahead.

Next month I hope to bring you the results of my building my new content creation software test-bed, with what I consider to be the fastest, most cost effective components. It should be interesting.