CP 950 Power Off

[Marcel Birgelen]

Crash-stopping a Linux computer is generally a bad idea as well.

Which means that a well-designed unit should always have a shutdown button or procedure.

How do they say it? It's complicated. While for a normal Linux workstation or server, yes, it's generally true that simply killing power to those systems is usually a bad idea due to data consistency. Older file systems would often greet you with a mandatory filesystem check on next boot. But Linux is everywhere those days, including embedded systems, ranging from mobile phones to routers, switches, watches, washing machines, etc.

Special file systems like SquashFS can be used to build robust systems that will survive a random power outage. It's obviously not just up to the operating system, the software running on top of it also needs to be able to handle such things.

About 10 to 20 years back, when stuff like the DSS-line and the CP850 were designed, Dolby's design philosophy apparently was to base as much of their designs on off-the-shelf server hardware, with Supermicro being their prime OEM partner. Somewhere along the road they must have decided to go back to more integrated designs. Their current IMS system and the CP950 are a good example. While the CP950 chassis seems to be based on a common backplane design, there is no off-the-shelf motherboard in there, for example.

[Jim Cassedy]

During one of my screenings at Dolby's SF HQ, I once asked one of the Dolbyguys specifically
about the lack of a power switch on some of their newer products, I and mentioned the CP-950
as an example. The response I got was a quizzical look and a reply something like "Why would
anybody want to turn it off?" . . . .

[Martin McCaffery]

Because most of the time I'm not using it.

[Jim Cassedy]

Hey, Martin! I'm on your side. . I think there outta be a law or something that everything has a
power switch. This is a free country. . we should be able to turn something off if we want to.

[Frank Cox]

I've given some thought to why so much really expensive professional cinema equipment seems to have such shoddy software and I've come to the conclusion (or at least the hypothesis) that it's because this kind of work isn't sexy enough to attract the really good people who work to build something because they want to make it beautiful. It's being done by the what's on today's list and when's the next coffee break due crowd.

"If it compiles, ship it."

You might get some of the real artists working on software for the production side but exhibition? Decoding an audio block or a jpeg or writing a playlist is up there with implementing a ruleset for this year's tax tables...

[Leo Enticknap]

A warning sign of a flat battery is the server no longer just boots up on its own (and it complains about what day it is).

And if that server doesn't have a proper monitor, keyboard, and mouse connected to it, you're snookered. Many's the time I've been asked to fix a server with a Supermicro motherboard that boots up to "System settings have changed: press F1 to continue or DEL to enter setup." If there is no keyboard connected (e.g. if the server is only operated via VNC or a TMS), you're stuck there. With a DSS200/cat862 system you have the additional problem that if you boot it up without resetting the BIOS clock, the mismatch between the BIOS time and the secure clock will cause the cat862 to refuse to accept the connection from the server.

[Marco Giustini]

the response i got was a quizzical look and a reply something like....

....that is what we forgot to add!!!!

[Steve Guttag]

Nah...I guarantee you that there is a very lopsided ratio of switch failure to software corruption in a product like a sound processor. Switches, in general, are some of the least reliable stuff and made only more so since RoHS rules came out (the elimination of cadmium really hurt switches/relays). While you could, via software, turn something off...you'd need to have a means to turn it back on and that would be a hardware connection or have a WOL type thing...meaning it isn't self-sufficient. It can do its own reboots. But, when it has power, it boots up and runs. 99% of its use-case is going to want that.

Those that want to turn it off are fee to do so (all ,or most all, of our customers do). You can use a power strip, power sequencer, if you want to be fancy, or just use the breakers (the traditional method) or even a contactor on the sound rack if you are afraid of using breakers as switches.

There are switch rated breakers but even for those that are not...the arguments against using breakers as switches doesn't hold. First, nobody is switching them at full load. Second, you could buy a whole stack of breakers from your local home center or electrical supply house for less money than a few of the specialty power switches in each component. I'd rather have 1 thing fail that costs $10 or so than 10 things that cost $50 or more plus the time to change them. The realities are, the number of breaker failures I've see due to use as switches are VERY low. Normally, it is the cheaper brands and even then it is after more than a decade of use.

Conversely, I've had to troubleshoot more calls than one would expect for a switch being off...that level-1 stuff that has someone asking "Did you turn it on?, is it plugged in?...etc." Who has time for that sort of call? If it just came on with the power...the call would be eliminated.

Conversely, please show the number of Dolby sound processors that have failed due to the lack of a power switch, and more importantly to this discussion, the lack of a power down sequence that puts things to bed nicely.

[Leo Enticknap]

Second, you could buy a whole stack of breakers from your local home center or electrical supply house for less money than a few of the specialty power switches in each component.

What would make me leery of that approach is that if you attempt to replace an individual breaker in a cabinet without cutting the power to that cabinet from somewhere upstream of it (which might be either practically impossible, or only possible if you're wiling to black out a significant chunk of the building) and you don't know precisely what you're doing, you risk ending up like Ted Bundy (i.e. electrocuted). But if you have a licensed electrician do it, that's pricey. Therefore a booth workflow that involves the requirement to replace breakers, even if only once every few years, is not something I'd want to design in to any system.

I agree with both Frank that cutting the power to any computer while it is running is to be avoided if possible, due to the risk of operating system drive corruption, and with you that I have never experienced this happening in practice in a CP750, 850, or 950. I have had reports of this happening with DSS servers quite a few times (the symptom being that it just presents a GRUB prompt when it's powered on again, and the cure being a from scratch reinstallation of the server software), which is why I advise a shutdown procedure consisting of ctrl-alt-F1 followed by ctrl-alt-del, and then cutting power to the device when the fan surge is heard; not just cutting the power while the thing is fully up and running.

[Marcel Birgelen]

Conversely, please show the number of Dolby sound processors that have failed due to the lack of a power switch, and more importantly to this discussion, the lack of a power down sequence that puts things to bed nicely.

Finding hard numbers for this will be close to impossible, at least with a high enough sample rate to make sense. I guess the CP850 is an outlier due to its design being based on a SuperMicro-style server. I wouldn't want to power-cycle that machine every day. Likewise, if I'd still had a CP650 or CP750 running, I'd probably consider letting it run 24/7, especially the CP750 with its flaky motherboard and history of PSU issues. The same for older server machines, especially those with hard drives instead of SSDs.

It's always hard to do a correct judgement on what's better for equipment: Letting it run and let it accrue "working hours" or incur the extra power cycles, which often comes with their own wear and tear on the equipment. The amount of moving parts inside the equipment like fans and hard drives might also be a factor in this.

As for using breakers as switches: U.S. breakers are different from European breakers, but I'm not a fan of abusing breakers as regular switches, as they're usually not designed for that purpose and extensive arcing due to high inrush currents will wear them out pretty quickly. Yeah, you're usually not switching at full load, so arcing should be at a minimum, but a heavy duty 2 or 3-phase switch for 20 to 40A will cost you an equivalent of like $60 to $100 around here, not really a substantial cost given the whole picture.

[Steve Guttag]

As to running the CP750 24/7, what accelerates its motherboard demise is heat. The longer you run it, the hotter it gets, the greater the likelihood of it fading out or failure. I suspect that had they put a fan in it to move the air...the failure rate of both its power supplies and its motherboard would have significantly dropped. The fact that there is a fan connector footprint on the MB shows that they were considering it.

Cycling power switches on individual equipment and their failure would FAR FAR exceed the cost of repair than paying for an electrician to change out a breaker (no, I'm not advocating that unqualified people change out circuit breakers nor am I advocating unqualified people change out switches in equipment either). The failure rate of breakers is very, very low, particularly decent quality ones like Square-D QO series.

On the DSS200. We tend to power cycle them weekly and started doing so when some of their software versions showed a memory leak. The only ones that will come up with the GRUB screen are those with the Western Digital 2TB drives, as supplied by Dolby (I believe it was the WD2003FYYZ or WD2003FYYS...I forget which). If you change the drives you'll fix that too. Even if it comes up to the GRUB screen, often a CTL-ALT-DEL will get it to boot up as the problem seem to be from a cold start.

[Marcel Birgelen]

Not all breakers are created equal, I have little long-time experience with U.S. breakers. I'm not entirely sure, but according to local electrical code, I believe, you're not allowed to design a breaker to be used as a regular power switch. Local fire code also often requires you to have a central, well marked switch or switches that disables the power in the entire room. Practically all installations that have individual breakers installed also have a separate power switch. Also, many breakers, at least around here, are only rated up to e.g. 50 cycles.

If you want to individually switch equipment in a rack that cannot be controlled via "light-outs" management, then a remote power switch like the APC-line "intelligent" PDUs are an option, although not entirely cheap. The upside is that they allow you to automate the power distribution in your equipment racks and also allow you to monitor consumption.

As for the GRUB prompt hanging on the DSS200... This indicates a boot-failure caused by the RAID controller. The 3Ware MegaRAID controller in there has bee EOLed by 3Ware/LSI/Broadcom long ago, but has anybody checked if there may be a firmware upgrade that supports newer drives?

[Leo Enticknap]

I've experienced the GRUB prompt more often on a DSS220, that has a software RAID (no 3Ware card) more often than on 200s and 100s, and have never been able to fix it just by power cycling.

As to running the CP750 24/7, what accelerates its motherboard demise is heat.

Which is why, whenever I've replaced failed CP750s, and if it's logistically feasible without many hours of extra work (e,g. cables in the rack aren't long enough as found), I've relocated the audio processor as far down the rack as possible, and at the very least, below the power amps. It always puzzles me why the default way for installing equipment in an audio rack seems to be with the amplifiers at the bottom and everything else, including heat sensitive, computer-based equipment, being above them, so that the heat from the amplifiers can radiate upwards and cook the other equipment. My only guess is that the intention was to place equipment with LCD front panels and knobs that will be manually operated close to eye height, i.e. ergonomics takes precedence. In an arthouse or screening room there may be some sense to that, but in a multiplex in which the audio processor is being operated by automation cues 99.9% of the time, I'm putting it below the amplifiers wherever possible.

[Marcel Birgelen]

It's a bit of a tradition of how most flight-case-able PA racks are built: The heavy equipment, like power amps, goes into the bottom of the rack and the lighter, more modular equipment is installed in the top. One of the reasons is rack stability, having the heavy load on the bottom ensures those racks not to be top-heavy, which can be dangerous during transport.

Also, it's more likely that the equipment here requires more human manipulation, putting it higher up in the rack allows for easier access and more visibility of the indicators on this equipment. A power amp, after setup, seldomly needs any kind of adjustments.

[Steve Guttag]

I, absolutely, locate the human controls, including the processor where they are easiest to see/reach and let other equipment work around that. So, on wall-mount racks, of the processor was at the bottom and the amps were higher in elevation. For floor standing racks, they tend to be towards the top...typically around the 34U mark. And, depending on the application, there may be a sweet spot for sitting versus standing.

As Marcel noted, one reason for locating amps (typically the heaviest equipment) on the bottom is for stability (and ease of rack loading or servicing). We've used QSC amplifiers, pretty much exclusively in cinemas. They've always vented properly...rear-to-front. This pulls heat from the back of the rack and blows it into the room rather than convection up to the upper equipment. To that, we, typically, add a filtered intake blower to the bottom of the rack. This will ensure a more-dust-free source of air gets into the rack rather than turning the back of the amps (and other equipment) fuzzy with dirt. You'd be amazed at the difference pressurizing a rack, just a little, does. Fans at the top of the racks almost do more harm than good as they draw dirty air into the rack. If the side vents in the rack are growing whiskers...that is the work of the top fan.

Back onto the breakers. Codes in the US vary by the national code, which local authorities decide to adopt, in part or in whole, on their own time table. The local can be in the form of state, county and/or city. So, the builder is under the most restrictive all of the above. There are fancy panels like Lentec that have quality breakers (Square-D) but with switches built in for sequencing. We have also had 3-phase contactors installed ahead of a subpanel to the sound rack. We control the contactor via an automation (and also have them install a manual override). The contactor feeds a small load-center with the breakers for the sound rack. However, the VAST majority of the booths I've been in, they just use the breakers as switches. The number of breakers I've seen fail in over 40 years from this action is under a dozen and that is with countless breakers in the pool with many of them "low-bid" quality. Square-D and Cutler-Hammer are on the lowest-end of that (with respect to failures).