DSS220 Can't Bootup

[Leo Enticknap]

The flaw in that argument is exposed by considering what happens in the real world, in the field, if a DSS220 craps out during a show. Sure, in theory, you pull the NFG PSU out, then slide the replacement in, reboot the bugger, and you're back up. But the practice goes something like:

It's a busy Saturday night in your suburban 15-plex. You have the legal bare minimum front-of-house staffing in place. Eventually, a customer walks out of an auditorium, fights her way to the front of the concessions line, and tells the guy behind the counter that the screen has gone dark and the sound has stopped in Spiderman vs. Hello Kitty. Which of the three screens playing that movie would this be, he replies. Buggered if I know, says the customer; I just went where I was told.

About 5-6 minutes later, the duty manager establishes which screen is down, and runs up to the booth. There is a green tail light on the projector and everything in the rack looks OK, but it takes another 2-3 minutes to figure out that there are no lights or fan noise on the server. So the server is completely dead, but why?

If it were you or I, no power getting to the server's motherboard would be the first thing we'd suspect. It would take us under a minute from there to zero in on the power supply module, and then swap in the spare. But even then, around 15 minutes has gone by - with turnaround times, likely long enough to give us no choice but to cancel the show and issue refunds.

But this isn't you or I ... it's a 20-something valley girl who, 20 minutes previously, was on her break reading about which of her friends is dating which other on Facebook, before that customer came out and all hell broke loose. Mention "power supply unit" to her, and she'll think you're talking about the latest energy drink being promoted on Gwyneth Paltrow's website (at $462 for a 16oz bottle, in case you were wondering). No matter how quick the potential fix, she's going to issue those refunds and call a tech.

There were 300 customers in the house that went down, and they paid $15 each. She had to give out $4,500 in refunds.

Alternative scenario: there are double redundant PSUs in those DSS220s. One morning, the opening manager hears a bleeping sound coming from one of the projector pedestals during the opening rounds (the primary unit had failed the previous evening). She calls the tech, who moves the next scheduled PM visit up by a few days, and swaps out the dead module. Having two of those $120 modules in the server saved the theater from having to hand out thousands in refunds.

[Steve Guttag]

But that choice is up to the exhibitor. You all install Q-SYS...do you put in dual cores and dual networks on every installation, including 1-core per screen? Dolby found ways to make the DSS line less expensive to the multiplex exhibitor (3-drive RAID instead of 4, make the backup PSU optional). Nothing stopped the dealer from selling the backup PSU to each server. We sure did (after that first round). How about on the CP750s...did you all put in the backup power supply for each unit? We only supplied one per complex.

I will say, on our NOC installations, we always were informed, before the manager discovered it, a failed PSU or RAID drive.

[Leo Enticknap]

Manufacturers can decide whether to leave choices up to exhibitors or not. Pre-DSS220, Dolby did not: double redundant PSUs were standard equipment in the DSS line. When something is offered to a customer as being an option, we, as the vendor, are in the position of having to sell it, and customers are naturally skeptical of sales pitches. When I bought my Honda from the dealership, I was given the hard sell on having the paint job coated with a graffiti spray paint-resistant gunk for $700. I thought about it, considered that I'd never, ever, heard of any of my friends, relatives, or co-workers ever having had graffiti sprayed onto their cars (including some who park their cars on the street in some less than perfect parts of South LA), and said no thanks. That extra power supply on the DSS220 (and on the CP750 and 850) now becomes the subject of the same process. I've never lost a show because a server's PSU crapped out, therefore I don't need this.

IMHO, the amount of money involved is so small that Dolby made a strategic and reputational error in changing double redundant PSUs from being standard equipment to a paid for option. The CP650 should have shown them that a machine that converts high voltage, low current AC to low voltage, higher current DC has to deal with a lot of heat and stress (and heat cycling, if it is regularly powered down and then up again), and therefore will not, almost guaranteed, last for the service life of the unit as a whole. When that unit is mission critical to the system it works within (if the DCP server or audio processor goes out during a show, you have lost at least one show), double redundant PSUs should be, IMHO, a no brainer.

[Marcel Birgelen]

I will say, on our NOC installations, we always were informed, before the manager discovered it, a failed PSU or RAID drive.

Unless you manage to configure your RAID as a RAID-0, in which the "R" in RAID becomes obsolete, you usually have some time to diagnose the problem if a single disk fails. In a single-PSU system, if the PSU dies, your server goes off the radar, usually without giving you any clue about the problem, while a single failed PSU will show up in your monitoring, just like that single broken disk in your RAID5 array. While a broken PSU may be high on the list while diagnosing the problem, in a redundant PSU situation, the system wouldn't have gone down and debugging would've been far easier, probably not with the usual stress involved of a show starting in a few minutes, or worse, a show that went dark.

I agree with Leo here. Those redundant PSUs just make up such a small part of the ticket price of the entire machine, especially in those OEM server-designs, they should've been the standard option, to avoid any unnecessary bargaining with the customer. Also, easily replaceable PSUs, preferably in a N+1 configuration, should really be the standard in this industry, especially since designing PSUs that can last a decade seems to be a lost art.

[Marco Giustini]

Leo,

let me propose another scenario

Alternative scenario: there are double redundant PSUs in those DSS220s. One morning, the opening manager hears a bleeping sound coming from one of the projector pedestals during the opening rounds (the primary unit had failed the previous evening). She had no idea the equipment could beep requesting help so she looks at her mobile looking for whatsapp notifications and walks away.

OR

she notices the error and ignores it as ‘it still works’.

[Leo Enticknap]

Yes; that's a risk. I've been called to Doremi and DSS servers and found two bad drives - no-one called a tech after the first one went out.

[Scott Norwood]

Has anyone looked at what is causing these power supply failures?

I'm genuinely curious because, in a data center environment, computer power supplies pretty much never fail. As in, maybe one out of one thousand per year. In comparison with other failures (motherboards, RAM, and disks, in that order...also, fans), power supplies don't even really rate.

Is Dolby using low-quality units? Or is it a problem with bad power being fed to them (versus the relatively stable power in data centers)? Or something else?

[Leo Enticknap]

I suspect that a data center is an inherently less hostile atmosphere: temperature and humidity kept within limits, filtered air circulating, racks designed with decent airflow to keep all those servers cool, etc. etc. Compare that to your typical projection booth, where temperatures in the 90s, air sticky with popcorn oil, thick layers of greasy dust on everything, servers sometimes mounted in a pedestal with very poor ventilation, etc. etc. Because neither staff nor customers ever spend any time in the booth in a digital cinema situation, it is the last priority for being kept clean and at a reasonable temperature.

[Steve Guttag]

Dolby uses Supermicro frames with the Ablecom power supplies. I've found them to be batch related. If I have a power supply go in a server, the other one is more likely to go before all is said and done...while others, with far longer times, never fail. Most of our servers are on double-conversion UPSes so the power should be rather clean.

[Leo Enticknap]

Same thing with RAID drives. If I have a drive that is original to the server fail, all of them are the same model and likely the same or a close batch. The ones that are still alive will have been operated for the same time and in the same environment as the one that just croaked; therefore, I always advise replacing the entire set. That advice isn't always taken, with the result that I'm usually back there to replace the others not long afterwards.

[Harold Hallikainen]

Has anyone had a power supply fail in a JSD-80, JSD-60, or JSD-100? I don't think I ever heard of a failed one when I was with USL. But maybe there were some I did not hear of.

Harold

[Carsten Kurz]

The CP750 power supply issue is an interesting one, as a very simple and cheap part is the reason for their high failure rate. At least according to a Dolby document dealing with the issue, the OEM switched over to a 'lesser' part during a production run, and did so without telling Dolby.

I wouldn't understand why power supplies have to fail earlier than other parts of the electronics. Yes, higher voltage, higher currents than in the signal path, but then, there are components and circuits to deal with that, and power supplies simply need to be built to cope with that. Audio processors are not actually manufacturing plants when it comes to current consumption.
Compare that to e.g. the extremely low failure rate of the old QSC USA amp series. They combined high power AND reliability. Making a standard switching power supply more robust probably costs just a few cents more nowaday.

[Gordon McLeod]

Has anyone had a power supply fail in a JSD-80, JSD-60, or JSD-100? I don't think I ever heard of a failed one when I was with USL. But maybe there were some I did not hear of.

Harold

I have had a'lot of those power supplies fail as of late. Thankfully they are available off the shelf and are quick to replace

[Harold Hallikainen]

Thanks for the update! I was at USL/QSC for about 13 years and was not aware of failures of those supplies. Could be that they started failing after that or that I was just not told of the failures. I thought they were pretty reliable. Now, the power supplies on the IRC-28 tended to fail pretty frequently. In one case, we found that the solder connections from the AC inlet to the PCB were bad. But then there were also just a lot of "old age" failures.

MANY years ago, back when I was designing equipment for the broadcast industry, we had a lot of switching power supply failures due to momentary power interruptions. Those power supplies used an NTC thermistor as an inrush current limiter. When powered up, they had a "high" resistance to limit the inrush current. After a second or two, the thermistor would heat up, lowering its resistance, so dissipation in the thermistor would drop and efficiency would increase. But, if you had a momentary power interruption, the thermistor would not cool before the power came back. The still hot thermistor would not limit the inrush current and the diode bridge would short, sometimes taking the switching transistor with it. We added a couple power resistors outside the supply to limit the inrush current no matter what the thermistor did. Also added varistors between each side of the line and ground at the power supply input to limit damage from lightning.

But, the power supplies used in the JSD did not seem to have the inrush problem.

Again, thanks for the update!

Harold

[Gordon McLeod]

The problem is equally common on the XD10P
we have had a fair number of the power bricks fail on the IR emitors for the newer caption/audio transmitters