UPS Failure Quirk, how worried should I be?

[Steve Guttag]

What were you on before Cyber Power? Also, we tend to have projector electronics as well as external servers on the UPS. Networked sound systems (QSYS) also go on a zero-transfer UPS/ATS system.

[Gordon McLeod]

What were you on before Cyber Power? Also, we tend to have projector electronics as well as external servers on the UPS. Networked sound systems (QSYS) also go on a zero-transfer UPS/ATS system.

APC and Tripplite

[Steve Guttag]

Were you having troubles with either or was it just a price thing on the Cyberpower?

[Gordon McLeod]

The electronics on the rack mount tripplites started to fail and they were not readily available locally

[Steve Guttag]

We found that the SMART series of Tripplites after they switched to the fancy LCD display did not have a good service life. The original ones with the 5 or so LEDs did okay. Every once in a while a fan failure other than the batteries. The SU series have been pretty reliable for us, including the later ones with LCD in their name (something like a 2-3 line LCD display. Again, every once in a while a fan will give up. The worst part of Tripplite is that when the batteries fail, it just shuts off and plays dead. But, with an ATS on the output, it is livable. Overall, I like the build quality of the APCs better though those sharpened battery caddies are nasty with their double-stock foam.

[Carsten Kurz]

Our Tripplite 1500 (no display) that came with our Sony is now in it's 11th year on the first battery set. Admittedly - it is only operated during show times, and completely shut off (no supply at all) between them. I think these Tripplite devices use a rather low charging current (as opposed to older APC systems that I know), so they treat their batteries better. That means, it also takes longer to recharge them after a power outage, but, as we rarely ever have power outages here, that's no big deal. Also, this UPS is not meant to operate OVER a full power outage, but just to shut down the server gracefully automatically, so only some minutes of rated power capacity is needed from the batteries.
In areas where a UPS really has to do work, a faster recharge strategy will be necessary in order to cope with subsequent outages. And sometimes, long bridge times are necessary.
I wish UPS would have this configurable.

[Bruce Cloutier]

As for notification we had wanted to know when line power dropped out here. We have a JNIOR powered by the UPS. We plugged a spare 5-12VDC wall transformer into a wall outlet and connected it directly to a spare digital input on the JNIOR. The alarm is set on the JNIOR input to trigger and send email on state change. This requires that the JNIOR have access to the network and email server of course.

We had power drop here for 20 seconds last night at 7:42 PM. An email eventually arrived from the JNIOR. In the console you can use the IOLOG -O command to display the I/O log and see when power precisely dropped and then was restored. Some DC power supplies store considerable amounts of energy and a digital input is only about a 1.2K Ohm load. So this doesn't usually respond to a 1 second power glitch.

If you want notice when the UPS fails I suppose you could power the JNIOR from the wall and plug the DC supply into the output of the UPS. It seems like a UPS should fail to passing line power and some annoying alerts as to the need for new batteries. But nothing is designed as you think it should be.

Some customers (we think) have JNIORs running on really small 12V lead acid batteries maintained by simple trickle chargers. You could set that up with sensing DC sources on both line and UPS. A JNIOR just chatting on the network and not powering any relays draws under 100 ma. So an almost dead battery (e.g. one not keeping the UPS happy) is probably good enough. A JNIOR not tasked with relay loads should run just fine off 9-10VDC so the battery could even have a bad cell. Just saying. Not that I would want to rely on failed batteries.

It's just an inexpensive use for an existing JNIOR. There is a lot of monitoring that you could do., Add a temperature sensor and monitor the booth/projector and send warnings or even engage a fan.

I think Carsten, you mentioned how underutilized these JNIORs are.

[Mark Gulbrandsen]

The electronics on the rack mount tripplites started to fail and they were not readily available locally

The one 1500 VA, Regen that I did have fail had factory part numbers on all the Semiconductors...

[Ryan Gallagher]

Breathed a sigh of relief today when the boss confirmed he had ordered the Battery Module for the christie pedestal in time before SXSW.
Grateful for all the thoughts here too, will keep in mind if we ever shop for new UPS units.

[Marcel Birgelen]

I think I've stated here before that I'm generally not a fan of these small in-rack UPSes, especially those from APC have not only let me down over the years, they've been a crime to work with. I've seen two of them going up in smoke, one I carried out of the building on a pallet jack, while smoking, the other one actually set the building on fire.

Like Carsten indicated, many of those units fail to even put themselves into bypass when stuff like batteries fail.

Quite recently, I had a whole bunch of rather recent APCs reject their lithium battery packs units due to "a firmware bug". Also, they're actively pushing you towards lithium-ion based solutions instead of the older, classic lead batteries. Having a bunch of high-density lithium batteries in your rack, nice... what COULD go wrong?

About a month or two ago, I encountered another APC that was strobing its output in a 5-second interval, luckily, most of the equipment behind the UPS had been decommissioned, but imagine having anything of value connected to that UPS...

Years ago, before Schneider Electric gobbled them up and eliminated the smaller units from their line-up, I was quite a fan of MGE UPSes, they seemed to be far better engineered than this APC dreck and from what I've heard the most recent iterations of Tripplites also aren't any better anymore.

If you're lucky and you're operating in an environment with a stable power grid, I highly doubt the effectiveness of UPSes for cinema playback purposes. Unless you invest in a "real" big beefy UPS system with some emergency power generating equipment as backup, you're not going to ride out any major power outage anyway. Heck, most likely you'll not have your lamp and power amplifiers on UPS anyway, so any power outage will impact the show anyway and will most likely require some manual intervention. It's not like we're running high-transaction database servers, so the chances of any serious file system corruption on the playback side of things is limited.

So, I really would consider if the extra maintenance, the extra failure modes and fire hazard are worth it. If you live in an area where the power supply is really unstable, then, well, maybe, but even then, a simple small rack UPS isn't going to do the trick for you.

[Steve Guttag]

We, generally, have UPSes on the servers, projector electronics and network based sound equipment (QSYS). We have not had the issues that Marcel speaks of with either APC or Tripplite. No fires, no smoking. The most common "failure" are batteries after 4-years. And that does seem to vary by battery brand.

The value of the UPS is what they are protecting. Since servers take between 5-10 minutes to come up after a power blip, it is best to not have them going through that cycle...particularly if a storm comes through and you get a sequence of those. Likewise, no good will come from exposing the equipment listed above to a bad power. It isn't like the power is clean up until the point of it going out...you get brown outs, crappy power and the like all through such events.

Often now, our networked audio is driving multiple screens (so far, our largest is up to 9 screens on a pair of Core 510c in a primary/redundant fashion. I'm not eager for the smallest of power blips going after them. We have APC double-conversion UPSes on them with such an overhead that they can run that rack for multiple hours on battery (and have).

We have gravitated and are now almost exclusively on double-conversion style UPSes for all applications.

While it is true that, for the most part, we are not keeping the lamps on (nor the amplifiers), the automation system is kept going and has the ability to pause the show when the power event occurs...something that a sudden power loss would not afford. Different servers behave differently on how they wake up from a sudden shutdown, mid-show.

Just looking at one of the ATSes to see what the up-time is (and the system went in 2020)

Screen Shot 2024-02-26 at 7.32.00 AM.png

Uptime is 1342 days (almost 1343) yet some recent events, about a month ago, in the early morning hours went by unscathed (Source B is "City Power").

[Ryan Gallagher]

Yeah my instincts are the same as Steve's. They are effectively higher quality power-conditioners and surge protection... that in order to perform that function reliably through momentary phases of instability need the battery feature. No one should expect them to outlast a true outage, nor be part of an equation to permit a screening to continue (not that anyone said that).

Anything that resembles a computer on the inside, has volatile memory or writable storage media, benefits from avoiding those unexpected power cuts and unclean power. (DCP servers/IMBs, Sound Processors, Managed Network Switches etc etc). Particularly also handy to avoid bricking anything during a firmware update too of other "dumber" equipment.

That said, I suspect most DCinema setups are not nearly as "integrated" as the IT world is, where your UPS can request a "clean shutdown" of critical equipment after a certain time interval. One could probably get there with what most would venues like mine would consider "excessive" automation and monitoring.

As a sidestory well before my time: our install learned the hard way that amps should not be on the UPS. The DCinema installers had it set up that way, and audio failed spectacularly and repeatedly during a premiere where they were driving the surround amps harder than the UPS would permit without going into a protective mode.

But, having the audio processor on a UPS might remove some chances of sending transients to the PA and damaging drivers etc.

Although Lithium, while a superior battery tech, might make me re-consider the benefits versus risks. Essentially any backup system you are MORE worried about than your power grid, seems like a awkward choice at best. Projectionists are understandably a little twitchy about adding fire-hazards to booths due to historic contexts!! (Especially fires of the non-extinguishable variety, we have a history with those!)

[Marcel Birgelen]

Let me be clear, I'm not against the idea of a UPS in general, but those cheap-ass Line-interactive UPSes everybody is deploying everywhere are more of a pain than a gain at any rate. Given my experience in failure rates and associated risks, I avoid them like the plague. APC & co. must have sold millions of those $400 - $2k units in order to sell their users a false sense of safety.

For surge protection, I really urge people to install separate units in their power distribution boards, they're also not THAT expensive and have proven to be pretty effective, even for inductive loads via the power grid, caused by lightning strikes. Keep in mind: Nothing will save you from a direct hit, other than good lightning conductor installation...

It seems that Steve only has experience with the more expensive double conversion "online" UPSes. Those "online" UPSes are an entire class above those crappy Line-Interactive units, they are also in a different price range. And, they generally are a hard sell around here, since the power grid itself is extremely stable. But those kind of systems are the only ones I'll advocate. Still, I urge everybody to put those systems into a separate room, away from your most precious equipment, if possible. Yes, you'll pay extra in wiring, but you can make up by buying bigger, more reliable units and centralize your UPS power there. Keep in mind that once that stuff is burning, you'll not putting it out any time soon... it burns until all stored energy has dissipated. The move for smaller, lighter units driven by Li-ion batteries will only make this situation worse.

We had a big datacenter fire in 2021 at the biggest hosting providers in France that knocked millions of sites and other cloud services off-line and caused severe data-loss for those who didn't have off-site backups. There were obvious a lot of things wrong in their ghetto-style datacenter, but this fire also started in the inverter of one of their UPS units. With big power comes big responsibility...

[Ryan Gallagher]

But I guess we were debating the benefits of these smaller rack UPS, versus larger alternatives. Not really UPS vs No UPS. ;-)
The issue with these smaller ones seems to be the extreme variability in quality, features, and failure behavior.

[Mark Gulbrandsen]

I only used the "on line" type in my installations at Steve's reccomendation. Am pretty sure there were several threads about this way back when Series 2 digital conversions started happening. Probably 2010, or 2011. I did do a fair number of Series 1 systems to cover 3D needs, including a couple of Giant screen stacked 3D systems, but none of those series 1 systems had UPS's iniatially, they were installed later on during service visits. Amazingly, none of those places ever had any outages when they didn't have the UPS's either.

I do use a standard 1500 VA APC UPS here at home. I do open it up, clean out the dust, and inspect it yearly. It's going on six years old, everything has been fine, but I am well aware of the "Garbage UPS's that APC produced in the past. Perhaps they started enough fires over the years that they were forced to put in some reliability. I do have outages here once or twice a year, and an occasional power bump. Everything has always kept on running.