32B, fmt-fpga and icp-fpga way too hot

[Marlon Martins]

Hi folks! I'm in need of help again

our 32B started to show a red light, seeing the logs, it was a alarm because fmt-fpga has passed 85 decrees C. this with the projector on "standby", no movie being projected.

the thing is, even with no air filters, all fans are ok, running with proper RPM, the temperature is reaching 85c. turning the room AC, the temp drops to 80c

the temps on the others sensors are way lower, as espected, and the other projector in the other booth is at 50c (fmt-fpga).

only the fmt-fpga and icp-fpga are hot (rest of temperatures are between 26c and 40c max)

what could possibly be? should i remove all boards and do a inspection? should i replace the heatsinks on the boards and replate the termal glue?

thanks!

bc68e52a-8f3f-4311-a3bd-d15589014e96.jpg

[Steve Guttag]

Are you sure that is what it said...it isn't supposed to alarm out at 85c. Checking the ICP fan, make sure it is clean/clear. I just looked at one and it is running at over 7650RPM. That is what keeps the ICP cool. Also makes sure the ICP board is clean and the heat sink is not full of dust.

As odd as it may sound, it could be that the ICP needs to be reseated because the edge connector could not be making good contact to convey the heat information.

More often than not, when I see over-temps, it is the formatter/DMD related with Red, when it is the top most DMD, being the one that cries first. 9 times out of 10 either the coolant has run low, thickened because someone just added more coolant rather than replacing (adding coolant thickens the mix because it was the water that evaporated out, not the glycol) or, there is a clog in the cooling lines.

Post the actual error and it will be easier to diagnose.

[Marlon Martins]

Are you sure that is what it said...it isn't supposed to alarm out at 85c. Checking the ICP fan, make sure it is clean/clear. I just looked at one and it is running at over 7650RPM. That is what keeps the ICP cool. Also makes sure the ICP board is clean and the heat sink is not full of dust.

icp fan is about 8000 rpm. will check for dust.

As odd as it may sound, it could be that the ICP needs to be reseated because the edge connector could not be making good contact to convey the heat information.

will do

Post the actual error and it will be easier to diagnose.

18.12.2021 14:08:07.411 FMT excursion: 85.125000 C
18.12.2021 14:08:07.411 One Second: Set State Change
18.12.2021 14:08:07.411 ICP Temperature/Voltage Status Change. Status Byte 2 Current Value=0x40 Previous Value=0x00
18.12.2021 14:08:09.580 FMT FPGA temperature returned to normal operating range
18.12.2021 14:08:09.580 One Second: Set State Change
18.12.2021 14:08:09.580 ICP Temperature/Voltage Status Change. Status Byte 2 Current Value=0x00 Previous Value=0x40

i got a "ti-icp - system status = fail" message for a few minutes when over 85c

[Leo Enticknap]

If the ICP fan is OK and the heatsink isn't gunked up, I'm wondering about a bad sensor in the ICP; especially given that if the ICP fan had stopped, that event should generate its own error message. Swapping that ICP with the one in the other house would tell you if the problem is on the board or not (does the fault move with the ICP?).

More often than not, when I see over-temps, it is the formatter/DMD related with Red, when it is the top most DMD, being the one that cries first. 9 times out of 10 either the coolant has run low, thickened because someone just added more coolant rather than replacing (adding coolant thickens the mix because it was the water that evaporated out, not the glycol) or, there is a clog in the cooling lines.

The former causes the latter, over time. If it's allowed to get that far it's a whole bunch of no fun to fix, involving poking the barb couplings with a thick needle to clear the clog, pushing distilled water through the DMD heatsink with a syringe until everything is out, sometimes replacing hose sections and clamps, etc. etc. I had one DP3000 that was so badly clogged that it took me a whole day to get coolant flowing again. A hairline crack had developed next to one of the hose clamps, and water had evaporated until the point that only a thick slime remained in the circuit. Because it was the Series 1 pressurized system, there was no visible way to see that it was getting low, and so no-one had touched it for about 10 years, until the pump finally failed because it couldn't move the slime anymore. I had blue goo stain all over my hands for days afterwards - no amount of washing and showering seemed to clean it off.

[Jason Raftery]

Marlon, I've seen this issue a few times and yes, the ICP FMT-FPGA alarms on the TI-side at 85C/185F. The current version of Communicator lists the wrong maximum warning temp. but will display the correct actual reading from the ICP. In my experience, this error has always been the result of a lack of airflow across the electronics bay, typically due to one or more of the following:

• Dirty/clogged front air filter
• Dirty/plugged card cage fans
• Dust buildup on mesh screen on card cage side of fan for light engine cooling
• Dirty/plugged metal screen in the cathode air inlet on lamphouse

If the ICP fan were the culprit, you should be seeing ICP fan speed low warnings or be able to hear it having difficulties.

Try removing the lamphouse at night when the projector is idle and see if the FMT-FPGA temp starts to drop. If it does, check and clean the metal mesh screen on the cathode air inlet on the rear right side of the lamphouse--I've seen a number of them plug up over the years as most sites (and some techs) don't check it as there's no mention of a need to do so in Barco's service documentation. If that lamphouse screen is plugged up, the cathode fan will not be able to pull warm air out of the card cage, which will result in a buildup of heat at the ICP overnight.

[Marlon Martins]

Thanks guys, tomorrow there will be no sessions, so i can find the cause at peace, will be back with results ☺

[Marlon Martins]

daaaammm, the mesh screen on the lighthouse was the culprit. very low airflow, full of clogged dust

photo_2021-12-20_23-53-13.jpg

temperature is now 55c

thanks guys for your expertise, this helped avoid hours and hours of troubleshooting

one more thing, the reflector of the lamphouse is cracked... is this a problem?

photo_2021-12-20_23-53-16.jpg

this projector was bought used, as they need a temporary projector until they can buy new ones, as the two 19b where "totaled" by the salty and moisture air here (cinema is on the front of a beach) so rust and salt kill electronics here 6x faster than normal

[Leo Enticknap]

Thanks for the report. As Jason points out, checking that grille isn't on any of the "smart maintenance" checklists, and so it's something we'll likely overlook if in a hurry. I must add it to mine.

That will do it, alright. I occasionally service projectors in high end home theaters on the Pacific Coast Highway in Malibu. If they've been installed without any thought to dehumidification, a service life of 3-4 years is doing well. Usually what happens is that a projector shuts down because of a fan failure caused by corrosion of the shaft and/or bearings, and I'll find that it's totally seized solid: even pushing hard with a finger will barely move it. And of course the bare metal in the chassis is totally brown.

If your cracked reflector isn't making it impossible to tweak the trixial alignment such that you have even illumination, I suppose you could just live with it. My fear would be that eventually, heat from the lamp will cause the reflector to disintegrate completely, causing the bulb to explode in the process, and potentially needing a replacement of the entire lamphouse module (as well as the bulb) to fix.

[Marlon Martins]

Thanks for the report. As Jason points out, checking that grille isn't on any of the "smart maintenance" checklists, and so it's something we'll likely overlook if in a hurry. I must add it to mine.

that's what I told them. din't even remember being a grill there. when we got this projector, there was a tear in the front filter, maybe dust got in and stuck there. or maybe this lamp-house is olllllddd

That will do it, alright. I occasionally service projectors in high end home theaters on the Pacific Coast Highway in Malibu. If they've been installed without any thought to dehumidification, a service life of 3-4 years is doing well. Usually what happens is that a projector shuts down because of a fan failure caused by corrosion of the shaft and/or bearings, and I'll find that it's totally seized solid: even pushing hard with a finger will barely move it. And of course the bare metal in the chassis is totally brown.

i changed about 10 fans from those 2 projectors in 3 years. most of them, wires got corroded and others stuck. when we removed the 19b's and did a "tear-down" of them , there was piles of salt inside

yesterday i spend most of the day opening the amps and soldering the pot pins at max volume, as even the volume pot start to have bad contact, cracks and low volume was happening. salt screw everything!

i recommended the owner to do a better "air filtration" for the next projectors, instead of letting outside air in the booth to circulate the one from the exhaust, close the rooms and put another AC to compensate the heat, to avoid salt from outside getting in again. he like to get the laser ones from barco in 1 year if the pandemic "pan out"

If your cracked reflector isn't making it impossible to tweak the trixial alignment such that you have even illumination, I suppose you could just live with it. My fear would be that eventually, heat from the lamp will cause the reflector to disintegrate completely, causing the bulb to explode in the process, and potentially needing a replacement of the entire lamp-house module (as well as the bulb) to fix.

ouch, i will tell the owner. perhaps get a replacement. thanks again!!!

[Marco Giustini]

Having a closed circuit extraction with AC will be perfect for the projector, filters and optics will just be spotless for years.

but if the AC fails, then the projector will overheat in no time.
ideally you want some sort of emergency extraction system ready to be activated in case the AC fails.

that grille if memory serves receives the air from the card cage so indeed that was your problem?

[Marco Giustini]

Apologies, double post

[Marlon Martins]

Having a closed circuit extraction with AC will be perfect for the projector, filters and optics will just be spotless for years.

but if the AC fails, then the projector will overheat in no time.
ideally you want some sort of emergency extraction system ready to be activated in case the AC fails.

We can just open the door to the booth to get fresh air if AC fails. considering the money trowed away on the 2 "salty" projectors that where decommissioned, running 2 AC's seem very cheap solution

that grille if memory serves receives the air from the card cage so indeed that was your problem?

Yes, cleaning the grill reduced the temps by almost 30c

[Leo Enticknap]

Having a closed circuit extraction with AC will be perfect for the projector...

Given the projector's location, that HVAC system would need to include dehumidification, which will push the cost up a lot: we're not talking about HVAC that just does heating and cooling. In 2002, I helped to design an archival film vault facility in northern England, 16 miles from the coast. The ambient humidity was usually high 50s to low 60s most of the time. The cost of installing a system that cooled and dried the air to SMPTE specs for nitrate and acetate was GBP 650,000, and the running costs (maintenance and electricity) about GBP 10K a year. Admittedly that was for a system covering a volume of space around that of six or seven typical single screen projection booths, and in a projection booth, you do not need to bring the humidity down to as low as 15%. But the fact remains that dehumidifying a significantly sized room, permanently, is seriously pricey. I haven't run the math, but I almost suspect that for a projection booth in a warm and humid environment, replacing equipment more often than would be necessary in a less hostile environment might even be cheaper than mitigating that environment using HVAC. I'm sure that there is an optimum cost-effective compromise somewhere (e.g. mitigate the environment somewhat using HVAC, but not all way down to the projector's optimum operating condition).

Laser projectors are more sensitive to high humidity than lamp projectors, which may turn out to be an issue in coastal and tropical locations.

[Marlon Martins]

but AC already dehumidify the ambient, but i don't know at what humidity level they would leave the booth at.

the thing about AC costs vs. projector parts, i can guarantee AC is WAY cheaper, for a simple reason: I'm in Brazil. the conversion from Dollars to Real is insane. the cost of buying or fixing a projector here are ludicrous.

just to give you guys a sample in base terms, the price of a 19B i saw online, is equivalent a US minimum wage of 4.6 years. The 19B bought here (a few years back, is pricier now) is equivalent to 33.6 years of a Brazilian minimum wage, so the value of a projector here is around SEVEN TIMES the "cost" for us compared to you in US

And we barely have assistance, parts, and qualified personnel. "Luckily" some cinemas closed with the pandemic, so we got 2 used projectors to replace the 19b's, and a cp750 that died also. If we need "proper" assistance, we need to wait perhaps a week or more for someone to "get here", that why i am the one fixing the stuff here . it's really sad the situation here, so your help is seven times more valuable to us

Thanks again!

[Marco Giustini]

indeed, a closed loop AC unit (split unit) will dehumidify the air by the very principle it works. If the system injects 100% fresh air all the time, then if you want to keep the humidity at bay you may need some extra attention.

Now, you should really have some ventilation in the room - that is, inject some fresh air - but it doesn't have to be a huge amount as the AC will take care of temperature (and humidity).

With prints it's a different story - and in my experience the issue is often too dry rather than the opposite, particularly in winter time. But if you want to keep the temp and humidity within specific parameters, then the system will cost an arm and a leg. But in this case we just need to keep humidity at bay to prevent an electronic device from getting damaged.