IMS3000 Connectivity Issue

[Sara Meyers]

Hello all,

The past few days I have had to unplug/plug my christie projector in order to get the IMS back online in my Theater 3 (turning the projector off and on does nothing), before I do this I get a blank internet browser page. Today was the first day this new thing happened: After rebooting the projector it appeared that the IMS came back online but then it wouldn't let me log in. I have tried throughout the day to no avail. I'm including a photo of what I get after I enter admin and pw- just goes back to the same page, no reason or anything.

I ended up being able to play it directly from the TMS - do we call this Doremi still? (I could access the network fine but when I tried to control the theater it showed that T3 had no content to be played.) When I loaded up the show from the Doremi (I'll call it that for now) It showed that for only 1 of my features that the KDM was missing even though it wasn't expired. And very strange that the other show loaded just fine kdm wise... I ended up manually ingesting them with a flash drive (again wouldn't send to T3 from network TMS) and it worked.

The only thing suspicious is that there is an ethernet cable plugged into my master management box that does not have a light illuminated (inserting that photo as well). But I imagine that I wouldn't get as far as the IMS login page if it was just connection.

Oh and I did reseat all ethernet cables which made no difference

Thanks for the help!!- Sara

[Steve Guttag]

You have an internal SSD failing and failing fast. You need to contact your tech support to get a replacement and, at the very least, a bootable USB drive.

[Leo Enticknap]

There seems to be an epidemic of this going on. I have had six IMS3000 boot flash drive failures among my customers since the start of November. As Steve notes, the workaround is to create a temporary bootable USB flash drive: if your regular tech is not able to do this for you, email cinemasupport [at] dolby.com, and they will be able to give you links to download the image file and a utility that will burn it to a USB stick. Once you've created the drive, simply insert it into any of the USB jacks on the faceplate of the IMS3000 and boot it. The only difficult part is that the IMS3000 will have reverted to its factory default IP addresses (192.168.100.50 / 24 on eth0, and DHCP on the other two) after booting from the USB stick, and so you'll need to configure a wired NIC on a PC or laptop to communicate with it on that address, log in, and then restore a settings backup that it should have saved on the content RAID hard drives automatically.

After that, your IMS3000 should work OK until you get a replacement official boot flash drive. If your IMS3000 is in warranty, Dolby will ship this to you free of charge. If it isn't, you'll need to order one through a Dolby dealer (the company I work for is one - feel free to PM me if you need a quote). As an alternative for if your IMS3000 is out of warranty, you could buy a Dupont female to USB 2.0 B female adapter, e.g. one of these, and use the USB stick permanently; though this is an unofficial solution deployed at your own risk, obviously.

[James Gardiner]

@Leo, could you not get a industrial grade "boot flash drive" like Dolby should be using and flash it like the USB and gto from there if you need to as well.. They are supposed to be more resilient to multiple writes unlike a typical USB is. That's why a USB is NOT a long-term fix.

[Leo Enticknap]

The boot flash is not written to that often: only when configuration settings are changed, is my understanding.

My guess is that Dolby uses that Transcend board with the Dupont connector to the motherboard because that was the most ergonomic way to do it within the form factor of the IMS2000 and 3000, and because it boots a bit quicker. I haven't done a stopwatch test, but my subjective impression is that from power on to being able to log in to the SMS web UI, it's about 30-60 seconds faster with a working Transcend board than it is with a USB stick in one of the faceplate jacks.

As has been noted in this thread, "emergency" USB sticks have been operating as the boot drive in DolRemi rack servers (which actually have a USB B jack on the motherboard for the boot drive, rather than the 9-pin Dupont header found on IMS2000s and '3000s) for many years without issue. I have opened up servers and discovered the original drive disconnected and a USB stick installed in its place that predate my starting at MiT, meaning that the USB stick has been functioning as the boot drive reliably for at least 6.5 years.

None of this is to advocate for an unofficial field mod solution as a permanent fix necessarily. I'm sure that the Transcend drives supplied by Dolby have been QC-ed and tested by them, which won't necessarily be the case if you buy a USB stick from Amazon. Furthermore, all USB sticks are not made equal, and that's another variable to consider. But I've now experienced enough failures of IMS3000 boot flash drives myself, and heard reports of them from other techs, to suspect that Dolby likely received a bad batch of them that has affected a significant number of IMS3000s in the field (this is absolutely not a criticism of Dolby: every OEM that assembles devices using components sourced from third parties is affected by this occasionally, ourselves included, and Dolby have been superb and supportive in shipping warranty replacements for the in warranty units I look after, including one for an IMS3000 with a warranty that expired a few days before the fault presented).

If you have an IMS3000 with a boot drive gone bad and it's still in warranty, you might as well take a replacement official one, if only because it's free. But if the warranty has expired, a good quality, brand name USB stick as the permanent replacement is, IMHO, at least worth considering. I wouldn't want to leave it in one of the faceplate jacks long term, because of the risk that someone might mistake it for a content drive, remove it, and thereby crash the IMS during a show. With a Dupont to USB pigtail adapter and the resulting arrangement zip tied to the board to make it secure, that risk is removed.

[Leo Enticknap]

Bugger me ... yet another one!

Reported by a service contract customer just now. Playback via Crestron automation wouldn't start; connected to remote access PC; saw "internal server error" on the web UI, then a 404 error when I refreshed.

This is an identical symptom to most of the boot flash failures I've dealt with in the last month, and very similar to the others.

The one that borked just now was installed on April 5, 2021. Two of the others entered service on August 25, 2021. I'd need to do some more digging through records to find the install dates of the others, but my rough memory is that they were all around the spring and summer of 2021.

[Steve Guttag]

Mine (that have had SSD issues) were installed summer of 2020...so they were likely assembled late 2019 or very early 2020. Since I ran into certificate issues (covid shutdowns so the equipment wasn't powered up much)...I'm going with late 2019. That is where I think the bad batch originated. So the question is...how far out does Dolby buy things like SSDs. Certainly their sales were VERY slow in 2020 so I could easily see 2021 units getting in on that batch.

[Leo Enticknap]

And another email to our service address this morning, describing the exact same symptom! This is for a unit that I did not install (in fact, I cannot recall ever having been at that site - not one of our service contract customers), so I don't know its provenance. At this rate I'm going to have to prepare a significant stock of boot flash drives and Dupont to USB pigtails ready to go. I'm thinking it might also be worth working up a cheat sheet, with annotated pictures, for A/V techs who know what they are doing, but are not trained on DCI equipment, on installing a temporary boot USB (probably just using the faceplate jack, as this would avoid the need to trigger a projector tamper), so that with that plus the image file, a remote site could get themselves back up with Teamviewer support from this end. Maybe Dolby already has a cheat sheet - I'll look on their support site when I get a moment.

Your guess at late 2019 makes sense: I'm pretty sure that none of the failures I've dealt with affected units sold before then, and I do look after a couple of dozen installed between the time I started at MiT (June 2017) and the covid outbreak. I'd have to check our records for the install dates of some other failed IMS3000s that I fixed but did not originally install, but I'm 99.9% sure that those were installed during (we did do some residence theater installs with IMS3000s during the spring to fall of 2020) or post-covid.

[Sara Meyers]

Update SSD replaced and all is well. Thanks for the help y'all!!