DOLBY IMS 2000 Stuck ingest progress that can not be deleted or canceled

[Mike Qu]

We are using one DOLBY IMS2000 board with CP4230 in an single room cinema.
There is one ingest progress that can not be deleted or canceled(The film is already be ingested).This cause we can not ingest anything now throw the GUI.And now all the ingest tasks are
not albe to be progress.
We have contact dolby china,they have no idea,and say have report to the US.
Anyone have any idea how to deal this problem?
We have reset the drives already,nothing change.

[Marcel Birgelen]

It's a bit difficult to guess what's causing this, but have you checked the Log Viewer to see if there is anything in the log related to this problem that's a bit more descriptive?

[Leo Enticknap]

Does it only do this with this specific CPL? In other words, do other movies ingest OK?

If so, I'm inclined to suspect a bad DCP. If you have access to one, try to ingest it into another model of server. If that also won't take it, there is your answer. If other DCPs ingest into your IMS2000 without complaining, that puts it beyond a reasonable doubt.

If this IMS2000 won't ingest other titles either, you could download a log file and put it through Dolby's online automated log analyzer. I'm afraid that I can't help you with what this looks like In Chinese, but with the IMS2000 running in English, you go to monitoring in the main menu, and you'll see this. Press the "detailed report" button, and it'll give you a file to download. When you have it, go to loganalyzer.dolbycustomer.com and upload it there. It'll give you a report that might offer some clues as to what the problem is. If it doesn't, your next step would be to email the log file to cinemasupport [at] dolby.com for their advice.

report.JPG

[Carsten Kurz]

We have reset the drives

What do you mean by this?

[Leo Enticknap]

I'm guessing, reinitialized the RAID.

[Marcel Birgelen]

That also was my first guess. My guess also is a bad DCP, but it could also be a problem on a local filesystem or the filesystem of the source (which is essentially the same as a bad DCP). Usually, the logs will tell you where to look.

[Carsten Kurz]

So, after the RAID had been reinitialized (if so) - was that ingest performed again, did that DCP become fully ingested, or would it still hang? That would now mean, no content at all on that RAID?

Maybe one could simply kill that job from a terminal...but I am sure Dolby will be able so solve this with a description and a full log.

[Mike Qu]

To all the friends above,yes,reset the drives means rebuild the raid,all the content(like DCP inside)gone,but the ingest progress still there.
The DCP that this progress deal with has already be ingested in to the server. I realy can not image how the progess stuck there.
The stuck progress give error hit as:Path is not a valid existing directory/file.
As the progress is in still in progress, no other progress can be deal by the server,that means we can not ingest anyting now.........
We have report and give logs to DOLBY CHINA，they have no idea about this
Today ,we try to update the server,but the first step of the 3-step-upgrade can not be done.

[Elia Orselli]

You should open a case on dolbycustomer portal (if you have credentials).
I would try the software update procedure via FTP folder.

[Carsten Kurz]

I think in an emergency situation - show stopper - they will also happily respond to a simple email to the support mail address.

[Dave Macaulay]

Can't help much from so far away but I have had weird issues like that when the OS flash drive developed problems. I haven't had it on an IMS but have replaced the flash drive on several chassis servers. The RAID volumes store OS data files and DCP content but there will be a replaceable flash drive in the server for boot and OS.
The flash drive can't be reformatted and reloaded in the server, only externally. Dolby has given me images but it is best to just replace the flash chip preloaded, these are available from Dolby and not super costly. Again I have not done an IMS and can't tell you where the chip is.
If you have a tech with good English or a translator, Dolby USA will assist by phone. I don't know if "free" internet calls like Skype work out of China but a direct call is probably worth whatever it costs.
They will want a log package. Maybe you can't reach the US online log analyzer but try it if you can. Not super helpful with serious software problems but will tell you a lot about the system status. Dolby support staff can tell much more from a log package than the online analyzer summarizes.

[Matt Smith]

At the risk of asking the obvious, have you tried rebooting the server? I could not tell from the above if this had been done.

Rebooting should kill any active transfer, and if you reboot the server through the GUI there should be a warning about continuing while an ingest is active. Re-initialising the RAID and your attempted software upgrade won’t reboot the server as far as I am aware.

I experienced something similar once where no content would ingest due to a active transfer that had actually been paused on the server. Although from the image this does not seem to be the case.

[Leo Enticknap]

Again I have not done an IMS and can't tell you where the chip is.

Me neither, but per the diagnostics/storage page, it looks like it is a physically separate storage device, and not a partition on the RAID volume, as it is in a Doremi rack server:

IMS2000_storage_1.JPG

IMS2000_storage_2.JPG

The next time I'm installing or reseating an IMS2000 or 3000, I'll see what I can see.

I did have one, where I took a log file during a planned maintenance visit to check for issues (not because anything appeared to be wrong with it). Dolby Log Analyzer said that some system files had failed an MD5 checksum test. When I ran this past Dolby, they suggested re-uploading the current software image and rebooting. I did so, and there was no change as a result. The same complaint appeared in a log taken after the re-flash. However, the IMS was working OK, and so I didn't take any further action. This was about a year ago, and I've had no reports of any problems from that site since.

[Carsten Kurz]

So, how do you upload an image to an IMS2000?

[Leo Enticknap]

Control > ingest > upload. The server software does not need a KDM - you just upload the server software file, then reboot. It is installed during the following boot sequence. The security manager and firmware (both of which are written to the media block component of the IMS, as distinct from the server) need a KDM, which you have to download from Dolby's FTP site first. Upload the KDM, then the firmware file, then reboot. You are supposed to do updates in the order server software > security manager > firmware.

That is for the IMS1000 and 2000, including their NEC-badged versions. For the NEC-badged versions, you need to get the security manager and firmware KDMs from NEC: they are not on Dolby's FTP site.

For Doremi rack servers, the sequence of files you need varies according to whether the thing has a Dolphin media block or an IMB. The system drive in a Doremi rack server is a little board containing some flash memory that hooks to a USB jack on the motherboard. It is possible to clone it to a USB stick in the front of the server: Doremi issued a cheat sheet with the list of commands for doing so. If the official boot drive craps out, you can hook your clone backup to the USB jack on the motherboard, until an official replacement arrives. The sheet has dire, red ink warnings about how this should only be a temporary measure, replace the official boot drive board with an OEM ASAP, etc. - but I know of servers that have been running on these USB sticks for years with no issues or problems.

The IMS3000 takes a single, combined file, which does not need a KDM. You just upload that file, reboot, and all is done.