CRU drive causes Doremi to shutdown abruptly

[Dustin Grush]

So, basically the title says most of it. We have a Doremi from 2014 and upon inserting the CRU drive for "Don't Breathe 2" the system basically shut off. He tried it 5 times in total, each time it "went down hard." I wasn't here to witness and I have no interest in trying to repeat it for fear of damage. I took the cover off of the CRU drive and nothing appears out of the ordinary. There are no bent pins on the dock. A second CRU drive was tried after this and ingested successfully. The offending drive was used successfully at our other drive-in 2 weeks ago. The system operated normally following this. Has anyone seen this before? We are having another drive shipped out, hoping it gets here on time. Thanks in advance.

[Carsten Kurz]

Not seen this before, but it is quite possible that the drive creates a short on the power supply lines. That would explain the hard shutdown you experience.
YOU should not have to diagnose this yourself, though - put a sticker on that drive which says 'caution - defective', and send it back to them.

In case the replacement does not get to you in time, you could open the CRU, take out the drive, and try it within another CRU sled. Without testing, not possible to say wether the sled, or the drive itself is causing the issue, though.

[Marcel Birgelen]

I've only seen this before with a SATA drive that had some kind of short on the power bus. I don't know if it was the system protecting itself or just collapsing due to the short, but it also went down hard, as in it crashed hard, because the power got pulled.

If you have a "disposable machine", you can try to hook it up to this machine via e.g. SATA or one of those SATA to USB dongles... otherwise, I wouldn't touch it anymore.

[Randy Stankey]

Yeah, I agree. It sounds like an overloaded USB.

My computer has two USB ports on the front panel where I used to keep a scanner plugged in at all times. One day, I was moving my chair, caught the USB cable and ripped it out of the socket. That incident must have trashed the USB socket because, now, whenever I plug anything into that socket it triggers a USB overload warning and shuts down that part of the USB bus.

That's what is guiding my thinking.

First, does it only happen on one USB port? If you plug another device into the same port, does it do the same thing?

Is the problem in the device? If that drive is plugged into another port on the same machine or on a different machine, what happens?

Assuming it's the device, what would happen if you put another hard drive into the same CRU? What would happen if you put the same drive into a different CRU? My guess is that it's either in the CRU or the drive. I don't know which. Experimenting would be the way to tell.

If the CRU is bad, the answer is simple. Swap CRUs and you're good to go. If the drive is bad, it gets more complicated depending on what's wrong with it. Maybe you could hook the drive up to another computer and clone it to a different drive. It might work. It might not. I don't know.

My big question is why does hooking a device to a USB port on a cinema server cause it to bomb?
Every other computer that I have used just shuts down the bus and puts up a warning. Even Linux.
What's different here? Do cinema servers not do this?

Even if they are supposed to, plugging something into a USB port and bombing the whole computer is a BIG problem!

If that's true then I agree with Carsten. Tag the drive as defective, send it back and get a replacement.

EDIT: I have another idea. Could it be the USB cable? Not simply the connectors. The conductors inside the cable, itself.
You've heard me tell stories about stage cables getting beat up. If something twisted, kinked or crushed the cable, it could be bad on the inside and you wouldn't know it unless you tested it with a multimeter or a cable tester.

It is conceivable that the drive could have been laying on a table and somebody threw something on top of it or was, otherwise, absent-minded and trashed the cable. That's what caused my computer to get a trashed USB port. I did a stupid.

[Dustin Grush]

Thanks for the quick replies everyone!

Randy,
This is the internal CRU dock on a SATA cable. I haven't tried it on an external USB connected CRU dock. There is no apparent damage to the device; although it isn't outside the realm of possibility that it was inadvertently dropped without my knowledge. My first thought was that the pins were bent on the CRU, which I have had happen before and luckily noticed increased effort to insert the drive. The dock has been used once successfully after the failure, so it's either intermittent or the drive.

I wont likely do any more testing unless the replacement drive won't be here in time. Not a huge issue for us, really. I mainly post to get opinions that can be useful down the road, either for myself or anyone else searching the forum looking for a quick answer when their tech support doesn't answer the phone right away .

I have experienced the USB failure that you mention, caused by a smashed USB cable that caused an internal short, which destroyed the entire USB bus on that motherboard. Sad because everything uses USB these days and investing in a PCI USB card for an old machine is money better spent on a "new" refurbished one in my opinion.

[Leo Enticknap]

The offending drive was used successfully at our other drive-in 2 weeks ago...

Same model of server, or different?

If you can't get a replacement drive in time, one option would be to FTP the DCP out of that other server into a laptop, and from there into the server that is offended by the drive.

[Dustin Grush]

Leo,
It is the same model.

Unfortunately it has been deleted from that server already. I have tried to FTP into the server to remove content in the past without luck. I think the login/password was incorrect, its been a while since I tried though.

[Leo Enticknap]

To FTP outgest from a Dolremi, the steps are as follows.

You first need to know the UUID of the CPL you wish to outgest. You can do this by going to the content page in the server's web UI, and clicking on a CPL entry. You will then see something like this:

IMS3000_findCPLUUID.JPG
Then, using an FTP client (e.g. Filezilla), establish a connection to the media NIC on the server, using the username "manager." Per forum rules, I can't post the password - email cinemasupport [at] dolby.com if you need it. Authentication is plain FTP (no TLS or SFTP), and the transfer type should be binary, not ASCII. Once logged in, go to the assets folder, and you'll find all the ingested CPLs in there, in folders named with the UUID. Here is the example above:

download_from_IMS3000.JPG
Simply download the entire folder to your local PC, and you're done. Getting it into another server from there can also be by the same method (though for Dolremi servers, you need to log in using the "ingest" username), or by putting it on a thumb or hard drive, partitioned and formatted such that the server can read it (on recent software versions at least, Dolremi will read NTFS, but not HFS+) and ingesting offline.

[Randy Stankey]

This is the internal CRU dock on a SATA cable.

Ah! I see! Bad assumption on my part!

I have the habit of always shutting a computer down whenever I change hard drives, even in bays.
I don't know whether those CRU bays are hot-swappable and I never bothered to find out. I just figured it is always safest to shut down.

USB devices are hot-swappable so I just assumed that you were using USB with a CRU carrier.

In the rare occasions when I have swapped in a bad hard drive via an internal bay, the computer has always failed POST, either with a bomb dialogue or simply beeped and shut down.

In any case, I find it curious that swapping in a bad hard drive would bork a computer like that.
I would expect the interface card to send up a flare (a system interrupt signal of some ilk) and then shut down.

[Carsten Kurz]

CRU Slots are eSATA and can be hotswapped (usually with no issues).

[Leo Enticknap]

I don't know whether those CRU bays are hot-swappable and I never bothered to find out. I just figured it is always safest to shut down.

They are warm-swappable, so to speak. On older GDC servers, you're supposed to press a "close" button in the UI to dismount them first. If the drive is ext3 and you don't, that can screw them up, because it doesn't write the final journal information and this can cause problems when mounting the drive in another computer. I don't know of any other server that needs you to initiate a dismount process before removing the drive (assuming it isn't actually being read from at the time). On a CRU-DX115-DC reader, you're supposed to push the gray button and then wait until the blue "drive present" LED flashes and then goes out, before unlatching the cartridge. I guess this is to spin down the drive in an orderly fashion, rather than have you physically move it while the platters are still rotating.

[Frank Cox]

My GDC SR-1000 has a close button in the gui that's supposed to be pressed before you yank a cru drive out of the adapter doodad. Then the list of the cru drive contents goes away.

[Marcel Birgelen]

Is there any particular reason why any server should mount a drive for ingestion other than read only? That way, yanking the drive only has minimal consequences, only if you're actually transferring files from the drive, but this can easily be detected by the OS.

Otherwise, any drive mounted read/write should ALWAYS be ejected by the OS first, not only to properly close the journal (as most modern file systems keep a journal even if they don't call it that way), but also to give the OS AND hardware the time to safely write back any data in the many layers of cache that hasn't made it yet to safe, permanent storage. Those cache layers are plenty and abound in any modern computer system.

[Carsten Kurz]

Most DCI servers allow to export DCPs to these drives as well, thus need write access to accomplish this. That said, the export function is usually called from the GUI, so, it would certainly be possible to unmount the drive, then remount rw upon calling the export function. On Doremis, CRU drives are mounted read only during normal operations, and are only remounted rw for DCP export. The Doremi GUI has no eject button. Our Sony server features an unmount button, but I never had any issues ejecting an ingest drive mechanically only. Maybe that button is only really useful while an export is running in order to leave the drive data structure clean even if you abort an export by ejecting.

[Marcel Birgelen]

All Linux kernels of the last 20 years or so have had the possibility to hot-remount a drive with different options, as in with rw/ro options, especially with filesystems close to home, like ext2/ext3. It would make sense to remount a drive with rw options only if such access is needed.

While display-wise it was always counter-intuitive to throw your removable media into the trash, but I kind-of like the Apple philosophy that you're only physically able to unmount a drive, once the operating system has given the "all-clear". Obviously, there should be some mechanical override for this, but it forgoes the problems with dirty filesystems and half-aborted transfers.