GDC server intermittent problem

[Mike Blakesley]

I hate intermittent problems, but we have one.

Our server will, out of the blue, just pause for maybe 1 or 2 seconds, then play again for about 5-7 seconds (or less), then pause again, repeat repeat repeat. The only way to make it stop doing this is to reboot the server, which means about a 5-minute interruption. After rebooting it will play fine, until the next time it happens.

Full disclosure: I have never tried just pressing Pause manuallly and waiting a few seconds, and I've never done anything like stop the show and navigate back to the same spot to restart it. I always figure rebooting is a more surefire fix.

Last Friday the movie played with no incident. On Saturday's show (with a near-full house) this problem appeared. Rebooted, it worked fine, and has worked fine since.

This problem seems to pop up every couple of months or so, just at random. Our tech has pulled the logs each time and they've never been able to find out what's going on.

Has anyone seen anything like this? We have a GDC SR1000 server and a Barco SP4K-15 projector.

[Ryan Gallagher]

Have not seen this with our GDC, but it is not in daily use either.

For clarity, are you describing a playback “hang/glitch”, or is it actually showing as the media being paused?

How many locations/devices interact with the playback controls via console, vnc, automation? Would rule out it being accidentally paused by a human first, or a sticky button somewhere?

[Fraser Falconer]

Hello Mike,
What sort of storage device are you using? Is it a PSD (Portable Storage device) or ESD ( Enterprise storage). Is the RAID array showing as online?

Cheers Fraser

[Frank Cox]

Sounds like one of the hard drives has a problem and it retries the read however-many times.

[Steve Guttag]

Almost always, freezes and pixelations point to the storage...either the drives themselves or the ability to get the data from the drives to the mediablock.

For us, the SR-1000 has has a near 100% service record (could be famous last words but that is the way it stands right now...I think we had one that had to be replaced a year or two into its life but that's it). We use them exclusively with either Enterprise storage and we are starting to use Cine Cache (4TB versions).

My least stable GDC systems, by far, are the SX-3000s with PSD storage. The PSD storage has been a loser for me...I've had the drives, the PSD and event he eSATA cable all go bad with that thing. One should make the distinction between the traditional PSD and the current SSD based ones (the entire unit is different, not just the drives). I don't have any experience with the new style PSD.

If you have the PSD (traditional with spinning rust drives), I would move away from that, if at all possible. I know of nobody with the Enterprise Storage that has regretted it. If you are on Enterprise Storage, I would change the drives.

Normally, GDC can determine what he issue is. If that can't that is odd too. Naturally, you should make sure you are on the most current version for your particular SR-1000 (there are quite a few "SR-1000" variants out there and they don't all use the same version of software/firmware...there is a whole family of "17" software).

[Chris Haller]

I work with two theaters here in Rochester in our chain that use 20 SR-1000s across the two buildings. Steve - you mention that you use the enterprise storage and cine cache. Do you have any experience with the 2TB built in flash storage?

I've got plenty of instances in which the SR-1000s with built in storage exhibit stuttering across various configurations - using the built in 7.1 audio processing, their AIB-3000, hooked into a CP950, a Trinnov Ovation2, paired with various different NEC laser projectors. My most recent stuttering incident at one of our sites was like last week, honestly. The problem I face semi-regularly is not like Mike's where it looks like there's a complete breakdown in playback, but more like your traditional DVD skipping. A little stutter, and the rest is history. Our service tech smoothed one regular instance, which I posted about here in the forums back in November/December, by updating the Dolby CP950 it was attached to in order to resolve buffer underflow issues. The rest occasionally persist.

What's amusing me the most is that GDC's most recent attempt at solving the perceptible stutter was to.......have me take out and reseat the SR-1000 in that projector. Woof. I suspect there might be an issue with the 2TB flash storage or the storage controller on the board for these IMBs. Who knows.

[Ryan Gallagher]

Personally I would imagine with a stutter or glitch related to drive/fs read error or buffering issue due to loads, the actual state of the playback engine still would indicate "play" rather than toggling to pause... which is what Mike's OP suggests he is seeing. But I'm not sure of GDC's UI behavior in the event such a hiccup, perhaps it detects such interruptions and reflects "pause" on the UI?

The playback UI flipping state to me does not immediately scream drives or file system. If there are touch panels involved perhaps one is going bad and exhibiting phantom touches? From the logs one should be able to see if "pause/play" events are actually being triggered around the timestamp of the problem occurring. If there are real play/pause events in the logs then drives seems even less likely a fix. No real play/pause events in logs would flip my thinking back where everyone else was headed with drives/array/file-system health etc.

[Steve Guttag]

Chris,
Thanks for the information. Thus far, we just have use the 4TB Cine Cache version (2TB wasn't near enough and it was really just a means for their streaming TMS/LMS system to cache content in case there were connectivity issues).

Thus far, zero reported instances of playback issues. If there was one, that would be the end of me and Cine-Cache. It's appeal was full-speed content transfers while the show is running. According to GDC, the new SSD based PSDs will also allow full-speed transfers during playback.

We, as a rule, do not use anyone's built-in audio processor on a server, except the IMS3000 with a Dolby Atmos system where the IMS3000 is the Atmos decoder. We then feed that immediately into a Q-SYS system for the remainder of the processing.

So, all of our SR-1000s use nothing but AES3 audio. They feed mostly CP750s, at this time.

[Mark Gulbrandsen]

I pretty much had zero issues with GDC servers except the rare hard drive failure. Since it doesn't appear the log is catching what is possibly a data drop out, I would call GDC tech Support and tell them you need a loaner server while they look at the one you have. I am assumming you're not running off an external data server, or Hard Drive Array where there could also be a problem... Literally, the only GDC that had a legitimate issue was in Jackson, WY where the stick of memory on the MB would fail after warm up. On that one a reboot wouldn't restore operation. You had to power it down for a half hour and let it sit. I found the issue by exchanging parts between servers.

[Mike Blakesley]

Answering the questions that have been brought up:

It's not acting like it's "on pause" in the interface -- the 'play/pause' button remains unchanged when this phenomenon is happening.

Besides the interface in the projection booth, there is only one computer connected to the projector that has the GDC web interface on it, and I'm not touching it when this happens - so there's no issue with sticky buttons or ham-fisted operators.

I believe (but I need to confirm) that we operate off of the RAID array, not the internal storage on the SA1000. I'm not exactly sure why that is, but all the indicators say the RAID is healthy.

It does seem like a possible hard drive issue though, from what I'm reading here.

[Ryan Gallagher]

Cool, ignore my thoughts about play/pause triggering then, go down the disk/file-system health debug path.

[Mike Blakesley]

Turns out we operate off of the CineCache after all.

[Steve Guttag]

So, you have no external drives? No PSD or Enterprise storage...no eSATA cable? Do you know if it is a 2TB or 4TB CineCache?

[Leo Enticknap]

My suggestion at this point would be to send a log package to us-support [at] gdc-tech.com. GDC log packages are encrypted, meaning that only GDC can dig in to them. Hopefully, they should be able to figure out what is going on with Mike's server. Agreed with everyone else that the symptom matches that of a failing RAID drive, or a spinning rust RAID being more than around 80% full; but as this server has onboard solid state storage, the problem is obviously neither in this case.