Announcement

Collapse
No announcement yet.

Show Playlists keep randomly deleting on IMS2000

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Show Playlists keep randomly deleting on IMS2000

    Hello,

    So I have run into this problem several times in the past year with my playlists on the IMS2000. Sometimes playlists just randomly delete for no reason that I can understand. The error is usually the same "SPL no longer exists" in red on schedule page. I scheduled these playlists usually 1-2 weeks at a time. This playlist was a single show scheduled all week and next week. Its worked fine and there have been no issues. Tonight the show didn't turn on for the 7pm and I went to the booth and the playlist was gone off the schedule. I quickly made another playlist and got the show going. This has happened multiple times and I am not sure why? It doesn't always happen? Today my wife added an additional trailer to the playlist around 12pm. Saved it as normal and it showed up perfectly fine. Then tonight the whole playlist was gone. Any reason for this? Pic attached. Thanks!


    20250416_232108635_iOS.jpg
    Attached Files

  • #2
    Do you have any TMS type device talking to the unit via the API? Or is this all programmed via the Player-Web-Interface?

    I have only seen this myself when I delete SPLs via the API but have left the schedules pointing at these SPLs intact.

    Comment


    • #3
      We program all the shows through the web interface. Like I said it was fine yesterday morning. And then randomly the SPL’s got deleted. we have nothing else hooked up here that could affect the SPL’s that I know of.

      Comment


      • #4
        Just spitballing. I wonder if something is amiss at the file system level. Like it exists in a cache but once it is written to disk it becomes unreadable on occasion?

        pull logs and look for FS or read errors?

        Comment


        • #5
          I’ll give that a try. Like I said this happens from time to time at least half a dozen times in the last year where the SPL just goes missing for no reason.

          Comment


          • #6
            Since Leo brought up industry specific AI troubleshooting knowledge in the adjacent thread....
            I thought I'd see what ChatGPT thought of your situation, grain of salt, some elements could be hallucinated. But it's step one agreed with my log suggestion. And a few other suggestions seem believable, I'm not an IMS2000 user so can't confirm any of these tips.

            You're definitely not alone—issues with playlists randomly disappearing from the schedule on the Dolby IMS2000 have been reported by others, and it’s understandably frustrating, especially in the middle of a run. Here are a few possible causes and troubleshooting tips to help get to the bottom of it:
            ? Possible Causes of "SPL No Longer Exists" / Playlist Deletion

            1. Playlist File Corruption or Metadata Conflict
            • If the playlist references a DCP (movie or trailer) that has been removed, renamed, or altered (e.g., re-ingested or replaced), the IMS may automatically remove the playlist as invalid.
            • Even small changes like updating a trailer or reimporting a CPL can break the link, causing the system to think the SPL no longer exists.
            2. Manual Overwrites or Edits
            • When your wife edited the playlist earlier in the day, it’s possible the original version was overwritten, deleted, or corrupted during save/export—especially if she saved it with the same name or changed assets that were part of a scheduled version.
            • Check if a new version of the playlist was created, or if the original was renamed.
            3. Improper "Publishing" or Caching of Schedule
            • IMS2000 relies on a publishing system to schedule shows. If something fails during that process (e.g., a network hiccup or interface issue), the show may vanish from the schedule even if the SPL file still exists.
            • A reboot or maintenance event might also cause this issue to surface.
            4. System Software Bugs or Firmware Glitches
            • There are known bugs in some Dolby IMS firmware versions where playlists or schedule entries are dropped without warning. If you're not running the latest stable firmware, an update may help.
            5. Storage Volume or Mount Issues
            • If the SPL is stored on external or network storage (rather than local RAID), temporary disconnects can make the IMS believe the file no longer exists.
            • Check system logs for any storage warnings.

            Recommended Troubleshooting Steps
            1. Check System Logs:
              • From the web UI or Dolby TMS interface, look for system messages around the time of deletion.
              • Filter for messages about SPLs, CPLs, or asset removals.
            2. Compare Playlist Versions:
              • Check if a similarly named or slightly different version of the playlist exists.
              • If editing playlists after they are scheduled, consider versioning (e.g., "Evening_Show_v2").
            3. Use "Finalize" or "Lock" Options (if available):
              • On some systems, you can "lock" or mark a playlist as final, helping prevent it from being altered or deleted.
            4. Avoid Editing Scheduled Playlists:
              • When possible, make a copy of the existing playlist, apply changes to the copy, and then update the schedule manually to use the new version.
              • This prevents invalidation errors caused by mid-schedule edits.
            5. Update IMS Firmware:
              • Check your Dolby IMS2000 firmware version. Some versions (especially early 2020s builds) have issues with playlist memory management or SPL corruption.
            6. Contact Dolby Support:
              • If this continues after you've ruled out user error and asset changes, Dolby support can review system logs and give firmware-specific advice.

            ? Best Practices to Avoid Future Deletions
            • Name playlists clearly and with versions.
            • Avoid modifying scheduled playlists.
            • Document changes to playlists after scheduling.
            • Back up playlists via the TMS interface when finalized.
            • Set up a daily system log download for tracking unusual behavior.


            Would you like help checking what firmware version you’re on or where to pull the relevant logs to see if a deeper cause is visible?

            Comment


            • #7
              The one about certain edits/actions causing the SPL to be overwritten under a new UUID seem particularly believable. If the Scheduler references UUID links only then this would cause breakage. The tip about saving edits out to new versions and then replacing old versions in scheduler seems logical enough.

              If I had to guess along these lines, edits that affected the TRT of the SPL would perhaps be incompatible with already scheduled times/events, they could avoid this incompatibility by intentionally breaking SPL links when TRT changes?

              You could test this next time it happens (or manually) by noting the UUID the scheduler it is looking for, and then go to your edited SPL (if it still exists at all), and see if the UUID is in fact now different than the one the scheduler was expecting?

              On the surface, other than TRT reasons, it also may be due to wanting to utilize the UUIDs to track exactly what assets have played, if you are able to swap CPLs around and have the same SPL UUID, you lose some of that uniqueness in the logging etc.
              Last edited by Ryan Gallagher; 04-17-2025, 01:24 PM.

              Comment


              • #8
                SPLs just down start changing or deleting.
                I would consider replacement of the boot DOM. Maybe it has become corrupt. On reboots, loosing inodes and causing files to delete from under it, causing the SPLs to disappear. Or something like that.

                Comment


                • #9
                  Likely a false alarm, but I'm wondering about a malware infection. Since starting that thread, I've encountered the same infection in an IMS2000 as well: identical symptoms.

                  In any case, could Mike download a "detailed report" log package from his IMS2000, upload it to the log analyzer, and post the link here? If it lets him do that through the web UI, then the problem isn't the malware infection I've encountered, because the infected report package is too big for the web UI to handle. In that scenario it'll think about it for 3-4 minutes and then give you a "could not generate log package" error. The USB drive method won't work either: you have to generate the report using the command line and SFTP method.

                  Comment


                  • #10
                    Originally posted by Leo Enticknap View Post
                    Likely a false alarm, but I'm wondering about a malware infection. Since starting that thread, I've encountered the same infection in an IMS2000 as well: identical symptoms.

                    In any case, could Mike download a "detailed report" log package from his IMS2000, upload it to the log analyzer, and post the link here?
                    So how do I get a detailed report from the IMS2000? I’ve never done that before?

                    Comment


                    • #11
                      Originally posted by Leo Enticknap View Post
                      Likely a false alarm, but I'm wondering about a malware infection. Since starting that thread, I've encountered the same infection in an IMS2000 as well: identical symptoms.

                      In any case, could Mike download a "detailed report" log package from his IMS2000, upload it to the log analyzer, and post the link here? If it lets him do that through the web UI, then the problem isn't the malware infection I've encountered, because the infected report package is too big for the web UI to handle. In that scenario it'll think about it for 3-4 minutes and then give you a "could not generate log package" error. The USB drive method won't work either: you have to generate the report using the command line and SFTP method.
                      In all cases since that malware thread, did they require Dolby sending out a new boot DOM like that first instance did?

                      Comment


                      • #12
                        I wrote a cheat sheet for this, but thanks to the screenshots in it, it's over the 200K limit for attaching here, even if I render the PDF squished as small as possible. So here goes for a copy-and-paste...

                        Dolby IMS3000
                        Log Download and Analysis Instructions

                        Note


                        These instructions were written using web UI screenshots from an IMS3000 running version 3.5.13 or later of the software/firmware bundle. They can also be used to download a package from an IMS3000 running older versions, or other Dolby/Doremi servers:
                        • IMS2000 (including the NEC-badged variant NP-90MS02)
                        • IMS1000 (including the NEC-badged variant NP-90MS01)
                        • ShowVault/IMB
                        • DCP-2K4
                        • DCP2000

                        The web interfaces will look significantly different, but the functions described below do exist within them.

                        Tools and Equipment Required

                        You will need the following:
                        • A laptop or desktop computer with connections both to the projection management LAN and the Internet, with a web browser installed, and to which you have administrator access.
                        • [possibly] a USB flash memory stick
                        • [possibly] a FTP client, e.g. FileZilla, installed on the computer
                        • [possibly] a SSH client, e.g. PuTTY, installed on the computer.
                        • The necessary login credentials for your IMS3000.

                        Introduction

                        In order to assist you with troubleshooting, MiT and/or Dolby technicians may ask you for a “detailed report” log package, which can be downloaded out of your server. These instructions explain how to do that.

                        Method 1 – via the Web User Interface (web UI)
                        • Log in to the IMS3000, using any account or privilege level.
                        • Navigate to Diagnostics → System Information, and click on “Detailed Report.”
                        • After a short delay, your computer will ask you for a location to save the file. Once it is saved, email it to whoever has asked for it as an attachment.
                        • If the report file is too big to attach to an email, please follow the instructions for uploading it to Dolby Log Analyzer, below.

                        dataurl781583.png

                        Known Bug With Method 1

                        There is a known issue with the IMS3000 software, whereby if the report package file exceeds 20 megabytes, the web UI will not let you download it. It will freeze, and you will need to reboot the IMS3000 before it will work again. This is likely to happen if your IMS3000 has been in use for 3-4 years or longer. If it does happen, reboot your IMS3000, and then try method 2.

                        Method 2 – Using a USB Flash Memory Stick

                        The IMS3000 is very picky about the partition table and volume format of drives it will accept. This procedure will create you one that is known good.
                        • Using a Windows computer, open a command prompt window (in administrator mode, if you are not logged in to an account with administrator privileges).
                        • Insert your USB stick into one of the computer’s jacks, and enter the following sequence of commands. NOTE: in the “list disk” readout, you will need to identify which one is your USB stick, and use the correct number in the “select” command. You can do this by identifying the drive that has the same capacity as your stick. In this example, we have an 8 GB drive, and Windows has called it Disk 7.

                        dataurl781586.png
                        • Using File Explorer, create a folder called “doremi” at the root level of your USB stick, and a folder called “report” within that, as so:

                        dataurl781584.png
                        • Eject this drive, and insert it into any of the USB jacks on the faceplate of the IMS3000.
                        • Wait for ten minutes, then remove the drive. Now reconnect it to your computer. The report package should appear in the “report” folder, as so:

                        dataurl781584.png
                        • Email it to whoever has asked for it as an attachment. If the report file is too big to attach to an email, please follow the instructions for uploading it to Dolby Log Analyzer, below.
                        Occasionally, this method doesn’t work either, in which case you’ll find the “report” folder empty. In that case, you need to progress to method 3.

                        Method 3 – Command Line and FTP
                        • Using a SSH client (in this example, PuTTY), connect to the management LAN’s IP address for the IMS3000 on port 22:

                        dataurl781585.png
                        • Enter the command sh /doremi/sbin/report.sh and press enter.
                        dataurl781587.png

                        It may take up to ten minutes to generate the report. When this is done, you should see something like the following:

                        dataurl781587.png
                        • Now open a FTP client. In this example, FileZilla is being used. Configure a connection as follows (substitute the IP address of your server for the one in this screenshot, obviously):
                        dataurl781587.png

                        dataurl781588.png

                        You should see the report package available for download, as so:

                        dataurl781588.png
                        • Email it to whoever has asked for it as an attachment. If the report file is too big to attach to an email, please follow the instructions for uploading it to Dolby Log Analyzer, below.

                        If the Report is Too Big to Emaildataurl781587.png
                        • Click on “share” (upper right)
                        • Copy the “Analysis” link, and then paste it into an email to whoever has asked for it

                        Originally posted by Ryan Gallagher
                        In all cases since that malware thread, did they require Dolby sending out a new boot DOM like that first instance did?
                        Yes. I've found that malware infection in two IMS3000s and one 2000 now, and Dolby's advice was to replace the boot flash drive outright in all three cases. Theoretically, writing a clean system image to the existing one should have had the same effect, but given the history of those boot chip drives failing anyways, outright replacement is, IMHO, the best option.

                        Last edited by Leo Enticknap; 04-19-2025, 10:19 AM.

                        Comment


                        • #13
                          Originally posted by Mike Reed View Post
                          Hello,

                          So I have run into this problem several times in the past year with my playlists on the IMS2000. Sometimes playlists just randomly delete for no reason that I can understand. The error is usually the same "SPL no longer exists" in red on schedule page. I scheduled these playlists usually 1-2 weeks at a time. This playlist was a single show scheduled all week and next week. Its worked fine and there have been no issues. Tonight the show didn't turn on for the 7pm and I went to the booth and the playlist was gone off the schedule. I quickly made another playlist and got the show going. This has happened multiple times and I am not sure why? It doesn't always happen? Today my wife added an additional trailer to the playlist around 12pm. Saved it as normal and it showed up perfectly fine. Then tonight the whole playlist was gone. Any reason for this? Pic attached. Thanks!


                          20250416_232108635_iOS.jpg
                          I see this issue all the time, but not when using the web interface, only when using a TMS and it makes a change to the SPL. Regardless, as others have hypothesized, its because there has been a change between the version of the spl that was scheduled and the one that is current. Even a slight change to the spl can cause this (adding in end credit cue times after its already been scheduled. ) It should however only happen if the spl is loaded at the time of change. If you load a different spl (I have a generic "lights up cue" spl that I load) anytime you make changes it should prevent this from happening.

                          Comment

                          Working...
                          X