Doremi ShowVault - Error /doremi partition is full

[Pietro Clarici]

Has anyone ever encountered this? The system can playback currently, but looks like I cannot write on the ShowVault system disk anymore. Every action that involves writing - from indexing new CPLs to loading KDMs and saving SPLs - does not work. I can access the file system both via the internal browser and FTP, but I can't find tmp folders or orphan files and I don't know if there is a terminal command I can run to clean the partition up and get it running again. Or maybe this means that the system disk is failing and I need a new one?
Thanks!

Pietro Clarici
Cinema Clarici - Italy

[Pete Naples]

Was this server by any chance installed by or under the VPF provided by XDC (now CinemaNext)?

[Pietro Clarici]

No, I bought it in 2011 from a dealer in Italy.
I'm asking here because my tech is swearing he has not seen this before, and could only suggest switching to a new SSD. I thought I'd check if we're both missing an easier solution.

[Marcel Birgelen]

Have you tried the "Disk Cleanup" in the "Diagnostic Tool" already?
What about running a file system check on it?

[Pietro Clarici]

Yes, a few times, unfortunately doesn't seem to fix it. I'm not sure if Disk Cleanup checks the InnoDisk SSD or just the RAID array.

Just ran a "df" command from terminal and it looks like the /doremi partition (/dev/sdg8) is indeed 100% used, so the disk could actually be functional, just full, thus blocking all new files. Any idea on where the missing space could be?

[Leo Enticknap]

Like your tech, I haven't seen this before, either. I wonder if logs and stuff have filled up the SSD, given the age of the server. It might be possible to go in with an FTP client and clear out the relevant folders, but I'd need to do some research to figure out where they are. Maybe ask Dolby?

[Pietro Clarici]

Thanks, I'll try the Dolby route via the reseller - let's just say they are less than helpful when contacted directly.
I already located the log folder and there are quite a few compressed .gz archives which are clearly not in use, but I cannot delete them even with admin access.

[Marcel Birgelen]

Sometimes, very large files that overflow a partition, can cause a partition to be 100% full, but show no files. This should vanish if you reboot the machine. If that doesn't help, I'm affraid your flash memory is defective.

[Pietro Clarici]

I'm not sure if this is it, but compared to my DCP2000s there is a very large file in /opt/dbs called smpte.dbs, I'm assuming it could be a database of all the SMPTE material that has been indexed over time. It's 300MB on this ShowVault and around 8MB on the other units, which are older. I'm almost tempted to back it up somewehere, delete all content and then switch it with an empty file to force a rebuild from scratch.

[Marcel Birgelen]

It's a binary database file. If you back it up, also note down the ownership and permission flags on it. It's hard to say if the Doremi software will recreate it if it's missing, but it may be worth a try.

[Carsten Kurz]

I never heard about such an issue before either. In this case, I would also guess there is an issue with the boot SSD, it seems to be old enough to cause problems. However, you could always try to create an emergency USB boot stick. That could work even from this USB SSD. It's a simple command. If there is a hardware issue with the SSD, it might show that everything works when starting off the USB stick. You should get a normal 4 or 8 GB USB stick for this. There is a technote from Doremi explaining the cloning process. I have a Doremi running from a USB stick for years now without issues. You could even have one or two more USB sticks for backup purposes (e.g. before you upgrade a system). As far as I remember, these Doremi SSDs, now even from Dolby, are not too expensive. You should create a system backup before, so you can restore your server settings after the SSD swap. I remember that Doremi at some point in time went from 2GB to 4GB system SSDs.


You have no write access to the SSD using normal procedures as long as the system is running off it. You need to remount the file systems you want to alter in RW mode first. This is not a plain vanilla Linux system.

[Marcel Birgelen]

The remounting can be done via:

mount -o remount,rw <mount location or full path to device name>

eg:

mount -o remount,rw /doremi

Also, if I remember correctly, the /opt partition is on md1, which is on the RAID array and not on the flash/SSD. So, clearing the dbs file should not fix the /doremi partition being full.

Edit: Which got me thinking... Since the partition is usually mounted read-only, I don't think anything is actively writing stuff to the /doremi partition during content ingestion. That partition is meant to be the "image" part of the configuration anyway.

If you can miss the content, have you simply tried to re-init the RAID?

su -
<Enter root password>
sh /doremi/sbin/reinit_raid.sh

[Marco Giustini]

You’ve probably checked this already but sometimes - not on Doremi servers though - the logs pile up a bit. Check into /car/log in case you had something weird happening there.

[Pietro Clarici]

Quick update, thanks for all the advice: turns out all databases were in fact corrupted and could not be written or purged - this is unusual because I'm on 2.8.x and DB recovery should be automatic since 2.6.4.
Anyway, I was able to stop and remove them via terminal, after a reboot all looks fine and the /doremi partition is back to its normal size.
In case anyone needs this in the future, log in as root and type:

/doremi/etc/init.d/drmldbd.init stop 
rm -rf /doremi/opt/dbs/doremi.db
rm -rf /doremi/opt/dbs/cpl.db
rm -rf /doremi/opt/dbs/smpte.db 
reboot

[Valter Almeida]

Hey guys
I have the same problem, I already ran the code that @Pietro Clarici said but the problem remains.
Does anyone know how to solve this problem