10 Gb Ethernet in Doremi ShowVault

[Lucas Iaccarino]

Hello all,
I was wondering: It´s possible to install - through PCI or similar - a 10 Gb Ethernet controller in a Doremi ShowVault? I have really high traffic for DCPs and very little time to ingest - the server keeps working all time. Any of you have any experience??

Thanks!

[Marcel Birgelen]

Maybe, if the controller is being recognized by the Linux kernel, you can get it to work. But even if it works, I don't know how much sense it will make, as the new bottleneck will be the RAID array itself.

[Lucas Iaccarino]

Thanks Marcel,
I´m trying to get some information about the RAID-5 Write speed at Standards Doremi SW (with reccomended HDD) but not find anything. Which do you think could be the improve between 1Gb Ethernet and 10Gb Ethernet?
I´m myself at a difficult position and ANY improve over 30% will be great.

[Marcel Birgelen]

It's a standard LVM software RAID, the performance is dependent on a large number of factors, like the drives being used, the effective speed of the SATA bus, the CPU and the speed of the memory. Not all ShowVaults have been created equally, as there is a bit of variation in hardware components over the years.
Have you benchmarked your current setup and do you see your network card getting filled up? The theoretical maximum transfer rate of 1 GBit/s is about 125 MByte/second, but if you're hitting anything close to 100~110 MByte/second or anything around 900 mbit straight on your network interface on a sustained level, not just in peaks, then a 10G card could bring SOME performance boost, otherwise I don't think it's worth trying.

[Carsten Kurz]

Ingest is not just a file copy process. You may benefit from 10GE on a central library/LMS connected to many servers, but on the individual server, ingest speed will not just follow network bandwidth. What is your actual setup - to/from where and how do you ingest? How many devices are involved?
What are your specific operating conditions that imply so many ingests on a single server?

[Lucas Iaccarino]

Thanks for your answer Carsten. I understand what are you saying. The Server is projecting almost all time (during the day is projecting the screenings and during night we have rehearsals) so the no-playback time are really littles (maybe a couple of hours during lunch time or 30 minutes between screening and screening). They are almost all different films and I`m managing it under the premise "NO INGEST DURING PLAYBACK" to avoid any risks.
At this setup, we have a main server and a LMS at our office that send the films to this device.
My goal, as I will not get more NO PLAYBACK time is to improve the ingest speed.

[Carsten Kurz]

Can you overcome some of that problem by having more storage space on the Doremi? How large is the RAID currently?

[Marcel Birgelen]

What kind of ingest speeds are you seeing from your LMS to your ShowVault? Because the speed at which the LMS can serve will also be a potential bottleneck, including the network connectivity of the LMS.

[Joao Lopes]

You need a "Festival Solution": Two ShowVault's (or more, see picture from web).

ShowVault #1 to project.

ShowVault #2 to ingest.

When needed swap drives.

DOREMI Festival.jpg

[Carsten Kurz]

Some people have actually been able to run the Doremi Linux software in a virtual machine. It starts up nicely, even without a media block present. You can ingest to a RAID, and use the drive set in a real Doremi, or use it as a network ingest source for a real Doremi.

Or, you could just buy a second hand Supermicro X7SBE on ebay.

[Marcel Birgelen]

It does run as a VM, but it can't use the virtualized ethernet drivers, at least not under VMWare (maybe it can using KVM, Xen or some other hypervisor). As a result, your network speed is limited to about gigabit ethernet, which is fine for most applications, but may become a bottleneck if you want to use it as a super-fast ingest source. The nice thing about VM environments is, that you can easily spin-up a second instance though.

[Lucas Iaccarino]

Thank you all.
Dear Joao: That´s an interesenting Set-up. ¿Where it is?
The raid itself is big, and I ingest all I can, but sometimes the films arrives at last moment, so my objetive is to improve the ingest speed. Maybe the idea to add a NAS or another server to receive the films is a good one.

So Marcel, for your words I understand that adding a 10GBE is not a good option.

[Marcel Birgelen]

I don't think it's worth the hassle. But the only proof of the pudding is in eating it. It's not just installing the card, it's making sure the kernel supports it and the startup scripts can initialize it and Doremi's tooling play nice with the interface.

Given the age of the hardware involved, I think that if it works, the improvement will be mostly marginal. That's also why I asked to benchmark your current setup to begin with. Are you pulling close to 900 mbit/s on a continuous basis if you're ingesting content? If so, then it MAY be worth trying, if not, then it's best to work on the other bottlenecks first or maybe consider a new strategy to begin with.

[Carsten Kurz]

The question is, would a Doremi ingest DCPs from a network source any faster than from a SATA connected local drive? There may be differences between 'copying' a DCP that has already been verified on an external system and an actual ingest that needs to involve verifying, CRC checks, book-keeping, etc. I don't know if content transfers in some TMS enironments will bypass all these operations and simply copy, assuming that the content is 'ready to play'. There may actually be a difference between a standard FTP and an explicit TMS transfer, even if both are performed using FTP protocol.

Also, if you use an LMS, NAS, whatever, there will always be TWO copy processes involved, one from the distribution source to the NAS/LMS, and then the ingest from the NAS/LMS to the Doremi. A direct ingest onto the Doremi thus should always be faster. If your content arrives directly on your local NAS (e.g. via satellite or broadband), then that would be faster because one local copy is passed.

The trouble with a 10GBit card really is, you need to find someone Linux-savy enough to go through the hassels of finding the right card with the right drivers and perform the installation so that it doesn't interfere with the Doremis realtime playout functions. As far as I know, normal Ingest on the Doremis are bandwidth controlled to allow safe playback during background operations. I don't know if that throttle is also active when no playout takes place. It would become complicated if your network link actually becomes much faster. Have you ever considered Pingest, or does that occur too risky to you? You wouldn't need to do use it every time, obviously.

One way to do the network driver installation is to use the emergency USB startup method - that way, your normal Doremi USB-SSD remains unmodified and 'safe' for regular operation. Once it works, you may perform the same installation on the Doremis native startup-device. Another method would be to invest into a second hand Supermicro X7SBE board and do the installation and tests on it. Currently I actually see three second hand workstations using the Supermicro board being sold on ebay in germany for around 300US$.

[Leslie Hartmier]

I'll summarize the below just to make it fast, but read the rest if you want more clarity, less brevity - 10Gb/s network would not benefit your transfer speed because of the limitations of the rest of the hardware and software of the ShowVault4. Due to those limitations, I can fairly confidently say that even a 1Gb network is rarely if ever running at 100% capacity for content ingestion.

---

Just for reference, the ShowVault4's SATA controller's fastest write speed is 300MB/s if NOTHING is in place that slows it down (and as i outline next, there absolutely is). The RAID will definitely slow it down (writing to RAID5 is slower than writing to a single drive, but it is the tradeoff for increased data security), as will the basic laws of physics. The fact that it is fakeRAID aka software RAID will also slow down the write speed. In addition, if your RAID drives are mechanical drives (if they are at all sizeable, they are) and not SSDs, they will also cause a reduction in speed, since even when not in a RAID, the maximum speed of a 7200rpm mechanical hard drive is 80 to 180MB/s.

So, let's assume that your ShowVault4 will write at 300MB/s (as pointed out above, it absolutely cannot and will not). If that was a transmission speed of a network, that would be 2.4Gb/s, so the 10Gb/s network would only ever work at 1/4 its maximum speed, but as I pointed out, when not in a RAID, a hard drive will not go faster than 180MB/s or 1.44Gb/s, and in a RAID, it will be much slower than that.

In essence, ingesting from a SATA CRU drive would run (at its literal maximum which it still won't do) at more than twice what your gigabit network connection would, but even if you did manage to engineer a 10Gbit/s network, you'd still be limited by the maximum write speed of SATA II, the maximum write speed of the RAID drive, the reduction in write speed of the software RAID, the integrity check of the ingestion, not to mention the simply fact that nothing ever reaches 100% efficiency.

If ti were possible to upgrade the ShowVault to support the maximum speed possible, you'd have to replace the current drives with far more expensive (for the same size) SSDs (which will guarantee a lifespan since SSDs are based on number of writes, not number of hours), replace the SATA II (300MB/s) with a SATA III (600MB/s) controller, the CRU drive with a SATA III version, and replace the gigabit network ports with 10Gb network cards. In honesty, I'm going to go out on a limb and say that I doubt that the drivers for a new SATA III controller or a 10Gb/s network card are in the ShowVault4 OS, and even if they were, the software RAID would also need to support the additional commands of SATA III, and again, that may not be the case. An added point is that the CRU drives themselves use mechanical drives, and as such, will not operate as fast as an SSD, so you lose speed there too. No matter what you successfully changed, you would never reach or exceed 4.8Gb/s (the maximum speed of SATA III, if you managed to replace the SATA controller). Not that it is relevant to the conversation, the ShowVault's USB is USB 2.0, so limited to 480MB/s, which leaves it completely out of the running for ingestion unless that is how the content showed up from the distributor.

I've not even gone into the fact that the Doremi controls the transmission bandwidth for various operations.

My opinion is that running 10Gb/s network will not benefit you all that much anyway. As for whether ingestion via network or CRU drive is faster, I always intended to test it with various sizes of files, but I never did get around to it. That said, I see transferring via network from a TMS or another content server as convenience, and transferring from a CRU drive as likely the fastest.

If it is possible to get running, the place I would see 10Gb/s networks as useful is if you are transferring content to multiple servers at once.

I would be interested in speed tests if you did sort this all out, and if it works, perhaps let us know what hardware and processes you used.

Leslie Hartmier