Film-Tech Forum ARCHIVE: DCP-o-matic with multiple encoding servers.

my profile | my password | search | faq & rules | forum home

»	Film-Tech Forum ARCHIVE » Operations » Digital Cinema Forum » DCP-o-matic with multiple encoding servers.

Author

Topic: DCP-o-matic with multiple encoding servers.

Stephen Furley
Film God

Posts: 3059
From: Coulsdon, Croydon, England
Registered: May 2002

posted 06-10-2014 02:10 AM

Has anybody got this to work, and if so how do you set it up? Should it just detect all available machines, or is it necessary to put the ip addresses or host names in manually? Nothing comes up if I look at the help.

Under preferences I've tried putting in the IP addresses of the connected machines, and also ticking the 'Use all available servers' box, but it doesn't use any of them. Under Tools|Encoding servers nothing shows up. Machines are connected via Gigabit Ethernet, and are able to ping each other. They are connected via a layer 2 switch, an Extreme x350-24t.

This is for enconing some students' work for a screening in London next week. One of them has produced something in a weird format, 4056 x 1708 (2.64:1). I assume that this would have to be letterboxed within a 2.39:1 container to preserve the original aspect ratio.

| IP: Logged

Carl Hetherington
Film Handler

Posts: 93
From: York, North Yorkshire, England
Registered: Jul 2012

posted 06-10-2014 04:45 AM

Is this on Windows/OSX/Linux? What version of DCP-o-matic? Do the machines have any firewalls enabled?

If it's easier you can email me at carl@dcpomatic.com and we can try to work it out.

| IP: Logged

Stephen Furley
Film God

Posts: 3059
From: Coulsdon, Croydon, England
Registered: May 2002

posted 06-10-2014 05:19 AM

I'm having to run it on one machine at the moment to get it finished in time. It's running on OSX. Will get back to you later this evening when we've got this rush job out of the way.

I moved it onto a brand new high-end machine, which is about twice as fast as the previous one which I was using, but it fell over half way and I had to re-start it. It's never done that before on the old machine, nor on my laptop.

Cannot contact student to find out if this really is supposed to be 2.64, but I don't think it is, faces look slightly squashed at that ratio, but more natural if the image is stretched to vertically to fill 2.39. There are credits just over half way through the file, then several minutes of black, then a few more seconds of picture. I've no idea what's going on, but I've learned from past experience not to rule out the possibility that a film looks odd because hat's how the director wanted it to look.

| IP: Logged

Carl Hetherington
Film Handler

Posts: 93
From: York, North Yorkshire, England
Registered: Jul 2012

posted 06-10-2014 05:28 AM

quote: Stephen Furley
I moved it onto a brand new high-end machine, which is about twice as fast as the previous one which I was using, but it fell over half way and I had to re-start it. It's never done that before on the old machine, nor on my laptop.

Any more details of this falling over (messages, the "log" file) might be useful in tracking it down.

| IP: Logged

Stephen Furley
Film God

Posts: 3059
From: Coulsdon, Croydon, England
Registered: May 2002

posted 06-10-2014 05:30 AM

No message; it just closed. It's got past the same point now, and seems ok. I've got to do a 4k version as well.

| IP: Logged

Brad Miller
Administrator

Posts: 17775
From: Plano, TX (36.2 miles NW of Rockwall)
Registered: May 99

posted 06-10-2014 02:58 PM

Stephen you probably have a multi-core processor. Go into the preferences and tell it not to use all of them. For example if you have a 4 core processor, tell DOM to only use 2 cores.

Likewise on the encoding servers if you have more than a few on the network and you tell it to "find and use all", it can crash from overload as well. (They won't show up in the preferences box, but they will be found.) Don't forget you have to have the encoding servers actually running on the other machines and they must be on the same subnet. The encoding servers if you maximize it will tell you what frame and machine it is processing from.

| IP: Logged

Stephen Furley
Film God

Posts: 3059
From: Coulsdon, Croydon, England
Registered: May 2002

posted 06-10-2014 04:03 PM

Thank you Brad. In the end we did the job on a single machine, but after it was finished we had another go. A colleague fished an old but quite high spec Dell server out of store and installed on it some version of Linux and 'DCP-o-matic server', which I've never heard of, but it has no graphical user interface. This connected to the brand new iMac running the normal version of the software.

We tried running the job again. My laptop can normally process about 2 fps, and the new iMac about 4.5. The new iMac together with the old Dell was still showing on the iMac screen as doing about 4.5, but the processing took just over half as long, and the CPU utilisation on the Dell was in the high 80s percent. The log showed some frames processed remotely, so it was working. I guess the frames per second displayed relates only to those processed on the local machine.

This is something which we will be doing more of in the future; DCP seems to be rapidly becoming the preferred format for submitting students' work to festival screenings. Six student's work was screened at Bradford in April, all in this format.

What would you say would be a reasonable maximum number of machines to use; three or four maybe? The room where we would normally do this has twelve, but I'm guessing that it would not be a good idea to use that many.

| IP: Logged

Carsten Kurz
Film God

Posts: 4340
From: Cologne, NRW, Germany
Registered: Aug 2009

posted 06-10-2014 04:25 PM

I have done a bit of testing on networked rendering - the problem is that DCP-o-matic GUI, the main application, sends individual uncompressed frames of the source material to the render clients. Every thread on these machines encodes one frame at a time. If you have a 4,6,8...threaded machine on the network, the 'master' needs to supply 4,6,8... uncompressed frames. Add another machine, and you need to supply twice as many uncompressed frames. This very quickly saturates a gigabit network, and the render clients can not be kept as busy as they could with a local encoding instance. A single uncompressed frame, 12Bit, can be as large as 10MByte at 2k resolution. A gigabit network can supply 80-100MByte/s if your lucky. So you can imagine how quickly the network throughput becomes your bottle-neck.

Make sure you have a 64bit version of windows running - you can max out the number of threads there. Under 32Bit versions, using more threads than physically available may crash. With a 64Bit windows, best performance e.g. for a dual-core, HT CPU is 4 threads. So, double the number of threads/core if you have a HT CPU.
I have rarely seen DCP-o-matic crash during encoding, but I usually keep the encoding machine alone.

Of course, if you do a lot of different projects, you can make good use of multiple fast machines by encoding every project on a separate machine.

So, in my opinion, network rendering only makes sense with a lot of slow machines on a gigabit network, e.g. a classic office environment. The faster the machines, the less improvement in overall rendering speed you will get. The economical sweetspot currently is a fast hexacore HT CPU, e.g. an i7-4930K. Only with expensive 8/12core/DualCPU mainboards will you be able to achieve more than 10-12fps.

- Carsten

| IP: Logged

Brad Miller
Administrator

Posts: 17775
From: Plano, TX (36.2 miles NW of Rockwall)
Registered: May 99

posted 06-10-2014 04:41 PM

Or the simpler answer, on an "average" PC made within the last 5 years or so, I wouldn't recommend more than 4 encoding server units. More than that and you have crashes and a bottleneck. Again I'm talking average. Your miles may vary. If you do have a spare machine or two, definitely use them though.

So long as you don't have more than one "master" on the network, there is no need to specify the IP addresses of the encoding servers. But let's say you had 8 computers to play with. You could make 1 and 5 masters and hard-code 2, 3, 4 as encoding servers for #1 and 6, 7, 8 as encoding servers for #5.

| IP: Logged

Marcel Birgelen
Film God

Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012

posted 06-10-2014 05:30 PM

quote: Carsten Kurz
This very quickly saturates a gigabit network, and the render clients can not be kept as busy as they could with a local encoding instance. A single uncompressed frame, 12Bit, can be as large as 10MByte at 2k resolution. A gigabit network can supply 80-100MByte/s if your lucky. So you can imagine how quickly the network throughput becomes your bottle-neck.

Buy a cheap managed switch that supports LACP and some decent NICs in your "server" that also support LACP and you can bundle multiple uplinks into one "virtual" uplink. It can be done really cheap those days.

But you'll probably notice that, once your network isn't the bottleneck anymore, your disks will become the new bottleneck.

| IP: Logged

Carl Hetherington
Film Handler

Posts: 93
From: York, North Yorkshire, England
Registered: Jul 2012

posted 06-11-2014 03:31 AM

quote: Brad Miller
Likewise on the encoding servers if you have more than a few on the network and you tell it to "find and use all", it can crash from overload as well. (They won't show up in the preferences box, but they will be found.) Don't forget you have to have the encoding servers actually running on the other machines and they must be on the same subnet. The encoding servers if you maximize it will tell you what frame and machine it is processing from.

Hi Brad, are you still using 32-bit Windows? Did we ever prove that lack of memory was the cause of lots of your crashes?

quote: Stephen Furley
The new iMac together with the old Dell was still showing on the iMac screen as doing about 4.5, but the processing took just over half as long, and the CPU utilisation on the Dell was in the high 80s percent. The log showed some frames processed remotely, so it was working. I guess the frames per second displayed relates only to those processed on the local machine.

This is strange. The frames per second display should describe the overall rate (including encoding servers). Do you have any part of the log file from this run?

quote: Carsten Kurz
The problem is that DCP-o-matic GUI, the main application, sends individual uncompressed frames of the source material to the render clients.

Recent versions send compressed files to the encoding servers if you are using image files (e.g. TIFFs as input). At some point I will get the main instance to send compressed data from movie files, too, though that is more complicated from a programming point of view.

| IP: Logged

Stephen Furley
Film God

Posts: 3059
From: Coulsdon, Croydon, England
Registered: May 2002

posted 06-11-2014 05:04 AM

Carl, I think the log is still on the machine. I'll have a look later and e-mail you a copy if it is.

| IP: Logged

All times are Central (GMT -6:00)

Printer-friendly view of this topic

The Film-Tech Forums are designed for various members related to the cinema industry to express their opinions, viewpoints and testimonials on various products, services and events based upon speculation, personal knowledge and factual information through use, therefore all views represented here allow no liability upon the publishers of this web site and the owners of said views assume no liability for any ill will resulting from these postings. The posts made here are for educational as well as entertainment purposes and as such anyone viewing this portion of the website must accept these views as statements of the author of that opinion and agrees to release the authors from any and all liability.