IMS3000 infected with malware

[James Gardiner]

I totally disagree with removing the projection network from all external connectivity. This can be done securely with properly trained experts managing your network.

All cinemas have an Admin PC that the manager uses. They need to be able to utilise the TMS. They also need to be able to manually control screens in case of emergency or faults. This may expose the projection network to a degree, but the time saving and ease of dealing with faults and emergencies far outweighs this in my opinion. But I do agree this connectivity needs to be done in a restricted way so only those workstations needed for these processes have access.

Plus, I commonly have external hi level techs coming in to analyse major bugs I have found over the years. This is a very important capability to have. Being extremely technical and having written a lot of my own software, I am known for finding very particular problems. Edge cases that can only really be reproduced in position. for example, a lot of updates for the Barco lately have been due to major issues I have discovered. (i.e. the FTP implementation was very broken, and the HTTP server TLS algorithms were completely out of date causing connectivity failure with newer systems as there was no common algorithm. New implementations deprecated and removed those that allowed the connection to occur. Took me ages to work that out.)

[Steve Guttag]

As mentioned, in the "booth off of the WAN" plan, they still leave the TMS "exposed" as the sole point of connectivity. So, if the TMS computer, typical a server, has the needed software to interact with the booth equipment and you have access to the TMS (via VPN or other remote-in software), you sort of have what you need for connectivity. I, typically, have our remote-pc on-site and it can reach both the internet and the projection/sound equipment. It is headless (as is its user, at times) and has whatever a tech needs for that booth with most things able to be controlled with just a smartphone as an interface. I prefer to update via this too as it has the updating thing on the same network as the equipment and it is on a UPS to avoid potential power interruptions (and often on an ATS in case the UPS is the source of failure).

With us using QSYS more and more and AVoverIT being more of a thing...the Internet Of Things is just going to continue to grow and grow. I'm even more possessive of those networks. Now you have the sound, possibly of the entire complex, riding on those network(s).

Of note, Extron (and I'm sure others) will built a router into many of their control processors to provide a means of keeping AV separated from other IT devices. So IT gets to that controller and the the Controller serves as the gateway. Everyone is finding a way to navigate the "needs" of the various entities.

[Marcel Birgelen]

A good starting point is to not allow anything on your internal operational networks to directly communicate with the Internet, not even for automatic updates. A lot of stuff is screaming nowadays when you do that, because it dearly wants to phone home, for whatever legitimate or less legitimate reasons. Still, opening up this door makes it very easy for any attacker to continue their attacks as soon as they have found a way in. If there is just one narrow way into those networks, then at least your attacker needs to follow the same narrow path as you do. This narrow path can be far easily monitored and further restricted if necessary.

Me, personally, I'm starting to get sick and tired of this thing called "cloud". In the end, the cloud is just someone else's computer and is just another method to force you into a subscription model, where you own nothing anymore. Meanwhile, if they don't sell your personal data outright, it's just waiting until they'll leak it for you wholesale, due to their shitty security.

[Mark Gulbrandsen]

I seriously doubt this happened at the distributor, it was more likely at some point in this drives life that it made it's way home with some Booth Monkey who planted the garbage on the drive and then took it back to the theater. Then of course Leo's theater got the infection. I know that Deluxe and Technicolor quality controls all the drives they place content on before they leave the facility. So this drive was likely at another location some place after it left the distributors facility...

[Marcel Birgelen]

I seriously doubt this happened at the distributor, it was more likely at some point in this drives life that it made it's way home with some Booth Monkey who planted the garbage on the drive and then took it back to the theater. Then of course Leo's theater got the infection. I know that Deluxe and Technicolor quality controls all the drives they place content on before they leave the facility. So this drive was likely at another location some place after it left the distributors facility...

Mark, it didn't get there via the drive. I want to bet my grey kitty on it. Be warned though, although it's cute, it needs a lot of attention...

There is no way in hell that, even if there was an infection on the drive, the code on it got executed during the ingestion process. Zero. Zilch. Even Microsoft has "kind-of" disarmed their "autorun" crap a long long long time ago. You can put all the trojans and viruses in the world on the same disk as your DCP, hook it up to your IMS3000, and start ingesting the movie on it. Nothing bad will ever happen, unless maybe, the movie you're ingesting happens to be an Adam Sandler movie.

But as in stuff happening "at the distributor": I've seen some pretty big companies putting out promotional content in the past, containing malware...

[Mark Gulbrandsen]

Either way Marcel, what on earth is a server doing on a huge network like that? That's pretty rediclous in itself. Of all the servers and TMS's I put in, only one TMS was ever hacked, if you want to call it that. We were able to figure out through the log in data who it was and that it was an ex-employee of the theater. Changing everyone's log in info fixed that problem. Perhaps this location should switch to satellite downlink unless they run such obscure content that the companies providing satellite down link don't have the content available... At a University, that may include student made content or from very small distributers...

[Marcel Birgelen]

I haven't seen a malware infected IMB-style/integrated server yet, but I sure do have seen infected DCP-2000s, because someone actually managed to hook them up to the public internet with all ports open towards the Internet. Someone found the "DMZ option" in their router, and this way, he could easily access his servers from another site and transfer data in between those sites... Nothing beats the sheer ignorance of some of those people that operate in this industry, in this particular case: "No way I'm going to pay a few hundred bucks for a VPN."

How it entered remains speculation, but even then, you can already start to exclude the quasi-impossible, simply based on how Linux works. But, It may be something easy like the projectionist hooking up their infected notebook onto said network to transfer some content, it wouldn't be the first time... I hope Dolby will come forward and identify what the attack vector was, so it can be fixed. Like James indicated, something like this happening is more or less just a matter of time.

[Bruce Cloutier]

On the open Internet there is a background level amounting to dozens of packets per minute from worms on thousands of infected machines all trying to attack whatever is connected. You can liken it to a kind of air pressure. I have been observing this for weeks in an effort to harden our network stack even though typically we run in protected environments. They're probing random ports, working to overwhelm standard ports (SSH, HTTP, HTTPS, Telnet), attempting logins, etc.

We have been assigned fixed external IP addresses. I connected to one address that we had never used before and this background of malicious packet traffic was already there. It took less than 10 minutes for a bot somewhere to successfully login to one of our default accounts where, humorously, it tried to execute Linux scripts that our OS flipped a bird to. Interesting to watch.

Eventually that external pressure manages to find a path to somehow infect something even in the most sterile network environments. Lookup up Stuxnet if you are not familiar. That was a virus released into the wild that eventually found its way into a very specific targeted location to successfully complete its mission. We had PLCs here not normally connected to the network on which the virus was found. It wasn't here to mess with us specifically. We don't enrich Uranium.

So something got to Leo's installation. Even if you could find out precisely how it managed that you wouldn't be able to prevent it from happening again. It'll find a different way to get to the next system. IT departments are fighting an uphill battle. Universities have smart people. Unfortunately they wear both black and white hats. And you cannot even trust the so called industry cybersecurity experts (CrowdStrike included).

It is not just the Internet that is compromised.

[Steve Guttag]

And remember the IT's credo..."If you're doing your job, we're not doing ours!"

[Marco Giustini]

On the open Internet there is a background level amounting to dozens of packets per minute from worms on thousands of infected machines all trying to attack whatever is connected. You can liken it to a kind of air pressure.

a bit OT: and this is why it's more difficult nowadays to stop software updates! As much as you/we might not like that, that's the real world, unfortunately.

[Leo Enticknap]

Agreed that the projection management and media LANs as an island with the only points of entry being the remote access PC and/or the TMS, with multiple NICs (and maybe the Deluxe e-delivery box or equivalent, if one is installed), is ideal. That does, however, make those points of entry into potential points of failure, given that, if and once infected, the fox has then gotten in to the henhouse, so to speak. The upshot is that we have to be seriously careful about protecting those points of entry from malware infections.

I hope Dolby will come forward and identify what the attack vector was, so it can be fixed.

I'm not sure that public identification is such a good idea, because even if they come up with a hotfix to close that route of infection, not everyone will download and install it (I regularly encounter projectors and servers that have never been updated since being installed new 5-10 years ago), and so you don't want to advertise the specifics of the vulnerability. But I do hope that they are able to issue a fix.

[Marco Giustini]

I'm not sure that public identification is such a good idea, because even if they come up with a hotfix to close that route of infection, not everyone will download and install it (I regularly encounter projectors and servers that have never been updated since being installed new 5-10 years ago), and so you don't want to advertise the specifics of the vulnerability. But I do hope that they are able to issue a fix.

Whatever it is, it's likely a known issue with Linux which has been known for a long time and never fixed on the IMS because cinema servers don't normally have to deal with those things.

[Steve Guttag]

I've toyed with the concept, on the media network, either run it through a firewall or even VLAN off the various catch servers (e.g. DCDC) so, again, a narrow pathway exists to FTP the content to the TMS. Thus far, we have not been infected (that I am aware). I question how well such a firewall will stop an infection of a catch server is the source. As it is, our booth Wifi is not on the booth network...they are firewalled. In the event you are able to gain access to the WiFi...you are in an empty world with only other Wifi devices that might be on it (nothing I've installed). If you know where you are going, you can get there but the network won't help you any to figure it out.