Ethernet Switches

[Harold Hallikainen]

Now and then, I'm finding Ethernet packets showing up at a device that they should not be showing up at. For example, I had a system fail when it was flooded with Atmos audio packets (I think they were supposed to be going from a server to a CP850). Based on my understanding of Ethernet, a switch should send packets out a port that a device is connected to only if the MAC address of that device matches the destination address of the Ethernet packets or the Ethernet packets is a broadcast packet.

Here's my understanding of how Ethernet should work:

A device on the network has a MAC address, an IP address, a subnet mask, and a gateway IP address.

The device uses ARP requests to find the MAC addresses of devices on the network based on IP addresses. When it needs to send a packet to an IP address, it will send a broadcast ARP request (unless it has done so previously) asking "Who has IP address?". The device with that IP address responds saying it has that IP address and provides its MAC address. Then, the original device wanting to send a packet to a local IP address makes an Ethernet packet addressed to that MAC address that includes the IP packet. The switch sends the packet to the correct port since it had previously heard that MAC address on that port (such as the ARP response having the packet source MAC address). So, the switch ideally sends packets only out the port that has the device with the destination MAC address (or broadcast Ethernet packets).

When a device on the network sends a packet to an IP address outside the subnet range, the device sends the packet to the MAC address of the gateway device (a router), which then sends the packet on. Since all Ethernet packets are "routed" by their MAC address (not the IP address), the device similarly has to do an ARP request asking who has the gateway IP address. The gateway (router) responds with its MAC address. The device then sends all Ethernet packets destined to an IP address outside the subnet to the gateway MAC address.

So, in my opinion, a router can send packets that are destined to multiple IP address (those outside the subnet) towards their final destination. This is often sending the packets out a DSL or cable modem. A switch knows nothing of IP addresses. It just "routes" packets based on MAC addresses. But, nonetheless, it should send only packets destined to the device MAC address and broadcast packets out the port driving that device.

Is this (approximately) correct? Why is my device getting flooded with packets destined to another MAC address?

Harold

[Marcel Birgelen]

In classic ethernet, there is either directed traffic or broadcast traffic. Broadcast traffic reaches all devices within the same "broadcast domain".

What you're seeing is most likely multicast traffic. Multicast is a complicated beast. In an ideal world, your switch would support "IGMP snooping" and know exactly who is member of what multicast group (essentially just an IP address in the multicast range) and who not.

Many if not most cheap switches don't support "IGMP snooping", so the systems involved fall-back to broadcast traffic. This broadcast traffic will then reach all "subscribers" on the same "broadcast LAN".

[Harold Hallikainen]

Thanks! I'm trying to find the capture from years ago where I found Atmos traffic showing up on a closed caption transmitter. Recently, I've heard of issues on "high traffic" networks. I've requested a WireShark capture on the port going to the captioning transmitter (typically by putting a hub between the switch and the cation transmitter and then have another computer running WireShark and capturing everything off the hub). I have not yet received a capture or a log out of the system. But, I'm just trying to be prepared. Where would multicast be used in a theater? Most of the stuff I can think of would be point to point (with ARP being broadcast).

[James Gardiner]

Unfortunately, I have seen devices that act badly often, sending out broadcast storms. Its a good way to bring down a whole segment. I common reason we saw cinelink noise on the screens in the early days with Series 1, doing TLS handshakes to send the keys to the enigma in the S1 projectors. The TLS connection would be smashed by a broadcast storm, fail to get the keys to the projector, and so unable to decrypt resulting in a non-unencrypted picture that shows as cinelink NOISE.

Its also a good reason why only qualified devices should be plugged into a projection network.
Also, in general, with time sensitive traffic. For example, Audio, this is a major reason they suggest keeper this traffic to their own physical segment of a network. i.e. separate dedicated switches of VLANed into virtual segments. Ensuring only devices that are required on that physical segment are the only ones on it.

Another example of a poor implementation is Barco network implementation. If you plug certain ports into the same segment, as it does this internal forwarding magic to make multiple devices work on the same IP. This "Feature" has a tendency to cause broadcast storms if you plug in the wrong ports. Takes the whole segment down. I was very surprised about this as there are tools like spanning tree protocol they could have implemented to mitigate this choice. I would prefer they just kept it all separate, like what is required if using a Barco/Dolby combination of projector server.

[Harold Hallikainen]

Thanks! I'll see if I can get them to do a Wireshark capture so I can see what's going on.

[Marcel Birgelen]

Atmos traffic includes multicast components. Like James indicated, those kind of traffic streams are best put into a dedicated broadcast domain e.g. via VLANs or separate physical switches. As for what's better: doing IGMP snooping on switches that support it v.s. simply going the broadcast route... I'm still torn after all those years, I've seen IGMP snooping going wrong, so that's why I generally try to avoid it. In order to avoid other devices getting overwhelmed with broadcast traffic that doesn't belong there, I usually create separate VLANs for separate traffic flows. One nice thing is that on many better managed switches, you can give priority to certain VLANs. It's always a good thing to give your real-time traffic preference above "normal LAN traffic".

[Steve Guttag]

I think you are getting good advice here. Multicast flooding is what I was thinking too...with my feeble networking skills. In the world of Q-SYS, IGMP snooping can be key to device discovery and is recommended. Plus, AES67 (which is what QLAN is a part of) operates as multicast (though AES67 can be unicast). Since you mention an interaction with Dolby Atmos...that uses AES67 to move the audio from the server/processor to downstream devices (DMA amplifier, DAC converters and Q-SYS). That is where you are getting your multicast traffic. Oddly, Dolby's products do not support IGMP snooping (dumb) so, if you have that turned on the Dolby stuff, you'll lose the stream. So, I wonder if you were to configure the switch with IGMP snooping (and you need one IGMP querier) but turn off IGMP snooping on the switch port that connects to the Atmos processor, if your flooding issue goes away.

As for my schemes...I do separate LANs rather than separate VLANs. The extra overhead of additional cables and switches is minor/low cost and I never have one form of traffic interfere with another. Fly-by-wire with Q-SYS is already a stranger to cinemas...I don't ever want any audio hiccups. I run redundant networks and they are separate networks. On the smaller systems, I'll use QLAN-B as a dual duty redundant QLAN and control and then let my router route the control stuff over to the main LAN for the booth equipment. The only downside to that is a single screen is not self-contained. That is, for a server to tell the sound to change volume...it is dependent on the router to send that command back to the QLAN. It hasn't proven to be an issue though.

[Harold Hallikainen]

Thanks for the comments! I know almost nothing about multicast. I did a little work with BLU link where we made a D/A that would pull 16 channels of audio from BLU link ( http://ftp.uslinc.com/Products/DAX-1...1.4_141003.pdf). BLU link is interesting in that it's a loop instead of star network. The DAX-16 was configurable as to what audio channels to pull from the network instead of requiring the source (the CP850) address which devices were to receive the audio.

For these applications where it is desirable to have multiple devices receive the same audio or have them be able to cherry pick channels off the network, I can see the use of UDP broadcast. But, I think this should be on a separate wire, not the one carrying closed captions and automation.

I don't know for sure if the current problem is due to audio on the network. I have yet to receive logs from the IRC indicating what the problem is or Wireshark captures where the source of the apparent network flooding can be identified.

Thanks for the ideas!

Harold

[Steve Guttag]

Dolby Atmos audio should not be on the normal control/management network. All of the Dolby Atmos processors have a separate NIC for Atmos audio. That doesn't stop someone from slamming it all into a switch making a flat network. After all, Dolby proudly proclaims that if using just their equipment (e.g. CP950A to a set of DMAs), you need not bother with managed switches or fooling with PTP configurations. Someone may have mistakenly made the leap to think that it can all reside on the same switch as the normal control/management LAN.

[Marcel Birgelen]

As for my schemes...I do separate LANs rather than separate VLANs. The extra overhead of additional cables and switches is minor/low cost and I never have one form of traffic interfere with another. Fly-by-wire with Q-SYS is already a stranger to cinemas...I don't ever want any audio hiccups. I run redundant networks and they are separate networks. On the smaller systems, I'll use QLAN-B as a dual duty redundant QLAN and control and then let my router route the control stuff over to the main LAN for the booth equipment. The only downside to that is a single screen is not self-contained. That is, for a server to tell the sound to change volume...it is dependent on the router to send that command back to the QLAN. It hasn't proven to be an issue though.

Both strategies are valid strategies. It's important that the people that maintain the system know how it works. If you don't have in-depth network knowledge or the system needs to be managed by people without that knowledge, it's best to stay away from complicated VLAN setups, because stuff can become an unmitigated mess pretty quickly.ww Also, it's good to have a clear demarcation point regarding responsibilities. You don't want your QLAN to be impacted by what the local IT firm is doing on the LAN for example. You also don't want third party components in the core of your network, while debugging some network issue.

[Dave Macaulay]

Atmos is a bit weird. The processor can put it on any network it feels like, I'm not sure if it can be told what NOT to use. Some setups require Atmos audio data to be on the management network and it plays happily enough, it does not need to be on the "Dolby Atmos Input".
As for AES67, that really wants to be on a separate network. We have used separate managed switches with stuff like IGMP snooping set up to suit the stream data. A current project will use VLN in a managed switch also doing management and media networks. I hope that works. The Dolby AES67 is really meant to be used with their DMA amps but can be used with JBL DSi-D "Dante" amps but talk to JBL before trying to configure it.

[Marcel Birgelen]

What do you consider to be "Atmos" in terms of transmission protocols? The BLU-Link protocol? Or the communication between the server and the audio processor (in case of a CP850/950A)? I've not seen BLU Link provisioned over the Management Port and neither AES67. I'm not saying it won't fly, but I don't think this is a Dolby-supported configuration.

If you do want to do AES67 in combination with Atmos, you should check the supplier if they do have a working configuration, since Dolby's AES67 implementation, up to this day, isn't conforming to industry standards, unfortunately.

[Steve Guttag]

Even talking to the manufacturer isn't a sure thing. I've been battling an IMS3000 to Q-SYS to Dolby DMA set up. Everything is AOK going into Q-SYS. But the DMA's are less than rock-solid for AES67 feeds from non-Dolby sources like Q-SYS. I consider it a good say when the DMA's only lose the AES67 clock once during the day for a brief (second or two) moment. I have watch dogs in Q-SYS that monitor if the DMA(s) bobble the streams (it check to see if they are getting a stable clock and individual streams via SNMP. If anything disappears for more than a few moments, it will toggle the stream and that always gets its back. I have not, as of yet tried the DMA with a straight up IMS3000 or CP950 system but I would presume that those are rock-solid.

As for Dolby Atmos, in general. Once it is in AES67 form (as it leaves the IMS3000 or the CP950A (or CP850), then it should be on an AES67 network and probably not co-mingling with conventional network traffic. If you are good with IT, I'm sure you can get it all to play nicely with each other but that isn't me. Now, before it becomes AES67 audio, THEN it can be on say the management/command network. A CP850/CP950A will then use AES3 channel 14, I believe to sync the audio from the data stream that is coming in via network.

[Harold Hallikainen]

The Wireshark capture I saw years ago and no longer have was, I believe, the CP850 requesting Atmos data from the server. That should be a point to point HTTP request, and no other devices on the network should see it. But, it was flooding an IRC-28C. I'm currently hearing of another data flood issue, but have not yet seen Wireshark captures, so I don't know what it is.

My original question had to do with whether an Ethernet switch should be sending data to a device that is not the destination of that packet. Network audio like AES67 would, I think, be on a separate wire. CP850 fetches should be point to point with the source and destination MAC addresses being the CP850 and the server. The switch should, in my opinion, send these packets out the appropriate ports and not to other devices.

THANKS for all the comments!

Harold

[Marcel Birgelen]

The Wireshark capture I saw years ago and no longer have was, I believe, the CP850 requesting Atmos data from the server. That should be a point to point HTTP request, and no other devices on the network should see it. But, it was flooding an IRC-28C. I'm currently hearing of another data flood issue, but have not yet seen Wireshark captures, so I don't know what it is.

You were part of the IAB standardization working group, so you should know better than me.

AFAIK, those are the data-streams involved in the communication between the server and the Atmos processor:

- Unicast TCP control commands, technically, those could also still be sent over serial.
- Unicast HTTP (TCP) custom API integration between server and processor, which uses a "REST-like" but proprietary API for the server-processor integration.
- Unicast TLS over TCP for the SMS communication via "Intra-Theater Messages", this is by far the most complex part.
- Unicast HTTP (TCP) AUX Data Trasfer Protocol. This does the heavy lifting of getting the actual IAB data-stream from the server to the audio processor.

The first tree data streams are all just control streams and are probably just a few bytes or maybe kilobytes worth of data at the most. Even if they somehow would end up on the wrong port, that shouldn't overwhelm any somewhat modern Ethernet device. The Unicast HTTP AUX data stream although, contains all the raw Atmos data. Still, this is a unicast stream and in a correctly switched network, this data should not end up on some other subscriber device on the same network.

A non-exclusive list why this could be happening:

- Your switch is not a switch, but a HUB... which at gigabit speeds, is pretty unlikely.
- A misbehaving switch
- A mis-configured port e.g. mirroring or port monitoring configured
- A network loop, which usually should come with some bonus extra issues.
- An IP conflict, although you'd expect more issues if this was the case
- A MAC address conflict: very unlikely, but I've seen it happen with cheap-o stuff where two devices ended up having the SAME MAC address. But even in this case, those were two network cards of the same manufacturer.

Your standard garden variety Layer2 switch has a pretty simple mode of operation. It listens to the FROM MAC addresses on every port and puts those into a MAC>Port table. If a MAC address moves from one port to another, it will update the table. It will also expire old entries if a MAC address hasn't been seen for a while.
It checks the TO MAC addresses on every incoming packet, if it finds it in the table, it will copy that packet onto the corresponding port (unless it's the same port it originated from). If there is NO entry for a MAC address, then it will silently drop the packet. Silently means: it doesn't send a message back to the sender, it usually however, logs this occurence by increasing a "dropped packet" counter for the incoming port. There are some other ways to handle "unknown MAC addresses" but they're not commonly deployed.