GDC False Fire Alarm Trigger

[Bruce Cloutier]

The GDC communicates with the JNIOR using MODBUS. Look MODBUS is great for controlling things (e.g. commanding a relay to close) but it is terrible for monitoring (e.g. watching for an input to change). In order to monitor a signal the client (GDC) has to poll (repeatedly ask for) the status of the JNIOR (server) input or condition to monitor. Polling is terrible in that you can never ask for status fast enough or often enough to see the change that you are looking for as promptly as you would want. You end up with a lot of repetitious communications with each and every one with the exact same response. The GDC scans the JNIOR repeatedly and eventually it appears to get the wrong result. The fire alarm inputs do not change state but the GDC picks up a false status. The show can be interrupted and there, thankfully, is no fire.

Here we see the same programming error and the same trap that DoReMi and others have fallen into. Unfortunately programmers these days do not have the experience and do not get the training they might need to properly implement a basic TCP/IP network connection. Here the point they need to get is that one protocol message DOES NOT equal one packet. It is a continuous stream of data.

In other works you cannot open the connection, read data and then simply parse the message. That program read statement returns the count of the characters/bytes obtained for a reason. The read statement returns whatever happens to be available in the incoming data buffer at the time. This can be a complete message, it might only be part of a message or it can even return much more than one message. So you need to take care to extract each complete message from the stream before processing it. If you need more data go back and read more. If you get too much then save the balance for the next message.

Unfortunately when things like MODBUS clients are developed and tested most (if not all) of the test messages are sufficiently packaged one to one in their respective packets. Testing in the real world takes time and effort. Not enough programmers actually use their own products in an actual application and for any real length of time. They should.

In GDCs case they apparently get only part of the message but parse the entire buffer area as if it was properly filled with the entire message and read uninitialized bytes as input status.

In their defense they probably pulled the MODBUS client package from some library and whomever had programmed that years back failed to setup the proper message acquisition code. In fact someone in the past might have pulled other code and, basically, no one really knows what is doing what and who might have been behind a lot of it. For some reason we see this programming deficiency more and more these days.

If you pull a network capture from the JNIOR right after the false alarm (Series 4 lets you do that) you will see that the JNIOR sends the proper reply to the poll request but splits that packet in two. It is the nature of the multi-tasking environment and the TCP/IP transmission algorithms that it can happen. It is 100% proper TCP/IP performance.

We worked with DoReMi in the past to get their related issue resolved and we can do the same with GDC. However, we have decided to limit the risk of this going forward by adjusting JANOS (the operating system running your Series 4 JNIORs) to try its hardest to package one message in one packet. I am not a "mask the symptom" and much more of a "fix the problem" kind of guy. In this case, I don't think "the problem" is fixable. Not with the way software is developed these days and how experience is valued (or not) when it comes to employment.

We will issue a JANOS v1.8.1 update very soon. I will let you know. That update will limit the issue for those of you with Series 4 JNIORs. We cannot do anything about this in the older Series 3 units. GDC will need to address it.

If you have GDC installations and have seen this or fear that you might, you can get this update. You can also likely get the pre-release (Release Candidate) from our support department to put your mind at rest even sooner if you want.

The JNIOR Protocol and in the Series 4 the JMP Protocol (optionally available through a Websocket connection) eliminates the need for polling. Those protocols actively signal input status changes. It is a logical alternative to MODBUS. Also, unlike MODBUS, these protocols allow for login and provide some level of security. Just saying...

[Steve Guttag]

I think you are taking the proper approach. You can point to GDC or Doremi or whomever all you want...but if the problem doesn't show until it is connected/working with the JNIOR, then it will be the JNIOR that is blamed. In the end, the exhibitor or their technicians don't really care about the under-the-hood problem, they just know they have an intolerable problem (false fire alarms or other triggers) and finger pointing just makes everyone look bad.

Or, option 2, connect something flammable to one of the relays and if you sense that the GDC server has issued a fire stop, trigger the relay to make the case true (start the fire). Then, the system will be proven to work!

[Bruce Cloutier]

Choose your battles I guess. I can do something about the issue so why not. You can stand on ceremony but... In this case since we are able to tweak any part of the system it is a minor adjustment. I can't do the same for the older Series 3. The Release Candidate for this update is in final test now. Should have the update file available by the end of next week and maybe before.

I just hope that if I post this with some technical detail every time we run into it, some programmer or manager will see it and perhaps not make this mistake anymore. There are enough other mistakes they can make and probably shouldn't.

There is another active support situation with a false fire alarm trigger but in that case we actually did see a short 200 millisecond pulse on the fire alarm input. No idea what that was. That I/O log comes in handy more often than you think. We're still waiting on another occurrence. I don't know why but we've had 3 or 4 of these fire signal calls in the past few weeks after years of nothing like it.

Thanks for the "Like".

[Steve Guttag]

Hey, at least you are trying to resolve the problem.

So that is a "No" for option 2 then?

[Marcel Birgelen]

Here we see the same programming error and the same trap that DoReMi and others have fallen into. Unfortunately programmers these days do not have the experience and do not get the training they might need to properly implement a basic TCP/IP network connection. Here the point they need to get is that one protocol message DOES NOT equal one packet. It is a continuous stream of data.

In other works you cannot open the connection, read data and then simply parse the message. That program read statement returns the count of the characters/bytes obtained for a reason. The read statement returns whatever happens to be available in the incoming data buffer at the time. This can be a complete message, it might only be part of a message or it can even return much more than one message. So you need to take care to extract each complete message from the stream before processing it. If you need more data go back and read more. If you get too much then save the balance for the next message.

Socket programming in general apparently requires some seasoned programming skills, at the very least to make it performant...

The fact that a "READ" doesn't necessarily always return a full protocol message and "SEND" doesn't necessarily send everything in your send-buffer seems to elude many programmers and because it all-to-often works out this way anyway, because of protocol messages being relatively small and modern networks tend to be fast and well-buffered, you'll constantly see that programmers keep falling in this trap. So, in the odd case a message gets split over multiple TCP packets or some buffer ends up to be full due to other traffic and the READ doesn't return a full protocol message, stuff flies out the door.

The simple fact is, this is a real rookie mistake and those developers that implemented it, can hardly call themselves developers. It doesn't even matter if it's a third-party library, even during stress-testing of your third-party library you should've been able to find out that sometimes, stuff breaks for no reason and the thing is no good.

I think that "eating your own dog food" is a good approach to eradicate bugs, but unfortunately, in this "gig economy", where everything is outsourced and everybody is hired for this one job, this is getting increasingly difficult.

The JNIOR Protocol and in the Series 4 the JMP Protocol (optionally available through a Websocket connection) eliminates the need for polling. Those protocols actively signal input status changes. It is a logical alternative to MODBUS. Also, unlike MODBUS, these protocols allow for login and provide some level of security. Just saying...

The PLC we're using over here allows you to lock MODBUS over TCP on certain IP addresses. This allows for some limited security.

[Bruce Cloutier]

The PLC we're using over here allows you to lock MODBUS over TCP on certain IP addresses. This allows for some limited security.

In Cinema we get little to no demand for secure communications. In our MODBUS server implementation there is a custom user command in which you can pass credentials and you can enable the login requirement for the protocol. To my knowledge in 15 years no one has done so or even asked about some other form of security for this. MODBUS is built-in to the OS on the Series 3 JNIOR but run as a separate application on the Series 4. It is much less prevalent these days and so it is relegated to an optional component.

To get around the polling issue we had one instance where they ran both a MODBUS server and client. Their client would command us through our server to enable relays for instance. The JNIOR would act as a MODBUS client and maintain input status on their remote server.

But I would recommend that if you really want to integrate the JNIOR that you implement the JNIOR Management Protocol (JMP) Protocol. That is all JSON based and handles I/O but file transfers, virtual command line sessions, syslog monitoring, registry access, message pump support and more. The JMP server listens to a port, supports STARTTLS, and does require login. It is also available through port 80 (and 443) as a Websockets upgrade to an HTTP connection.

On topic though... 80% of programmers move on to other projects or jobs about 80% through any project. The remaining 20% are left to test and correct the not yet ready for prime time mass of code. That last 20% of the project then takes 80% of the effort. Given that the tendency is to load up the programming staff with recent graduates (who have no legacy technology training let alone experience) you end up with things like this TCPIP sockets issue over and over. I was trying to avoid the words "rookie" or "newbie" but they apply. And, these are the groups that think they can make my car drive itself!

Someone I know, who has a Tesla, had her car reboot while driving along on the turnpike. Haha. I am thinking that I am going to start driving classic cars sans electronics. At least then I can stay on the move after the EMPs. ;-)

Oh... And NO Steve option #2 won't be implemented... um, at least not intentionally.

[Bruce Cloutier]

We've released JANOS v1.9 which is the operating system for Series 4 JNIOR controllers. This alters the behavior of the network stack to better associate 1 (one) message with 1 (one) packet. This addresses the problem of me having to rant every time we have trouble communicating with software developed by rookies.

The update is available on jnior.com which is same as integpg.com. I'll post a separate announcement about the update.