JNIOR Updates

[Bruce Cloutier]

I posted this in another thread. In thinking about that I would like to reaffirm our commitment to being transparent and helpful. So I decided to repost it here so that it might be found by everyone (and not just those who might care somehow about NMAP scripts).

The Series 4 JNIORs first shipped in 2014. These were configured with JANOS v1.0.0. Our development on the firmware side has been a continuous activity. This has slowed in recent years as the product reaches maximum reliability and bugs become rare. In my reviewing this document I note that JANOS v1.3 implemented a significant improvement in the level of reliability in the product. So I would strongly suggest that if you are running any Series 4 (410, 412, 414) with an OS prior to v1.3 that you do an update. We do only support (with bug fixes) the latest release of JANOS.

The attached document was written for internal use and not worded for general release. I have not redacted it in any way. We use SVN and Redmine support systems and the links will not be functional as they are for our use inside INTEG. If there is anything where anyone would like a better description, just ask.

The Series 4 JNIOR runs no 3rd party code. I have authored 100% of the firmware, every byte of it, and I personally stand behind any update. If you update from a really old OS you will likely also need to update any application that you might be running. Kevin, who many of you have met, is 100% responsible for these applications. The two of us first created the Series 3 back in 2004 so if we cannot help you no one likely can.

We will be happy to expand on the update procedure and options that you might have. It might help with your confidence level if you have a little more insight into what is going on. Just running some All-In-One update project, while convenient, can be scary if you for one moment question a step. Just prompt me for more information.

There is no reason why everyone shouldn't be running the latest v2.2. I get it that if it is not broken then don't fix it. If you get lulled into thinking (because the thing can be really reliable) that it is not broken, we would like to make sure that your opinion never needs to vary from that.

[Bruce Cloutier]

Typically one would use the JNIOR Support Tool and open an 'Update Project' under the Update tab. You can then publish the update to 1 or more JNIORs accessible on the local network. The update project is itself a ZIP file that you never have to expand. You just open it directly with the Support Tool and it obtains the script and all of the content from it as needed. A typical update project though might contain a dozen or so steps and update several items. If you have faith then I suppose that you would be comfortable publishing one to the devices in your entire house.

The Support Tool is itself free to download from our site. It is important to run the latest version of the tool. There is also a Java based version that can be used with Linux systems.

I sense that most of us are really sensitive to updating. There is a fear that after the update you will regret it. Taking it a step at a time and understanding what is actually happening will go a long way towards maintaining your level of confidence.

The main action is in updating the operating system, JANOS. The OS update is distributed in a UPD file (which you can separately download) contained typically in the update project. The UPD file contains an encrypted image of the new JANOS version. It also contains the JanosClasses.jar Java runtime library (java.lang.Object on up) that is part of the /etc read-only folder in the JNIOR file system.

To manually perform the update you would transfer the UPD file to the JNIOR. The /temp folder is a good place for it because, well, it is temporary and we won't need the UPD file after the update. The folder also has sufficient space to hold the almost 1MB UPD file. From the command line (accessible from the COM port, through Telnet or under the Console tab in the UI) you can ingest/load the UPD file using the JRUPDATE command. The update is not done until the unit reboots.

The processor within the JNIOR actually can hold 2 distinct copies of the operating system, the active copy (that is running) and a saved copy. The JRUPDATE command unpacks the new copy of JANOS and loads it into the Saved OS space. The command also immediately overwrites the JanosClasses.jar file. That is not an issue as running applications have already likely loaded and cached the classes they need.

Upon reboot the update request is detected and the Active and Saved OS copies are swapped. Unlike every other device warning you not to remove power and all of that, you can feel free to yank power all you want. I spent quite a while developing a fault tolerant reentrant update procedure. We beat the hell out of it updating with another JNIOR randomly and rapidly messing with the supply of power trying to get it to fail. You won't brick the unit. Not that way anyway.

The JNIOR will come up with the new OS. The previous OS has been moved to the Saved OS area. The JRUPDATE command can be issued to swap them back if for some reason you feel the need. The Stats command does tell you what might be present in the Saved OS area.

As a minimum you can do this to get the OS updated. Existing applications will run under the new OS without modification. I mean, there might be the rare case when you are coming from a really old version of the OS. If there is any such issue, rest assured that we can take care of it (almost) immediately for you.

[Bruce Cloutier]

Along the lines of what might change in an update causing some confusion is our WebUI. You know, what shows up when you go to the JNIOR's IP address with your web browser. This is the biggest and probably the only real question in updating.

The first Series 4 (c2014-2015) were issued with the Dynamic Configuration Pages applets following from the Series 3. Of course the Java applets have been an issue even at that point for years. I hadn't had the opportunity yet though to create a proper dynamic HTML site for the product. Those applets relied upon the binary JNIOR Protocol and worked for the Series 4 in basic ways where configuration remained consistent with the older series.

Once we had implemented a form of server-side scripting and enhanced our use of websocket interfaces we were able to release a new WebUi. We unfortunately still called it the Dynamic Configuration Pages or DCP an admittedly poor choice of acronym. That happened in 2016 with JANOS v1.5. Most Update Projects then properly removed the applets and installed the WebUI for you.

The JNIOR web server has the ability to utilize a ZIP file (including JAR files) as a virtual file folder. The name of the ZIP file and its location defines the path to its content. We released the WebUI as a www.zip file placed in the /flash folder. So you can see then how the server could then pull the entire website from the one file which implemented a virtual /flash/www folder. The nice aspect is that the website is managed as a single file and there is no chance (or very little chance) that any part of the website would get out of sync. To update the WebUI (back then) you need only overwrite the one www.zip file in the /flash folder.

So if you updated with an update project this move happened automatically for you. But if you updated with a custom update project or manually you might have be stuck with the applets or no WebUI at all.

To add to the possible confusion, many JNIORs are in use where the customer has implemented (or we have) a custom web site. You need only put your website files in your own www.zip file or physically in /flash/www folders. The issue then was that you would lose access to the WebUI for management of your JNIOR. We also had the risk of overwriting your custom website in trying to update our WebUI. The solution was for us to move the default location of the WebUI to /flash/www/config by renaming the WebUI ZIP to config.zip and shipping it in the /flash/www folder. That is where it is today. JANOS knows that if it cannot find a home page in the standard /flash/www folder that it will attempt to look for one in the /flash/www/config (virtual) folder and give you the WebUI as a default.

Again most update projects relocated the WebUI for you. In some cases we added a Registry key to tell JANOS to look in the /flash/www/config location in case you weren't running the latest OS.

Okay, long story. The bottom line is if you update, the only repeated issue that we have encountered is the loss of the WebUI. You would need to make sure that you have it as /flash/www/config.zip and remove any old /flash/www.zip as well as any potential Registry keys relocating your website. You might even have some web files in the actual /flash/www folder that should be removed too. This confusion happens but really really infrequently.

But you can see how with it this way you can create a custom website and still get to the WebUI by referencing the /config subfolder. You can move that around be simply renaming the ZIP and putting it in a different location. The ZIP virtual folder thing is awesome for website maintenance. That might have been patentable but that is not us. I am listed as inventor on too many patents as it is.

Our current WebUI allows you to completely control and manage your JNIOR though the web interface. There is no need for Telnet or FTP.

[Bruce Cloutier]

On the issue of software/firmware updates in general, there is an inherent tension between the "if it ain't broke, don't fix it" mentality of many end users, and the "we want you to install it as soon as it's released" mentality of OEMs, the helpdesks of which will often ask you to update to current before they're willing to talk to you about whatever fault you're troubleshooting.

Along these lines we have released JANOS v2.3 the firmware for all Series 4 JNIOR. You can get the update from the INTEG website.

The issue with "it ain't broke" is that it isn't until it is. We would rather not have a known issue impact anyone's opinion of the product and so we are proactive with the suggestion that everyone should update to the latest OS.

We discovered an issue where in busy networks a JNIOR might lock-up and become unavailable. Perhaps even repeatably. This requiring someone to remove power from the unit in order to bring it back. This applies only to the Series 4. We would very much not like this to happen to anyone especially those of you who might have to drive a considerable distance to a theater in order to rectify this not having anyone on site that you might trust to handle it.

This has been corrected with this release of v2.3 along with a few other concerns.

But no, this update won't break anything. No protocol or user interface has been revamped to create work for you. We won't start popping up ads or mining your personal information. There is no feature that you will now have to subscribe monthly to continue using. You won't have to create an INTEG account to proceed. None of that BS that we all love about updates. It is why my production equipment remains disconnected from the network at all times. I get it. But... please get the update.

[Steve Guttag]

For me, release notes are key to judging the necessity of an update. So, if in your release notes for 2.3, there were words to the effect of "prevent lockups on busy networks" it would be more likely to be installed. And while all manufacturers will tell you that the new version won't "break" anything or cause issues...we all know that they can and do. It is impossible to know/test every way something might be used and the right combination of things could cause the break.

If I had units affected...my course of action on an update like this would be to wait a week or two (unless I was having the problem it purports to fix) to see if any "oops, we fixed, the fix" versions come out. If it remains in the clear then try it on one unit...see if it is stable and doesn't cause any issue with how I install the device. If all is good, then it would be scheduled for all affected units.

[Bruce Cloutier]

All valid points. The reality is that there is one and only one person responsible for, and with full vision of, every single difference between v2.2 and v2.3 let alone every build prior and ever. And so the claim of the alleged safety of the update isn't in any way a blind marketing assertion.

But in reality there is a decade of this series of JNIOR out there and most are absolutely still running. The mass of Series 3 still in use is incredible. The reputation of it being reliable even with archaic firmware (we've encountered some with pre-v1.0 firmware) absolutely amazes us. I just shake my head because for me even the slightest hint of a bug robs me of sleep.

What I would like to avoid is that customer who calls all pissed off because he has to reboot JNIORs all of the time and has been over and over forever. Never thinking to call us before. Probably because they don't want to sit on hold and then talk to someone in India faking an America accent who provides support for 30 different companies and can't help except to recommend that you reinstall, remove cookies, and reboot. Oh, and update. They're probably thinking that they aren't paying for support so the situation is what it is. Well it ain't.

By the way, we have a class of bug that we designate as a gremlin. A gremlin is the kind of glitch that causes random symptoms and that might only affect something obvious once in a great while. For the tech guys, consider the effect of some aspect of the system using memory allocated by an application. One that when the application terminates and its memory allocations are released doesn't inform the system which continues to write into some now part of memory that may or may not be reused for something else right away. We eradicated a gremlin of this kind in v2.3 along with the race condition leading to the described potential lock-up.

I am as paranoid as all of you w.r.t. updates. All it takes is to be burnt by one company (probably MS) to impact your approach to any and all other updating. Hell I won't even leave my production machines connected to the network. Why those run Windows is beyond me. A deterministic pick and place machine running on a multi-tasking OS that in the middle of an assembly might decide to run off and modify system elements so it can popup ads for your enjoyment.

Kevin and I once set aside a month to hunt gremlins. That begat v2.0. Just saying.

The important point is that with this product if there is a bug it is nobody's fault but my own. You can also yell at Kevin if its an application program issue. We stand behind the safety of this update.

By the way, waiting is a good approach regardless. There is no rush.

[Bruce Cloutier]

Here are our (unedited) internal release notes for the Series 4 all time. The links in this document are references back to the issues in Redmine here and won't function for you. If you have any questions just ask. We have no interest in hiding anything (even if it involves an embarrassing episode in code). I would be more than happy to supply technical insight even if to satisfy a curiosity.

Applications (JAR files like cinema.jar) are Kevin's responsibility and therefore not represented in this. You should feel free to hit up our support email address with any questions that you might have along those lines.

These things are sprinkled all over the planet. We absolutely want them to all function flawlessly.

So we never had this level of control over the product with Series 3. Now with the Series 3 (310, 312 and 314) hardware failing (soldered batteries dead, flash storage worn out) it is time to put those out to pasture. There is so much more value in the Series 4.

It would be nice if only each unit pinged us when it boot. I kind of regret that not being there. We have no feedback as to how many units are in use and where. I realize that not every JNIOR has access to the Internet. There is the privacy issue. Considering that some are in very secure facilities. We talk about adding this all of the time. It is a current debate. There is the opt-in approach but no one would bother. You guys can chime in on this point if you like. We have a thought of offering a NOC approach for JNIOR.

The PING -V command pings all configured IP addresses in a unit. It is a good validation of your network connection. The last step pings our servers here as a confirmation of network access. That is actually an ICMP PING and we don't capture it. Not that anyone runs PING -V or even knows about it. It is documented. Just FYI.

[Harold Hallikainen]

Very nice! When I heard of gremlins that I could not duplicate, I'd add more logging to try to catch the failure in the log. That would usually eventually expose the cause of the issue. For example, we had an issue of reboots on busy networks. The library code for the Ethernet chip would read an error code, but then just reboot. I added code that decodes the error code to plain English and logs it. Logging is our friend!

Harold

[Bruce Cloutier]

Kevin I think goes overboard with logging. From a debugging standpoint it is an absolute necessity. But I think too much logging can become a burden on the file system. It is just hard to know exactly when it is okay to pull some of the logging back out. In the OS we log issues along with some form of reference so I can go back to code to determine more exactly what happened. A more serious issue will SCRAM the system and produce a dump log file. That being more or less a stack dump which is a very manual process to decipher; It is very dependent on the OS version; And, usually pretty informative.

With each new JANOS release (now v2.3) these events become more and more rare. Our update projects usually remove any dump.log and errors.log file. It is a good idea to remove them before you start trying to debug some issue so you can see what is new and pertinent.

In addition to logging there are some undocumented options to commands that we use to view the state of one subsystem or another. Those are there and used now less and less if at all. Still not sure when to just remove them. For example GC -B and GC -M commands both display top memory allocations by block and by memory size. Use of these periodically can easily identify memory leaks. The big hex values refer back to the routines responsible for the allocation. We used to find a routine here or there that would essentially malloc() some memory and then not free() it. But now you are more likely to find memory building up as a Java application keeps adding stuff to a Vector class, Hashtable or something without bothering to remove it at any point.

My JVM does not support remote debugging. There is a preliminary partially useful DBG command in JANOS v2.3 which is the beginning of a command line debugger for applications. That command is not hidden but also not really documented. It also is bytecode oriented at the moment as source code is not normally supplied with applications. The THD command is very useful in seeing what an application might be doing at any point in time as well.

Just fun low-level OS stuff.

I think it is now 40 years since I wrote my first (preemptive) multi-tasking firmware. You know... back in the days before PCs and when if you wanted to print something you had to use a modified IBM Selectric typewriter and write your own parallel port driver to spin the print wheel, strike the character, and move separately the head and carriage. All kinds of fun in and by itself.

[Bruce Cloutier]

JANOS v2.4 OS for Series 4 JNIOR has been released.

This update is now available for download from our website (integpg.com or jnior.com). Of course we highly recommend the update.

Most notably is an effort we undertook to root out any remaining gremlins (nefarious bugs). Our low-level memory manager was enhanced to scrutinize in great detail any block of memory being returned to the pool for any sign of overrun (even 1 byte). Yes, overrunning a memory block is bad but you can usually get away with a minor indiscretion given that blocks contain padding. We had never been as sensitive to anything that any application or the OS tried to get away with as we are now. As a result we did discover and rectify a few issues.

You would think that having a JNIOR on the open Internet would be challenging. The product survives quite well out there. It is the IT departments who think it is worth running every possible malicious attack on every device on their own networks that prove to be the foe here. In a production facility we encountered an IT scanner that gave us real difficulties. It took the diagnostic modifications to the memory manager to track that one down and resolve our sensitivity to it.

While this update is fully compatible with any you are currently running, there are a couple of notable new features.

The Series 4 captures network traffic and has had the ability to generate a PCAP capture file (NETSTAT -C) that when downloaded would open up in Wireshark. That has been very useful but still a bit cumbersome and requires installing Wireshark. The NETSTAT command has been augmented with S, P and D options which display filtered packets realtime from the command line. This has proven to be intoxicating and NETSTAT -S has been a GO TO step to verify for instance that a unit is indeed communicating with the projector or media system right now! I can expand on its usage if anyone is interested.

I know that many of you dread the command line console experience. Those of you familiar with Linux terminal procedures would have noted limitations with our command line. With v2.4 we implement piping and conditional execution. As a result there are some useful tricks that we are starting to rely upon. For instance the CAT command (which displays file content) can accept a wildcard specification and thereby combine .LOG and .LOG.BAK for subsequent search by GREP through piping. It is just a level of function that we should have had all along.

This is the most stable version of the OS to date.

Here are the cryptic release notes for internal geeks. As always if you would like me to decrypt any of it, just ask.

v2.4 notes.png

[Bruce Cloutier]

Just an update that we are working zero (0) issues with JANOS v2.4 JNIORs.

There is one glitch that we have identified relating to the TAB auto-complete available at the command line where it messes up while entering a second parameter in the command. This is not a new problem. As you can imagine this is really no problem at all but I have twice had to correct it for one reason or another. So it has been rewritten for the future v2.4.1 release when it will function properly even when used in the middle of a command while editing. TAB can be very helpful in command entry. Play with it and you will find its shortcomings. Well, hopefully, that feature will be straightened out with the next release.

Prior to v2.4 there apparently had been a gremlin or gremlins leading to a random assertion resulting in a rare dump.log file or some inexplicable entry in errors.log. If you are running an older OS and have seen these files then you can definitely benefit from the update. We went the extra mile in an effort to hunt these things down before the v2.4 release. If you have updated to v2.4 and get a dump or some inexplicable errors.log entry please bring it to our attention promptly. As of this writing there has been no case of it.

We also believe in the jinx as a powerful diagnostic tool. That is what this email is. I am challenging fate by applying the jinx in making the following statement. v2.4 JANOS is solid!

Thanks go to all for the continued employment of JNIOR. Let us know how we can help you in that.

[Bruce Cloutier]

So with JANOS v2.4 the DATE command reports the incorrect date and time. THIS IS MY FAULT. The time and date in the JNIOR is correct however.

A routine to display the date that is used by some functions was updated to optionally include milliseconds. This was part of the effort to improve clock accuracy (so we could watch it to the millisecond). So the DATE -HM command which displays both the software (OS) clock and the hardware clock with milliseconds does report the correct time and date. But, ugh, in a few other places the time variable for one of the parameters to the routine needed to be uint64 but was declared in those cases as uint32 and so... weird date modulus mathematics. This was missed in testing.

Sorry, this will be corrected with the v2.4.1 release when we get to that. The fix has been verified in beta code here.

[Bruce Cloutier]

We are watching for an upcoming anniversary. The JNIOR Series 3 shipped from 2005 up until 2015. The Series 4 first shipped in 2013. Okay so it will shortly be 10-year-old technology but there is actually no technical reason to not still consider it current. I am committed in the supply chain to be producing this series for at least another 3 years.

We have been beating around the concept of Series 5. Other than (for fun) I want to beef up the processor we haven't collected anything for a wish list. Anyway, I am open to suggestions. We keep doing the "what if" thing.

[Bruce Cloutier]

JANOS v2.4.1 OS for JNIOR Series 4 has been released.

This update will be available for download from our website next week after our Memorial Day weekend. (So you get advanced notice)

There were no major issues driving this update. The advanced memory checking algorithms added with v2.4 triggered a couple of minor corrections. This update (other than fixing a DATE command issue that was introduced in v2.4) basically improves the command line experience and adds operational and network metrics that may be useful in evaluating performance.

We still highly recommend updating to v2.4.1. The fact remains that we are best able to support your JNIORs when we are working with the latest released version of the operating system. In part this is because we can assume that problems are not from issues that have already been corrected. It is also because some diagnostic information includes memory addresses and stack traces that can only be related back to the current released source code. This increases the chances that we can successfully correct situations and as fast as possible.


v2.4.1 (Released - Sep 1 2023)

• Resolved HELP manual generation paging issue
• Enhanced TAB auto-fill to work when editing a line
• Allow you to force context for auto-fill using Ctrl-F and Ctrl-R
• Fix memory ownership issue with cstring_t
• Fixed DATE command problem introduced with v2.4
• Increased ARP database size
• Corrected issue with Thread.interrupt use before the thread starts
• Added network performance information to NETSTAT
• Added bandwidth information to NETSTAT -A
• Added PS -H showing a history of operation.
• Added shutdown messages to the system log file.

[Rafael Barbosa]

I'm having problems with a Junior 410 with GDC server, the CUE command to turn the room lights on and off fails randomly. The pulses arrive very quickly, I need to put a 2 second delay between the pulses but I don't know where to configure this. I have already configured the delay through DGC but Jnior does not respect this configuration. The Jnior configuration tab via the web interface does not appear. I accessed it via the Junior Main Applet, but I didn't find where to configure a delay. Is this Janos you're talking about a program that replaces Web access?