Aging tracks created through an all-digital workflow aren't guaranteed to play back.

[Frank Cox]

https://www.mixonline.com/business/i...ut-hard-drives

Inside Iron Mountain: It’s Time to Talk About Hard Drives


Iron Mountain Media and Archive Services sounds the alarm: Aging tracks created through an all-digital workflow aren't guaranteed to play back.
By Steve Harvey ⋅
Published: 08/19/2024 ⋅ Updated: 08/21/2024
Shelves of media assets stretch as far as the eye can see at Iron Mountain’s secure, climate-controlled, underground facility in Boyers, Penn. Photo: Courtesy of Iron Mountain.
New York, NY (August 19, 2024)—A few years ago, archiving specialist Iron Mountain Media and Archive Services did a survey of its vaults and discovered an alarming trend: Of the thousands and thousands of archived hard disk drives from the 1990s that clients ask the company to work on, around one-fifth are unreadable. Iron Mountain has a broad customer base, but if you focus strictly on the music business, says Robert Koszela, Global Director Studio Growth and Strategic Initiatives, “That means there are historic sessions from the early to mid-’90s that are dying.”

Until the turn of the millennium, the workflow for record releases was simple enough. Once the multitrack was mixed, the 2-track master was turned into a piece of vinyl, a cassette tape or, starting in 1982, a compact disc, and those original tapes—by and large— then went into storage. Around 2000, with the advent of 5.1-surround releases, then in 2005 with the debut of the Guitar Hero video game, things started to get complicated. When rights holders went to the vaults to transfer, remix and repurpose some of their catalog tracks for these new platforms, they discovered that some tapes were deteriorating while others were unplayable. Not all assets had been stored under optimum conditions. Some recordings had been made on machines that were now obsolete, in formats that could no longer be easily played. And some recordings were missing.

In short, for the past 25 or more years, the music industry has been focused on its magnetic tape archives, and on the remediation, digitization and migration of assets to more accessible, reliable storage. Hard drives also became a focus of the industry during that period, ever since the emergence of the first DAWs in the late 1980s. But unlike tape, surely, all you need to do, decades later, is connect a drive and open the files. Well, not necessarily. And Iron Mountain would like to alert the music industry at large to the fact that, even though you may have followed recommended best practices at the time, those archived drives may now be no more easily playable than a 40-year-old reel of Ampex 456 tape.

“The big challenge that we face is just getting the word out there,” says Koszela, who racked up years of experience on the record label side with UMG before joining Iron Mountain Media and Archive Services. Iron Mountain handles millions of data storage assets for a diverse list of clients, from Fortune 500 companies to major players in the entertainment industry, so the company has a significant sample size to analyze, he points out. “In our line of work, if we discover an inherent problem with a format, it makes sense to let everybody know. It may sound like a sales pitch, but it’s not; it’s a call for action.” CAN YOU PLAY?


Many asset owners—labels, artists, artists’ estates—sleep soundly at night believing that their recordings are safe in a climate-controlled vault, Koszela notes. But just like tape, hard drives are susceptible to any number of issues that may only be discovered when, for example, the project is pulled off the shelf to create an immersive mix.

“It’s so sad to see a project come into the studio, a hard drive in a brand-new case with the wrapper and the tags from wherever they bought it still in there,” Koszela says. “Next to it is a case with the safety drive in it. Everything’s in order. And both of them are bricks.”

Let’s say a drive containing a 1995 session does spin up. “You’ve got to update the Pro Tools session and you’re probably going to have to fix some plug-ins,” Koszela warns. “You’re off to the races, and you can create an immersive mix—but not if you wait too long and let that stuff die.” Iron Mountain archivists at work among the vaults in the company’s Boyers, Penn., facility. Photo: Courtesy of Iron Mountain.
A lot has changed in the world of digital media during the past three decades, so legacy disk formats like Jaz and Zip, obsolete and unsupported connections, and even something as simple as a lost, proprietary wall wart for the enclosure can be a challenge with some older archived assets. Based on years of experience, Iron Mountain has developed hubs at its facilities that can power up, connect to and read virtually any storage medium. If the disk platters spin and aren’t damaged, Iron Mountain Media and Archive Services techs can access the content.

As with tape, pulling an archived drive off the shelf and discovering any challenges to playing it will typically only occur if there is a commercial imperative. “Most of the resources are freed up based on exploitation,” Koszela confirms, such as the label’s need for that immersive mix. However, an archived drive may hold an early transfer from tape at a lower resolution than is now the norm. If there’s enough budget, someone may need to go back to the tape and— hopefully, barring any issues—transfer it again at today’s accepted standard of 24 bits/192 kHz. How IMES Automated The Archive


Again, like tape, it’s not always that easy to pinpoint the exact asset that needs to be pulled. As with old tapes, where there may be little more than a ballpoint scrawl on the box label, the metadata—the writing, barcodes or other information carried along with the content—is also critical to finding the right disk drive in the archive, and the right version of the track that you are looking for. “The outside of the case might just have an artist’s name as an acronym,” Koszela says. “You don’t know if that’s a video session, an interview or what it is.”

While with UMG, Koszela would take the assets from the projects that came through Interscope’s studios and send them to the label’s archive team. “We started receiving a lot of black cases that didn’t have anything on them,” he recalls. “We would open up the drive, mount it, open up the catalog tree so we could see all the folders, then screenshot and print it, and put that in the case.” It wasn’t the most elegant solution, he admits. “But it allowed us to quickly see what’s on the drive without going through the trouble of spinning it up.”

There are apps that now make that task easier. “Neofinder goes through the catalog tree and makes everything searchable,” he says. But these days, metadata might be embedded with the files on disk or in the cloud, not on a box or a case. Searching through that data could eventually be a task for AI, Koszela believes. Robert Koszela, Global Director Studio Growth and Strategic Initiatives, Iron Mountain Media and Archive Services. Photo: Courtesy of Iron Mountain. DO YOU KNOW WHAT YOU HAVE?


When people in the 1990s started producing music using DAWs, the entire workflow became digital, from writing to demo to tracking and mixing, but there’s a potential challenge there, years later, for anyone trying to find the complete and final master. “What if somebody brought something in on an Akai MPX [sampler] and they didn’t fly those tracks in, they triggered them?” Koszela asks. “Did the samples ever get copied to a master hard drive? And if they did, are they labeled?”

Similarly, in today’s production workflow, a session could easily have been tracked at one studio, overdubbed at another, had strings added at yet another, then mixed and even remixed, perhaps across continents. “Who has the final copy of the session that consolidates everything?” Koszela asks once more. “If that master is lost, is there a copy or a version from earlier in the production workflow that will suffice, such as a producer’s, engineer’s or studio’s backup copy?

“It’s a plus that the data’s probably out there somewhere, but it’s also a minus, because there’s so much of it, and in so many different states of completion. Who’s got the right version? Is the master lost? Probably not, but will you ever find it? Possibly not.”

Smaller entities, like independent labels or artist’s estates, with little budget for asset preservation, are generally letting drives sit in the archive. The bigger labels are unlikely to find and address any issues unless an asset is being commercially repurposed. Without some proactive initiative, Koszela says, “My worry is that these assets will just be lost. People need to know that their hard drives are dying.”

[Harold Hallikainen]

Interesting to see stuff about Iron Mountain. When we introduced the JSD-100, many were sold to Digital Cinema Implementation Partners (DCIP) who then leased them to theaters. We had to send our source code to Iron Mountain to archive it for the duration of the leases (which, I think, was 10 years). Each year we'd send another CD with any firmware updates.

[Brad Miller]

This is not surprising in the least bit. In fact I think most of us here have predicted this as something that would happen much sooner than most people predicted.

This is precisely why that idea from a number of years ago where data would be visually printed onto 35mm film for long term storage was such a good idea. Properly stored, a polyester roll of film should last 100 years in storage. And ideally 3 copies would be created and placed into 3 different locations such as western/central/eastern US.

[Steve Guttag]

Accelerated aging tests showed that properly stored film should make it 1000-years.

[Frank Cox]

I'm sure we've had this discussion here before, and what I said then I'll say again:

I don't see what's so damn difficult about keeping old files in a readable condition.

I have files on my computer dating back to the mid-70's. Some of them started out on tapes, migrated to floppy disks, then to hard drives, then to other hard drives and on from there.

Every time I get a new computer I copy everything that I want to keep from my old computer, and I have multiple computers and multiple backups as well.

The only counter-argument that I can see here is that I'm dealing with a few hundred gigabytes of data while these guys are talking about petabytes.

But I'm also not a professional archivist. It's their job to deal with this stuff.

[Mark Ogden]

It's interesting that the article starts talking about hardware problems, specifically hard drives, but then devolves into the real issue facing archiving: human error due to mishandling, lack of proper record keeping, sloppy storage, theft, etc. Film may last 100 or 1000 years, but that's useless if nobody knows where things are. I wonder what became of the camera negatives for two of cinema's greatest achievements, Citizen Kane and Casablanca. They haven't been seen in decades. I wonder what became of the camera negative for Laserblast, one of cinema's least achievements. A colorist friend of mine went looking for it a few years back, no one has any idea what became of it. The best they could find was a fine-grain IP that the film's make-up artist had in his storage unit. I've always been a big fan of The Mamas and the Papas. I wish their back catalog would be remastered, but it all the original masters went up in flames in the Universal back-lot fire in 2008, along with the master tapes of some of music's greatest artists. In 2004, the entire Motown Records master tape collection was nearly destroyed when the pipes broke in the warehouse they were stored in, the upper floors, which were rented by a food storage company, collapsed, covering this incredible collection in Thousand Island dressing. It took $12 million dollars and a herculean effort to save it all. Hundreds of analog-realm films and audio recordings have been lost over the years for reasons that had nothing to do with the inadequacy of their storage medium. This is the real issue.

As far as digital storage and longevity, there are promising results from M-disc and LTO drives, and the digital files on film thing the Brad mentioned has promise too. Even so, all it takes is some putz with a Sharpie to mislabel things before it's all lost for good.

A lengthy but fascinating and infuriating (and way too long to cut 'n paste) article about the Universal fire: https://www.nytimes.com/2019/06/11/m...ecordings.html

[Steve Guttag]

Another couple of things film has going for it...it is unencrypted and it is self-explanatory as to how it works.

Digital storage tries to thwart the user at every turn. It is often encrypted and it depends on specific technology to retrieve it. So, one not only need to have the content, they have to have the prescribed playback method as well as the documentation as to how to play it back and any decoding methods...which are deliberately designed to require some other authority to possess to allow decrypting. I would hope that any archiving entity wouldn't encrypt things as the last step in retrieval could be eliminated.

[Frank Cox]

It seems obvious that materials in a long-term archive should not be encrypted or copy protected.

That applies to the software as well as the data.

You're creating the next great opus using proprietary software that says "mother may I" before running? That software is not fit for purpose.

Thirty years later, the authentication server has been disabled for decades and you want to access your files? Sure. The rest of us get to point and laugh.

[Leo Enticknap]

As others have noted, this is very old news. This scenario was being predicted when I was a film archiving student in the mid-1990s. At the time, the received wisdom was that film will last forever if kept cool and dry, so let's archive everything (including digital data) on film.

I haven't worked professionally in media archiving for 12 years, and so I'm not up to date with the current debates around best practice. However, I will make the following observations.

Maintaining the ideal storage environment for the long term preservation of photographic and magnetic media has proven to be a lot more easily said than done than was believed back in the '90s and '00s. The kicker is keeping it both cool and dry: one or the other is not too difficult, but every artificial cooling technology known has the side effect of increasing humidity, meaning that doing both at the same time uses a phenomenal amount of energy, not to mention the capital investment and maintenance costs of cooling and dehumidification infrastructure. About ten years ago, I heard the figure of $10-15 a year to keep a single 2,000ft reel of nitrate at the FIAF and SMPTE-recommended temperature and humidity levels. Given that most large film archives are either non-profit (and almost all the big nonprofit archives worldwide are funded primarily from tax dollars/pounds/euros) or operated by for-profit businesses and treated as a cost center rather than a revenue-generating unit, this money is not forthcoming with no questions asked. There is also the political dimension: the passive conservation approach is very energy intensive, and is likely to face mounting opposition from the greenie brigade, which is now very influential with lawmakers.

The cost of data storage, either directly owned (the street price for you or I to buy a 10TB HDD is now below $200) or rented from a datacenter (aka "the cloud"), has come down exponentially over the last 10-20 years, likely beating even Moore's Law. We know how to prevent data loss (regular migration between short-term media formats and regular replacement of worn out RAID drives in datacenters), just as we know how to delay for centuries the day when our nitrate starts rotting into an explosive goo. I still have a some of the first ever computer files I created, on 5.25" CP/M floppies in the mid-1980s. They were copied to 3.5" floppies when I later got a computer that had drives for both in it, then to a CD-R, then to internal hard drives, and now they're sitting on my NAS. The challenges of reading them are significant (reading documents created with Spellbinder and early versions of MS Word cannot be done just by double-clicking on them, but it can be done with a bit of effort), but not insurmountable. The cost of doing digital media preservation is now one helluva lot less than it was even in relatively recent history.

The ethical arguments for preserving original analog media increasingly have to be balanced against the cost and environmental impact of doing so. The ethical arguments for preserving (physical) digital media are almost nonexistent if transcoding file formats into other formats and then discarding the original is not being contemplated. As long as the file can be copied bit for bit, including its metadata, this is all that is needed.

So it boils down to intent and resource allocation, just like it always did. If something is believed to be important enough to preserve, it will almost certainly be preserved. Most of what the Romans wrote is lost, but the works of Tacitus and Pliny the Younger survive. Around 90% of Beethoven's output is known to survive, but name most other composers for hire in Germany in the early c19, and little if anything they wrote will still be around. A lot of B-movies and adverts from the 1940s have gone, but Citizen Kane is still with us. The few exceptions to that general rule is stuff that only started to be believed to be historically or culturally important a significant time after it was made and then lost, e.g. Hitchcock's The Mountain Eagle.

Which brings us back to the Iron Mountain datacenter article. It's effectively an ad: pay us for our services, and your stuff won't disappear. That 1990s album that only made it to no. 84 and then the band disappeared without trace - you wouldn't want to risk a bunch of academics and critics suddenly deciding that it's a work of genius and then you don't have it to sell again, do you? We can't preserve all the writing, audio, video, and data we create - never have been and never will be - but there are people out there aggressively trying to sell you a chance to get yours into the ark.

[Frank Cox]

early versions of MS Word cannot be done just by double-clicking on them

If all you want out of an old Microsoft Word file is the text content you can get that with the gnu strings program (comes built-in with Linux, don't know about where it is on MS Windows but it's probably not hard to find).

Otherwise you can run the appropriate version of MS Word in something like dosemu and load the file into the original program.

But since old versions of that program supported only very basic text formatting, the strings program is probably all you need.

[Bobby Henderson]

The issue of data archiving is a big problem which affects far more than just the entertainment industry.

So many people take it for granted their precious data is safe, whether it's work documents, personal photos/videos or whatever. They don't consider all the different variables that are threats to their data. I still encounter people who don't even understand how to manage their files, organizing them in different folders.

The first computers I learned how to use didn't have a graphical user interface. The screen was black and had a blinking cursor on a command line. Yet these people can't bother to open Windows File Explorer or Finder on a Mac. Needless to say, they're woefully not-prepared when a hard disc drive starts going bad. Now most newer computers have solid state discs. Those provide far less warning if and when they want to brick themselves.

We can personally control migrating a bunch of data from an aging hard disc to a new "fresh" drive. One thing that is out of our control is the software applications needed to read that data. If the files are saved in a proprietary format supported only by a specific application then an operational copy of that application is often needed to accurately open the files for viewing or editing.

The situation gets worse from there. Not all software vendors maintain backward compatibility with old files. For example, I have graphics files that are over 30 years old. Some are saved in ancient versions of CorelDRAW. The current and past few versions of CorelDRAW will not open or import CDR files made before version 6. The only solution is having an older working version of CorelDRAW that can open those files and re-save them in later versions. Many older versions of CorelDRAW won't install in Windows 10 or 11. A vintage PC or old version of Windows running in a virtual machine would be needed.

Many pieces of software require activation. They have to "phone home" to validate the installation. What if that particular software company goes out of business and its activation servers are taken offline? Any files saved in that application will need to be converted to another format in order to preserve the content.

Even something as simple as fonts can be a problem. A few font formats introduced in the 1990's died soon after that decade ended. Adobe recently removed support of Postscript Type 1 fonts in its applications, and Adobe is the company that created Postscript! Type 1 fonts still work in a variety of other applications and can still be installed in Windows -for the time being. Type 1 fonts can be converted to OpenType format; to do that effectively requires buying an app like TransType.

[Mark Gulbrandsen]

Film is actually the only safe way to archive digital data. Film can last well over 200 years according to tests that Kodak has done. IN 200 years they will have a new system we can only dream of today...

[Harold Hallikainen]

Years ago, I wrote a program to convert WordStar documents to HTML.

On CP/M, I still have Z80mu that will run CP/M programs under DOS. Then running that under DosBox, I can run stuff from the 1980s. I recently reassembled some source code from 1980.

Harold

[Randy Stankey]

Years ago, I remember listening to a story from NPR about the Smithsonian Institution experimenting with preserving digital data using some kind of photogravure technique onto glass plates. They didn't give a lot of details about the process. The segment was mostly about the Smithsonian's efforts to preserve their data and less about the actual processes used.

Even back then, probably close to twenty years ago, the Smithsonian was concerned about data preservation. The article talked about the fact that virtually every information storage technique in use, today, has shortcomings which allow data to degrade over time. Even things like Egyptian hieroglyphics or cuneiform tablets carved into stone or clay might still be legible after thousands of years but decoding them after all that time is still a big problem. It wasn't until the mid-1800s that Champollion finally deciphered Egyptian hieroglyphics.

The end of the article summed things up, saying, "Yes, we can use lasers and other technology to etch data onto a long-lasting medium like glass but, in another 5,000 years, who is going to know how to read it? Will we have to leave some kind of Rosetta Stone behind?

[Scott Norwood]

The biggest problem that I can see with digital preservation is that it requires constant maintenance and is generally an all-or-nothing affair (you either have the complete file or you have lost everything). Yes, it is easy enough to copy data to new storage formats and media as time passes, but someone has to actually care enough to do that, and have the time, knowledge, and resources to carry out those tasks. The idea of being able to put something on the shelf, come back in fifty years, take it off of the shelf, and look at it does not exist with digital storage. No one is going to find a hard disk in Grandma's basement in fifty years and have a great time looking at all of her old pictures that evening.

But, yes, if we accept the issues of limited media life, and avoid proprietary, undocumented file formats, and have multiple copies and someone to ensure all of this, digital storage is fine. The Library of Congress is doing an excellent job with such things. But Aunt Millie isn't equipped to do any of this, and probably won't have the knowledge or resources to delegate the work to someone who is. For anything with personal value (pictures, etc.) where there is no economic or other incentive (or Congressional mandate) to regularly preserve it, physical formats will always be best.