best practice or method for archiving large amounts of content (grid down)

[Smurf Hunter]

I have in my personal google drive, well over 100GB of documents, manuals, instructions, and other material that could potentially be valuable if we ever found ourselves without the internet long term.  The cheap and easy path is to use USB thumb drives.  But how long do those last? I know that CD-R discs degrade over time (I was one of those geeks burning audio and MP3 CDs in the mid 1990s.)  Most still work, but some are starting to fade already.

I know the old school folks would say to print and bind it, but seriously that would cost me several hundreds of dollars and I'd need book shelves on 3 walls of a bedroom to stack it all. I'd like to stay happily married and don't feel like renting a storage unit to house a SHTF library.

I'm not paranoid about EMP, though something robust in that regard is a bonus.  I'm thinking an external HDD might be practical.  Keep it unplugged and in a safe place.
Also, how long is "long"?  Is USB guaranteed to be an available interface a decade or longer from today?  In my life we've gone from PCs with no HDD, 5.25" floppy, 3.5" floppy, CD-ROM, now the newest laptops don't have any removable media except via USB ports.  Most are designed to leverage cloud storage.  More than I view times I've come across an shoe box filled with floppy disks containing old projects.  Point is, it may not seem that we'll outlive our present digital mediums of choice, but we likely will.

Thoughts, ideas, concerns?

[fritz_monroe]

I'd go with a couple of different devices.  They are all pretty cheap now.  I just picked up a 32G microSD for $7.95.

Maybe get a couple thumb drives, a couple microSDs, and an external hard drive.  You'd only be out abou $100 for all 3.

[David in MN]

I'd love to say a thumb drive with a way to access on cheaply chargable phones.

But I'm the old school guy. My wife studied electrical engineering and I did aerospace engineering and we have a bookshelf literally full of engineering manuals, calculus, and physics. If the world got turned off we'd debate whose control systems book to use. As long as I'm living and have access to my books all the NACA airfoil data is safe. I have vector calculus books and the thermodynamics of propulsion books. Her electrical control system books are priceless.

The value of these books cannot be overstated. I have a Zumdahl Chemistry book. I just picked up my copy of Advanced Strength and Applied Elasticity by Ugural and Fenster. It's not often I recall my days of matrix calculus to solve vector problems with elastic materials.

We've gone over these books before. In many ways they are part of a past life no longer relevant. But our bookshelf is the difference between the stone age and the information age. And both of us wrote in the margins and highlighted. We can navigate them because we know the notes and underlines.

Long way of saying I think you need hard copies. I look like an insane person with dog-eared pages and highlighter everywhere with notes in the margin but that's how my brain works. I'll admit I get lost at times when in the margin del theta becomes del theta hat but I'm so old school that I'm using paper because I don't have a chalkboard.

If any of this makes sense you are beyond super nerd.

[Mr. Bill]

From my closet:



I think the only foolproof (or fool-resistant) method, besides paper, is to save files on a stable and currently-common storage medium (or preferably a couple different media!), and make a fresh copy every 5 or 10 years onto whatever is then stable and common.

I see a number of sites and articles online ranking storage media for their longevity.  Basically, it looks like write-once optical media (CD/DVD/BluRay) are expected to hold data longer than electronic media.  "CD rot" (oxidation of the reflective layer) was an issue with some late-1980s/early-1990s disks, but allegedly is no longer an issue as long as you don't buy cheap junk.

BD-R HTL disks (BluRay Recordable High-To-Low) are supposed to have very good stability because they use an inorganic material for recording, rather than an organic dye.  A single-layer disk holds 25 GB, so your collection would fit on maybe 5 disks.

I've had good luck using GNU ddrescue for recovering data from damaged optical media.  This is Linux software, but I assume there are some similar tools for Windows and Mac.

[Smurf Hunter]

Some good points.

David: for full on text books, I agree that just keeping a hard copy is great.  But that is a different use case than the operating manuals for a dozen tools or appliances.
There is admittedly a lot of random (crap?) like tear down instructions for firearms I don't own, but are commonly found today. For teaching, or preserving knowledge, I'm not at all suggesting we substitute long form academic tomes with "<some subject> for Dummies". However imagine you found yourself in some acute survival scenario - you want details, but relevant, pertinent details that can help immediately. You bring up excellent points, which I hope better people than us are addressing. Maybe there's some secret role within the Library of Congress doing this?


Bill: Yeah, I may just need to shovel everything onto the latest medium every decade or less.  Maybe in 2030, assuming we aren't living out Lord of Flies, 10TB nano-storage cards will be cracker-jack prizes.

[Alan Georges]

One solution:
https://en.wikipedia.org/wiki/Klaatu_(The_Day_the_Earth_Stood_Still)#/media/File:Klaatu.JPG
Good capacity but they're not quite bulletproof.

Bill: Yeah, I may just need to shovel everything onto the latest medium every decade or less.  Maybe in 2030, assuming we aren't living out Lord of Flies, 10TB nano-storage cards will be cracker-jack prizes.

I think this is the way to go.  Several copies on several different types of devices, and pick out a rainy afternoon once a decade or so to move over to the latest and greatest format.  As you point out, at the rate storage device sizes and associated bandwidth are increasing, it's trivial to load things that were considered big ten years before.

[Carver]

I update with every new media. Some old formats cannot be read by newer OS's and programs.
Any magnetic-based media is subject to deteriation.
Vinyl albums will last forever, not tapes.

[Mr. Bill]

Consider file formats also, and whether they will be readable by the software of the future.

Just for fun, I dug up our oldest household inventory file, created in dBase II around 1984 or so, on an Apple II running CP/M.  We did (long ago) get the files off of the Apple II onto some antique PC, and have preserved them ever since, just because.

The most recent version of Microsoft Excel that will read a dBase II file is Excel 2003.  I've been looking for other software that might read it, and I think there might be some tools available (and I might even have an ancient Excel on an ancient PC in a closet somewhere).  But this is unsuitable for a post-SHTF scenario.

This isn't data I need, but it was an interesting exercise (and time-waster).

[Smurf Hunter]

Consider file formats also, and whether they will be readable by the software of the future.

Just for fun, I dug up our oldest household inventory file, created in dBase II around 1984 or so, on an Apple II running CP/M.  We did (long ago) get the files off of the Apple II onto some antique PC, and have preserved them ever since, just because.

The most recent version of Microsoft Excel that will read a dBase II file is Excel 2003.  I've been looking for other software that might read it, and I think there might be some tools available (and I might even have an ancient Excel on an ancient PC in a closet somewhere).  But this is unsuitable for a post-SHTF scenario.

This isn't data I need, but it was an interesting exercise (and time-waster).

If I physically have the bits of an obsolete format on a modern machine, I potentially can doing some scripting and parse out what I need.  This is assuming no proprietary compression or encoding of course.

One of my first professional programming jobs was working as a contractor for the WA state appellate court.  They had an IBM big iron database (DB2?) that was older than me.  So old that it used the EBCIDEC character set, and not ASCII or UNICODE that is common today.  The problem was per project requirements, the database could not be modified, however we needed a "modern" desktop app to read/write from that database.  So I spent a month writing java code (this was 2001?) that allowed the java database driver to translate to UTF-8 so we could actually read the resulting data. In hindsight it was an ugly kludge that mapped character to character, but I guess it worked enough.

I wasn't thinking about archiving things like databases, but that's doable.  Most everything from mysql, mongo, etc. has a way to dump or export.  Ideally you can do this in some document format like JSON.  That allows import into a different platform/schema potentially.  An old school mysql dump really only injects back into mysql unless you do a lot of manual edits.

[David in MN]

Some good points.

David: for full on text books, I agree that just keeping a hard copy is great.  But that is a different use case than the operating manuals for a dozen tools or appliances.
There is admittedly a lot of random (crap?) like tear down instructions for firearms I don't own, but are commonly found today. For teaching, or preserving knowledge, I'm not at all suggesting we substitute long form academic tomes with "<some subject> for Dummies". However imagine you found yourself in some acute survival scenario - you want details, but relevant, pertinent details that can help immediately. You bring up excellent points, which I hope better people than us are addressing. Maybe there's some secret role within the Library of Congress doing this?

Hmmm... You make me think. I tend not to keep manuals and such things as I have a very strong photographic memory, particularly with mechanical things which I attribute to years of Lego. I also get really suspect of owner's manuals of products that don't provide any value for troubleshooting and (if I'm blunt) were written by lawyers to avoid lawsuits. So even my documents on, say, the 1911 are long form books with very critical detail written by gunsmiths.

But we did used to have this system called microfiche that *might* have some value here. I could jury rig a reader pretty easy and you could carry quite a bit. I'll confess I keep the old Boy Scout Handbook and a copy of the SAS Survival guide in the car as emergency reference. Just a few pages from those on a small microfiche would e pretty powerful.

Here's another kooky idea (while I'm in blue sky mode)... How about an audio recording? If you could audio record your data even a crappy 1980s walkman would do the trick with some rechargeable batteries.

Also utilize photographs. I get a bevvy of food and wine magazines and when I run to the grocer or cheese shop I take pics of the recipe on my phone and go from there.

I think back to the film The Book of Eli where Denzel has a Bible on Ipod. Lots of ways to store information.

[FreeLancer]

Eli’s iPod was just for music. His Bible was stored (and backed) up on more mundane media.

[David in MN]

Eli’s iPod was just for music. His Bible was stored (and backed) up on more mundane media.

Do I remember wrong? Eh, it happens. Wasn't a great film as I recall.

[FreeLancer]

It was integral to the big reveal.

[Smurf Hunter]

It was integral to the big reveal.

Right. HE was the bible.

Anyhow, I don't see any logistical advantage in moving from one potentially obsolete or fragile medium to another.  I think an SD card is at least as durable as a magnetic cassette tape if stored safely.  That's the category of archives, or the encyclopedia of stuff we value and want to retain.

There are also cheat sheets and quick instructions.  Does your 12 year old know how to start the generator?  Does your spouse know how to estimate dosages for fish antibiotics?  Any single one of these warrant a single laminated page, but when you end up with 100 such pages, that changes things a little.

I could quit working today, and spend 40 hours a week until I'm at retirement age and probably couldn't document half of the crap in my brain. Truth is most is only interesting to me, but who knows what might be handy in the future...

[Mr. Bill]

If I physically have the bits of an obsolete format on a modern machine, I potentially can doing some scripting and parse out what I need.

Yeah, I was just looking at my old dBase II file, and it's just a 520-byte header followed by plain ASCII text with fixed field and record lengths.  I can read it with 'less' and LibreOffice Writer.  It would probably take me a few minutes to get it all into a spreadsheet.  But most people don't have the skills to do that, and in a no-Internet situation it would be difficult to learn how.

Gosh, I'm having fun looing at that old household inventory data!

• Apple setup (Apple II+. Serial, CP/M, memory, and Videx video cards. 2 floppy disk drives. Amber monitor. Fan & surge suppressor. Joystick.) - $1200
• Two boxes for 5 1/4 inch floppies, total capacity 120 disks - $50
• Daisywriter printer, tractor feed, separate power supply - $500
• Boomerang - $20
• Geiger Counter kit - $130
• Bird-of-paradise tile picture - $10

(Wait, what bird-of-paradise tile picture?  I don't remember that!  And I really ought to build that Geiger counter some day...)

[/reminisce]

Ahem.  Anyway, I was more concerned with file formats for text+image information.  I expect ASCII, UTF8, and PDF will remain supported for the rest of my life; ditto for even the earliest versions of HTML.  Microsoft Word 97 .DOC files ought to last another decade or two.  E-book formats, though, I'm not so sure.  My wife has a huge collection of public-domain e-books in EPUB format, which is non-proprietary and (in 2019) common, but may become less common if Amazon continues to grow the market share for its proprietary formats.  You can convert between EPUB and other formats, but only if you've downloaded a copy of Calibre before TEOTWAWKI and have somebody who knows how to use it.

[FreeLancer]

Is there any consensus on the expected reliable lifespan for SSDs when powered regularly vs sitting in a safe?

[fritz_monroe]

Is there any consensus on the expected reliable lifespan for SSDs when powered regularly vs sitting in a safe?

Brief search says that flash media is typically good for 5-10 years.  Since SSD is basically just flash memory, that should last about the same.

But the interesting thing I hadn't heard about is   M-DISC.  Supposed to last up to 1000 years.  Actually, maybe I had heard of it and didn't really pay attention to it.

[Smurf Hunter]

Conventional flash media lifespan was limited by the amount of read/write cycles.  So theoretically, if you bought a few new USB drives, copied over your content and stored them under good conditions, you should be able to get a decade or longer.  It stands to reason the newer stuff is slightly better, but we also have no real evidence.

Mr Bill.   
I sometimes miss dot matrix printers.  They were noisy and the paper ugly, but they seemed more reliable in many ways.  I have school aged kids and our household is lucky if an inkjet makes it 3 years.  It's awful, and you cannot easily dispose of them when they completely fail.  Don't get me started on ink cartridges...

[Carver]

The Dead Sea scrolls survived eons; scribe it on a durable media and hide it in a cave. Translate that into present day terms.

[Bradbn4]

There is such a thing as 'bit rot' for magnetic media. EBCIDEC is the Big blue (IBM) method for character representation.  The more the media is used for example SSD/Flash memory the quicker the media 'rots' out.

There use to be a conversion program(s) that would allow you to decode Dbase II or WordStar programs into other formats.   This was back in the Windows 95 days....I just checked last week that my old hardware that ran the program has decided it was time to be recycled.

The Gutenberg project was concerned about long term format readability.  At first the only provided the books in 8-bit ascii.  Now they offer the media in a few additional formats...but always 8-bit ascii.  The Gutenberg project is open source books, etc that can be downloaded freely.

https://www.gutenberg.org/ 

I use to own a copy of one of those conversion programs to let me read LIT books on my dedicated book reader.  That same function can be found in Calibre.  My current go-to book reader on Windows & it allows me to sync to ebooks.

One place to look for conversion software.
https://www.softpedia.com/downloadTag/conversion

Right now I keep backups on USB memory drives, external memory drives, and a old file server.  The hard part is to keep the multitude of devices "synced".