The Old Joel on Software Forum: Part 3 (of 5)

Very long term storage.

I believe cave drawings are about the oldest thing we can find in an archive type environment. Those of us who wrote on a TRS-80 or saved to 5.25" floppies or reel tapes are finding it difficult to bring that information back.

So...if we want to store information. Long term (200+ years) how would we do it. Computer information seems wrong. Even if you have the storage (if a DVD would last 200 years), like reel tapes, the devices to read it would be gone.

As technology, at least at the state it is in today cannot be depended upon for long term storage, what to do? How do we store our family records, journals, and blogs to be enjoyed by future generations, without setting aside a climate controlled room?

Oldman
Thursday, September 25, 2003

My money is on biotech to solve this problem. We could design a single-cell organism with a portion of its DNA encoded with the information we want to store. The error rate in DNA during single-cell reproduction is vanishingly small, so with the appropriate checksums and a few petri dishes, we could have data storage lasting millions of years.

Yes, I read way too much sci-fi when I should be working.

Craig
Thursday, September 25, 2003

:The error rate in DNA during single-cell reproduction is vanishingly small:

Are you sure about that?

Francis Crick
Thursday, September 25, 2003

"Are you sure about that? "

Hey, it worked on Star Trek, so it must be true.

Craig
Thursday, September 25, 2003

The trick is to constantly keep moving the data to new storage as it comes out.

If you have anything on floppy now, move it straight onto recordable CD or DVD. When the next technology comes out, move the data from the CD/DVD onto that.

The same thing for media also applies to software. If your original app is so old that you can't find a computer to run it then it doesn't matter if you can read it off the media. You need to keep the data in a format that is readable.

Almost Anonymous
Thursday, September 25, 2003

While not actually practical for every application, I don't think there are many long term storage systems as effective as a book.

Kero
Thursday, September 25, 2003

In case of a dystopic tech-poor future, I'd consider etching it on metal in tiny writing. With macroscopic instructions and maybe a rosetta stone. I'm sure there's a materials engineer that can fill in the missing details like protecting the etching surface so it can be viewed without danger of scratching. If enough demand can be spurred, a factory can be made for this.

If this is prohibitive because of sheer mass of data, you can go digital and etch onto metal the specs of the device required to play it. It should be a device engineered for ease of production. Perhaps you can etch enough information so a dystopic world can figure out how to develop the tech required. Mention there's also the history of pornography on the media, so people will have additional motivation to do it. Unless they're a different species or something... hopefully the perverts will be the technically gifted ones...

Tayssir John Gabbour
Thursday, September 25, 2003

Books have so far been the absolute best recording devices.

But while the medium which the information rests on has finite properties, the actual information does not. And just as importantly are the required applications that can make use of that data.

I predict that computer technology will evolve to such levels of power that today's current machines will be nothing more then mere "emulations" within its data banks. We already witness this today with so many game platform emulators, and ofcourse our beloved Amiga emulator (et al.).

The transition will always be constant, and no doubt there will be a percentage of information lost. As the massive knowledge base for human beings expands, we will continue to abstract ourselves from understanding every level of detail and just expand on the applications of it all.

Whoa! Sorry, I guess dozed off there and digressed to far.

:}

sedwo
Thursday, September 25, 2003

Awww, c'mon!

Write your personal archive on CD's, DVD or even a large IDE drive.

Make sure you have two copies, stored at different locations.

When you feel that the current media you are using for storage is not used anymore, simply copy it to new media (SuperUltraMega-DVD Pro Turbo II, or whatever they will have in 2010).

It's that simple.

It's no rocket science.

I don't know why this question appears here and on slashdot again and again!

software enthusiast
Thursday, September 25, 2003

I've already had an argument with my dad about data storage. I sent him something in ASCII and he said "why didn't you send it in MS Word, it looks so much better?".

I explained how I can still read with Mozilla my UNIX email files from 1990, while he had trouble exporting Outlook email to another computer with Outlook!

And I still can read all the ASCII files I saved 15 years ago.

It'll be a very enlightening when 5? 10? 15? years from now we won't be able to read and convert our MS Word '95 files...

Dave Torok
Thursday, September 25, 2003

"It's that simple.

It's not rocket science."

The problem with data that you want to keep is that often you don't realize that you want to keep it until far in the future, probably long after you should have done several technology shifts. Secondly, I agree with the poster that it _is_ an issue -- How much technological baggage will we all be converting and restoring every couple of years, building more and more of a process. By the time I'm 50 I'll have to dedicate myself full time to re-encoding video files, converting data files, etc.

Dennis Forbes
Thursday, September 25, 2003

Just curious - what are you storing that needs to be kept for 200+ years?

Nick
Thursday, September 25, 2003

Nick: 200 years? If you have data from an apple from say, 12-18 years ago, you may have trouble getting it: some of those have different trakc lengths than are used now, if you can get a working 5.25" drive. And the software to read it.

If you want it for 10 years, that's doable with foresight, If you want anything for any more than that, you have to work at it. Things change to fast nowadays.

Mike Swieton
Thursday, September 25, 2003

~~~ By the time I'm 50 I'll have to dedicate myself full time to re-encoding video files, converting data files, etc. ~~~

I consider this a non-issue.

I remember the migration from C64 to PC, from 5"25 floppies to 3"5 floppies, from 3"5 floppies to CDs, etc.

It's not that bad!

In the future:

Storage devices (SuperDuperMegaRambo-DVD II TurboPro 6+R) will be so large, that you'll take 10 normal DVD and store them on a RamboDVD II ... .

Processor speeds will be so high, that you'll encode your 100 DVDs worth of movies to GigaDivX PRO 2015 in just minutes.

software enthusiast
Thursday, September 25, 2003

This is attempting to predict the future, isn't it? There's really no way to know what storage mediums will be in effect 200 years from now. Crystals? Holography? Heck, at the very least, the storage format will be wildly different.

I say, just make sure you keep your *data* on a current storage medium. If it's on an obsolete or near-obsolete storage medium, move *all* of that data to a new storage medium, and convert it to an ubiquitous format if necessary.

For example, I still have some documents on 3.5" disks, in MS Works for Windows 3.1 format. I should copy those all onto my hard drive, convert them to something like RTF, then burn them onto a CD.

The Pedant, Brent P. Newhall
Thursday, September 25, 2003

Paper!!!

1500 year old books are still readable.

Is that good enough?

--
ee

eclectic_echidna
Thursday, September 25, 2003

>It'll be a very enlightening when 5? 10? 15? years from now we won't be able to read and convert our MS Word '95 files...

Dèjá commencé. Because of the binary OLE embedded objects, you can have a word file which you can only read part of. Unless you have an old machine with all the right components installed.

WordML is a start in the right direction, but it still embeds binary blobs for many things.

mb
Thursday, September 25, 2003

Ummm on DNA error rates. DNS is a redundant tree, the scriptase is meant to strip in sequence and copy in sequence but occasionally it gets dropped on the way or recombined in a different order that switches on some other behaviour.

As this is happening in lots of cells all of the time its not such a big deal, but its not a great way to encode how many lightbulbs were sold in June.

But it wouldn't be a bad way to store the method of creating new lightbulbs.

Currently all electro-magnetic methods of storing data have relied upon the medium to remain relatively stable over the required time.

All of them, save possibly one, have suffered from one kind of deformation or another. Mylar becomes brittle and the oxides painted upon it start to shed;Hard drives wear, and if the platters themselves don't degrade the read electronics do, or the motors give out; Pre-recorded CDs if kept out of sunlight and in completely stable ambient conditions may last between 25 and 30 years but again the numerous rare earths and metals have a tendency to want to depart from the plastic; Writeable CDs now seem to barely last the time it takes to write them.

The one medium which seems quite stable (but isn't old enough to really tell), is the magneto-optical disk, the re-writeable CD. Because of the way the ablateable material is sandwiched in the disk it seems that if the disk is kept under optimum conditions (which none of us probably do), then it seems that it may outlast the pre-recorded CD.

But even so, it wouldn't be that long.

A non electro-magnetic solution might be to etch the information, much the same way as processors are etched, but in a material that is extremely stable under its normal conditions. I don't know what that material would be, none of the silicates, perhaps somethine like tungsten or platinum.

And as it was an etching and not some kind of magnetic domain you'd need a photo-optical reader of some kind, probably not a very quick retrieval, and definitely a WORM arrangement.

Come to think of it this could be a reasonable defence in any RIAA action for copying CDs.... 'I was just preserving my collection guv...'

Simon Lucy
Thursday, September 25, 2003

Spot the deliberate mistake and why I'm not a biologist.

Simon Lucy
Thursday, September 25, 2003

I was a fairly early adopter of rocordable CD technology. We've all heard how CDs should last 100 years or more. Well, they don't. I have CDs recorded from 1995 to the present using a variety of recordable media and NONE of my CDs from before 2000 are readible. I keep the CDs in a cool, dry, dark place and seldom use them, so it's not a situation of the media being left on the dashboard of a car.

Do NOT trust recordable CDs for long time storage.

X. J. Scott
Thursday, September 25, 2003

The problem with books are that they are very heavy, can't be copied for offsite archives, can't store audio and video and are susceptible to moisture and fire.

By the way, the experts expect that hard disk sizes will continue to increase by orders of magnitudes, but the access times won't. You might have a exabyte of storage, but it will take a year to search.

pdq
Thursday, September 25, 2003

200 years is rather extreme. That vast majority of data is just not important enough to be kept for that long and those things that are important are likely to be maintained by copying from one medium to another. You can get COTS hardware that can read/write media that has a shelf life of about 50 years. Specifically, magneto-optical media will last that long. Also, because it is commonly used for research data and for storing medical records which have to be kept for a long time for legal reasons (for example, mammograms have to be kept well past the lifetime of the patient), you're likely to avoid the problem of having media that cannot be read because the hardware to read it no longer exists.

As an alternative, you just keep everything in RAID on redundant servers and on a tape backup system and copy the data from the old RAID to the new one when you upgrade the server.

Matt Latourette
Thursday, September 25, 2003

The best solution is to travel back into the past as needed when you need the information.

Oh by the way, say hi to Marty McFly for me !

F
Thursday, September 25, 2003

Etch-a-sketch

Ben
Thursday, September 25, 2003

I believe the Harpers of Pern had this exact same problem.

They turned everything into music, which was then mandatory to be taught to the youngters, they would sing it all as entertainment, and in this way the next time the thread fell, they knew to hide in a cave and watch the dragons flame it all in the air....

Aussie Chick
Friday, September 26, 2003

Outsource your storage requirements.

Then when you've passed away, and nobody pays the bill anymore, it'll all be deleted and nobody will care.

Problem solved.

Realist
Friday, September 26, 2003

hmmm... The idea of passing on passing knowledge on via musical instruction gave me an idea.

First off, let's assume that information is free and open -- no copyrights or impedance of access for commercial gain. Secondly, assume that all those involved were maximally altruistic, and gave consideration not to their own interests but for the interests of all those involved in the idea I am proposing. Lastly, assume that this idea appeals to huge pool of people.

The problem?

The permanent storage of information. An eternal Library of Alexandria.

The idea?

1. Every participant allocates a certain amount of electronic storage, accessible at any time to any other participant.
2. Each "text" will be indexed, accessible via a unique number.
3. The location of any "text" will not be fixed or tied to any one location.
4. The "texts" within this system will be transfered, copied, and stored in a completely *random* and automatic fashion in the storage locations of all participants.
5. Any text deemed worthy by any particpant may be entered into the system.
6. The one aspect that I think will be hardest to enforce -- There will be no deletion of any "text" from the *system*, for any reason. (individual "texts" may be deleted, as long as another copy exists in the system).
7. Accessing the "texts" will be done via a P2P mechanism.
8. All "texts" will be in a commonly understood format.

From these rules will emerge the system that will store information with permanence and complete accessiblity for all time.

The one problem I see is the ballooning of the storage required due to the requirement that no text may be erased from the system. BUT, won't that be solved by the increasing effeciency of the storage mediums of the future?

Shawn Leslie
Friday, September 26, 2003

I think the Harpers of Pern would be more User Friendly.

Aussie Chick
Friday, September 26, 2003

There's all that tedious mucking about with dragons though.

There is always a downside.

Simon Lucy
Friday, September 26, 2003

send out all your data as radiowaves to the universe. it will bounce back and forth between planets, galaxies. then you need a telescope to read back the data whenever you want.

it will be there for million years. the problem if the universe will shrink into one black hole it will be difficult to read normal data from the noise :)

na
Friday, September 26, 2003

Their are two proven technologies in the field of extreme-long term data storage are:

A) write it on clay tablets Sumarian style (the market leader in the field, used by most of the ancient civilisations)

B) carve it on the walls of your temples (a technology heavily used by the Egytians but with little market penetration elsewhere)

Both have a track record of preserving at much of the data stored on them for six thousand years and are still going strong. With proper care and attention lifetimes beyond that of the human race (theprefered retrieval mechanism) are theoretically achievable. Oh and both are free of licensing considerations, with only the one of cost of creating the records.

Books are for early adoptors.

bil
Friday, September 26, 2003

Stone circles are good - low bit density but good reliability!

On a more serious note, glass photographic plates are very good. I worked in astronomy where all sky survey photos are taken on 14" square glass plates.
Some plates are >100 years old since you want to see how something has moved over these timescales.

These plates are scanned to find individual objects. It's expensive having to keep copying the scanned data to new formats ( 9track - exabyte - dat - cd -dvd ) and none will last 100 years.
The latest idea was to print the digitised location data back to glass plates as a 2d bar code which can then be read with the same scanning machines as the original plates!

MArtin Beckett
Friday, September 26, 2003

I struggled with this last year. I helped someone retrieve the text of a book he wrote in the 80's on an Apple IIe with AppleWorks, stored on 5.25" disks.

Luckily, we found a transition machine, an old Apple IIgs with a 5.25" drive and a 3.5" drive. After copying the files to the 3.5", we were able to load them on a Macintosh computer. (We had to cover a hole in the disk so it appeared to the Mac as a low density disk). Text formatting was minimal and in the form of text markups. If we didn't have the Applie IIgs, or if the file format was in binary, this would have been much harder.

The upshot is that practically speaking, a 15 year-old retrieval can also be challenging.

Will
Friday, September 26, 2003

I've yet to see a 1,500 year old book wiritten on paper, though presumably Chinese example exist.

Modern paper is so cheapskate it wll discolour and get brittle in a short time, and that is not taking into account the fading of the ink!

Carving on temple walls or clay tablets has a very, very low chance of surviving. The materials are too valuable and are nearly always recycled (and in the same vein, make sure your descendents don't keep goats if you leave everytning on paper).

The answer of course is to keep transferring your data to modern media. Not all of it will get transferred, but we only have a very small proportion of all the data produced in the last few hundred years anyway.

The main problem future historians will have is not the loss of data, but the overwhelming amount of it that has been saved.

Stephen Jones
Friday, September 26, 2003

According to the National Archives and Records Administration, acid free paper will last several hundred years "without significant deterioration under normal use and storage conditions."

So, if your great-great-great...great grandchildren discover the paper is deteriorating, they can use a photocopier to transfer the information to fresh, acid-free paper.

It might also make sense to print the information in plain English, using an OCR font, in case someone wants a digital version to play with.

Russell Thackston
Friday, September 26, 2003

"I've yet to see a 1,500 year old book wiritten on paper"

The Dead Sea Scrolls would be one example.

Dennis Atkins
Friday, September 26, 2003

More like "scroll fragments"...

Rick
Friday, September 26, 2003

Ask the Long Now Foundation ( www.longnow.com ) !

Although famous for their clock, they are also interested in preserving data. Tayssir John Gabbour posted about tiny writing etched in metal, with readable instructions.

Their solution is rather elegant, the text is written in a spiral, where the text size gradually decreases. The first part of the message is large enough to be read by the human eye, then you'll need a magnifying glass etc.

I'm sure that's what the www.rosettaproject.org site is about, but the link seems to be down.

AJS
Friday, September 26, 2003

Use mylar paper tape http://www.westnc.com/mylar.html . You can fit about 1MB source on a spool about 10" in diameter.

Bill Rayer
Saturday, September 27, 2003