Data Storage

[stand@desk]

Looking for ideas for improved long term data storage..

I currently use four flash drives to back-up my data..I have experimented with using "cloud-based" storage solutions but have not consistently used one. Plus, I am not sure how "safe" it is to store personal data on the cloud. What happens if the cloud goes down?

I opened some pictures from one of my flash drives today and some of the pictures were distorted..I suppose flash drives have a shelf life before the files saved on them become distorted and unrecoverable..It made me realize that our data may not be as secure as we may want to believe it is..

I know ERErs are more likely to be concerned about their data storage than the masses, our data is a big part of our lives..I'm looking for a strategy to keep my data safe for the long term with less maintenance, and to hear about others' experiences with data storage.

Happy EREing!

[rube]

I use the cloud to synchronize my data with two laptops. So I got it at three places: laptop 1, laptop 2 and online (cloud).
Once in a while I make also a copy that won't be synchronized with the cloud, just in case.
However, I do this is only for my administration (I scan everything) and a few other files.

For other files (pictures etc.) I use a NAS to make back ups at irregular intervals. Plus, laptop and NAS are often at the same location.

So I'll be interesting for solutions of others also.

[slowth]

A proper data archive must have at least two sources, local and offsite. Local is easy. Use whichever archive medium you prefer. Offsite is critical. I will never use cloud storage. I'm not placing my data in the hands of some unknown party. I would go to the house of a family member or friend and place something like a NAS on their network. I would then use some tool like rsync to perform local and offsite daily archives. Aside from the initial setup, the offsite backup is not high maintenance.

[jennypenny]

I have two seagate external hard drives. One is always connected to the computer and a full backup runs every night. The other resides in a fireproof safe. Once a week, I switch the two drives.

[apocryphal]

Crashplan is the most interesting option I've found, they allow you to backup to any or all of 3 locations:
- their cloud based service
- to a 'trusted computer' over the internet (a friend or family member for example)
- a local drive

If you use a friends computer or local drive it's free. All your data is encrypted both over the network and on your friends computer so they can't access your data. You can choose what time the backups happen, etc. No need to leave your computer on all the time if you don't want to. I don't use it myself but know several people who do.

[George the original one]

What data do you have that is so personal that you wouldn't put it on the cloud?

My financial records are stored at the financial institution and I have a local paper copy. Any other versions are unofficial and somewhat obfuscated, so I don't see why they can't be trusted to the cloud.

Birth certificates, passports, drivers license, etc. don't exist as electronic records.

Anything else electronic is merely "noise". I'd mourn their loss, but life goes on unimpeded if they're lost.

[slowth]

All your data is encrypted both over the network and on your friends computer so they can't access your data.

Using a cloud backup service and letting them encrypt your data is a terrible idea. You no longer control your own data. Encryption is good, but do it on your end.

[slowth]

What data do you have that is so personal that you wouldn't put it on the cloud?

It's not that's it's personal. Encryption would take care of that. But they control your data. You are trusting some company to control a critical backup of your data. This is foolish.

[lilacorchid]

If it were something that was critical, I would back up to an SSD and a HDD. If your HDD (moving platter type of drive) were to suffer a hardware failure, it is easier (most of the time) to recover the data. An SSD offers speed and does not degrade the data, but if it fails, you are pretty much SOL as far as recovery goes. A USB stick will not degrade the data either.*

I see a lot of HDDs fail at work. Moving parts and all.

*Everything is subject to failure, but an SSD and USB stick are solid state and don't have moving parts to break.

[tzxn3]

http://ollydbg.de/Paperbak/

Acid-free paper is the only easily available information storage medium known to last centuries.

[luxagraf]

> Anything else electronic is merely "noise". I'd mourn their loss, but life goes on unimpeded if they're lost.

The more I back things up, the more I think it's all noise.

That said, I make local backups to two drives, send encrypted (on my end) copies to Amazon Glacier (using Arq), rsync copies to a cloud server I control (insofar as I have root access, not that the physical server is under my control) and I also make a physical offsite hard disk backup that I mail back and forth with my father (I get his data, he gets mine; we swap disks about twice a year).

That's what you call overkill, but all told it only costs me $8 month (which is $5 a month in server fees (which I would pay anyway since that server also hosts sites which directly/indirectly make the bulk of my income) and ~$3 in physical hard drives (~$80 drives replaced every roughly every 4 years, and that price is always dropping) [edit: forgot the $1.24/month for Glacier]. So, not particularly pricey, not that much work once it's all set up, but...

For long term backups, I'm more worried about data storage formats than the ability to keep the 0s and 1s around. It's one thing to keep files around for 20 years, even 40 years (I still have files from the 1980s), it's a whole other thing to actually open them and see them rendered properly in modern software. Personally I stick to text files only for long term backups.

[JohnnyH]

I keep 90% of my programs (portable) and all my personal data in a TrueCrypt drive... Once opened I'll run an incremental sync to either my work or home computer. So, one copy work, one copy home, one copy with me on SSD external... No need for cloud, especially since syncing a multi-GB file would destroy bandwidth.

[henrik]

It's not that's it's personal. Encryption would take care of that. But they control your data. You are trusting some company to control a critical backup of your data. This is foolish.

If you put it on a hard drive, you're trusting the company who produced the hard drive and possibly your electric company for current stability. If it's not about the privacy, then what's the difference? I'm pretty sure the chance of failure is a lot lower in a cloud service than your own or your friend's hard drive.

[Chad]

I have two hard drives for backup of my SSD. I keep one on hand at all times and store the other one at my parents house. I switch them every month or so.

[henrik]

I thought about it for a while, but I really cannot think of anything in my personal data that would cause me grief or problems when lost. Work stuff is backed up at work and for my own stuff I have an external drive at home. My laptop is mostly not at home, so it's sort of offsite, I consider that enough. I have very rarely needed anything that goes back more than a year or two.

What I'm much more worried about though is my government's enthusiasm for everything electronic. Birth and marriage certificates, house titles, taxes, company registry, drivers licenses-- practically nothing is done on paper any more. Even a growing share of civil contracts have digital signatures. It's great for ease and efficiency in getting everything done, especially remotely, but I sure hope they have some really off site backups, hopefully on the moon or thereabouts.

[slowth]

If you put it on a hard drive, you're trusting the company who produced the hard drive and possibly your electric company for current stability. If it's not about the privacy, then what's the difference? I'm pretty sure the chance of failure is a lot lower in a cloud service than your own or your friend's hard drive.

The risk of hard drive failure is why you have multiple backups. You are talking about two totally separate issues. What's the difference? Trust a cloud company with your critical data, important enough that you would make multiple backups vs hardware failure. See the difference? A company which has no regulation whatsoever and can hold your data ransom if they so choose. It's utterly idiotic to put your data in the control of someone else. The cloud company ultimately controls your access to your data. That's the difference.

[JohnnyH]

I wouldn't trust cloud only storage, or a propriety encryption... For one, I don't even have internet (outside cell) at my house. Two, I'm often traveling with no internet.
I don't even trust purchased content to the cloud... Amazon might decide I'm a bastard and never give me access to the books I bought ever again.

I use cloud storage for critical data, but I have local copies and I encrypt it (with massive key) before I upload it.

[jacob]

"Real men don't backup. They cry."

1) For really important stuff, I only trust paper. I print things out and put them in a fire safe.
2) For semi-important stuff, like work output, I rely on libraries around the world to preserve my publications. The philosophy here is that if it's not published, it's not important.
3) I have other people look out for my stuff. I rely heavily on cvs and the likes for work stuff.
4) For sentimental stuff, like pictures, music, old essays, homepages, ... Well, I don't really care so much if I lose it, because I find I never look at it anyway. If you want a picture of what I looked like when I was 20, you won't get it from me.

In fact, I think I spend much more time worrying about losing old data than actually using it. There's only been a few cases where someone has requested some work/results/stuff I did ten years ago and it wasn't easily available. In general I store old stuff in a haphazard way in multiple places in multiple formats and on multiple continents(!) I can usually dig it up, but it would take some time. If I can't dig it up...the world moves on.

For most of my life, I've been using *nix, so practically all my data is stored in text files and thus still readable. That's more luck than intentional design. The only regret I have in terms of data storage is some music I composed 15 years ago that was stored in some ancient cakewalk format that I can no longer access because I can't find a computer old enough to run old software I no longer have. Labs often have the same problem. What do you do with 50 year old data stored on an 8 inch floppy?!

[henrik]

The cloud company ultimately controls your access to your data. That's the difference.

Ok, you're right about access. But like you said, that's why you have multiple backups:)

[lilacorchid]

"Real men don't backup. They cry."

+1

I have seen near tears from grown adults about work files. When given the choice whether they want to pay up to $5K to get the data back, they always decline, which tells me they are nearly crying about something that isn't even that important.

Hard drives fail. Usually you get a lot of warning if you know what you are seeing, but not always.