More Digital Big Bang -- IDC Study

A while back, I wrote about what I called the digital big bang -- a rapid transformation in human society brought about by the ubiquitous growth and use of information.

It turned out to be one of my more popular posts, maybe because I took a stab at looking at some of the fundamental social issues that were in play as a result of this transformation.

Today, IDC released a study (sponsored by EMC) titled The Expanding Digital Universe.  In this research, they try to estimate the total amount of information we've generated, and -- more importantly -- how fast it's growing.

We had worked with UC Berkeley on an earlier study several years back; we thought it'd be great to have an updated view on how things had changed.  And, boy, had they changed.  Much bigger numbers, to start with.

So just how much information have we generated so far? 

The answer is -- drum roll please -- 161 exabytes of information created by humans through 2006.  For those of you who lose track of zeros, that's 161 billion gigabytes.

So what does this mean?

If you read the study and some of the press releases, the paper-based colorful analogies are kind of fun.  Multiple stacks of books from the earth to the sun, covering vast swaths of geography with feet of paper, tons of books for every human on earth, and so on.

Good thing we have digital storage.

But once you get past the fun analogies, there's some serious stuff.

First, the growth estimates are kind of sobering.

Six times as much information by 2010.  That's not so far away. 

In three short years, every byte in the world will now be replaced by six.

Second, the study states that organizations will be responsible for managing (security, privacy, reliability and compliance) 85% of the information. 

That's most of it, folks.

To me, that's even more sobering, given that most organizations are struggling mightily to do these basic information management tasks for what they already have, let alone six times as much information.

What This Means To Me

During the early discussions around global warming, much debate on the data -- was it happening, or was it not?  As the data came in, the conclusions were clear -- things were getting warmer. 

The global warming debate shifted to what to do about it.

I look at the IDC study as a relatively conclusive study that says the facts are in. 

The debate should now shift to what to do about it.

As such, I see it underscoring much of what I write about here:

• Social issues, such as who owns my information? 
• Corporate governance issues, as in who's responsible for managing this most important asset? 
• IT role evolution: are we technologists, or are we informationists? 
• How does IT get quick wins to wrestle the information beast to the ground?
• And do we need to think in terms of new tools to handle this information?  See here and here.

One thing is clear -- we're in the middle of a digital big bang. 

And things are changing very, very fast.

Comments

[Fred-san]

I doubt those figures. There is way too much double counting and bloat-up going on. Everytime someone forwards a funny e-mail with a jpg, it's saved onto hundreds of PC's and servers. When you create an almost blank Word document, that takes up a meg or more. One site I know of has over 50TB of storage and guess what, it's 30 copies of a single 400GB database, plus lots of logs, backups and just-in-case-we-need-it dumps. To me that's a 400GB database not a 50TB storage requirement.

[Chuck Hollis]

I think you bring up some good points.

First, I would agree that there's lots of inadequate information management by many, many of people. I know I fall into that category.

More importantly, the study looked at the amount of information we as a society generated, not what we actually attempted to store and manage.

The first number is much larger than the second.

Even if you look at the IDC study's numbers with a healthy bit of skepticism, it's very very hard to deny the underlying trends, at least as I see them:

1 -- our society is rapidly being transformed into an information economy

2 -- this is causing the creation of a mind-boggling amount of information that is growing at a near-exponential rate

3 -- in addition to the usual technology concerns, this raises important issues in both our society and the business world around us

4 -- IT thinkers have a role to play in helping all of us come to terms with the digital big bang.

Thanks for reading and commenting!