Data Diaries: Data expands to fill the space available

Data DiariesOur world is increasingly data-driven, to the extent that many point to data as being the world’s most important natural resource of the 21st century. And yet, unlike many other natural resources such as oil, gas or water that are constrained in supply or finite in nature, data is truly unlimited. The world’s data is growing at an unparalleled rate thanks largely to technology.

But how much data is there in the world? How fast is it really growing?

Well it turns out, these are not particularly new questions to ask. In fact, people have been trying to tackle this conundrum for decades.

Back in the 1940s, Fremont Rider, a Librarian at Wesleyan University in Connecticut, was pondering the same thing, having noted that American university libraries were doubling in size every sixteen years. Applying some simple arithmetic, Rider speculated that the Yale Library in 2040 would have “approximately 200 million volumes, which will occupy over 6,000 miles of shelves, requiring a cataloguing staff of over six thousand people.”

By the 1960s, other scholars were noting similar trends in their own fields, and yet the increases were no longer linear in nature. Derek Price concluded that the number of new journals had instead grown exponentially, increasing by at least a factor of ten during every half-century.

When the Sloan Digital Sky Survey started in 2000, its telescope in New Mexico collected more data in its first few weeks than had been amassed in the entire history of astronomy. This discovery once more suggests that an even steeper curve exists in terms of data growth and volume.

But what about if we consider the growth of data in all forms, fields and domains of humanity – how much is there?

There's clearly a lot more data in the world than ever before, but it's very hard to conceptualize how much there really is. Matthew Stent

Google’s CEO Eric Schmidt famously estimated that the world creates around five exabytes of data each and every day. That means that every two days we create as much data as we did from the dawn of time up until 2003.

Think about that for a moment… 2003 is well within our recent memory: many of us owned laptops, mobile phones, digital cameras and had broadband Internet in our homes at the time. And yet we eclipse all of that data-making in just the short span of today and tomorrow.

And now the real killer is that Eric Schmidt’s prediction was one he made in 2010, before the world had even heard of Snapchat, Pinterest, Whatsapp and Fitbit, to name a few.

I’d even go so far as to predict the world is now creating as much data every few hours as from the start of time through 2003.

And that data acceleration shows no sign of abating any time soon…

Data did that.

A close-up on Matthew:

Matthew Stent leads the IBM Center for Applied Insights team in Europe and has a long-held passion for championing market intelligence and insights within IBM. His past fields of expertise are broad, covering a variety of industries, geographic regions and technologies. In his current role, Matthew remains focused on delivering deep insights that make a tangible difference to IBM and its clients. For the past eight years, Matthew has resided in the forests of central Sweden where he loves to travel and experience the great outdoors.

Fun fact: Matthew owns three chainsaws and is not afraid to use them!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Pingbacks & Trackbacks