Conventional wisdom considers Moore’s Law to be IT’s primary governing force but data storage evolution has an equal if not more profound impact on the ways that consumers and businesses use technology. That’s because storage advances more quickly than microprocessors (doubling in performance every 12–14 months, compared to 18 months for CPUs) and growth in storage capacity runs at a fairly steady 60% YOY.
That last factoid points to storage’s intimate connection to the subject of big data. Why is that the case? Partly because attention to the subject of big data began to coalesce as reports like “The Expanding Digital Universe” study (2007, updated in 2011 by IDC, the University of California and EMC) explored the challenges and opportunities organizations faced in capturing full value from their ever-growing storage infrastructures and investments.
In fact, many big data solutions leverage large capacity SATA and other high-volume drives and array technologies to support core analytics features. Additionally, non-mechanical storage technologies, including capacious solid state drives (SSD) and Flash memory often have roles in high performance analytics and data warehouse offerings that play in or near to the big data market.
Overall, without the benefits of 'big' storage technologies the performance of many of these solutions would likely be small potatoes. So is big storage an absolutely necessary part of big data? Actually, no. For example, consider stream computing solutions which are capable of supporting a variety of often highly complex real time analytics processes by means of modest commodity and even generic storage and other hardware components.
This might seem contradictory but it simply underscores the multi-dimensional nature of big data strategies and offerings. Just as no one tool can be used to complete every job, no single big data technology can address every challenge or answer every question.
However, that itself is a critical point. It is the rare craftsperson who only requires a modicum of tools and the lucky company that faces few and simple challenges. These are good points to remember when considering which big data vendors and which solutions best fit the needs of your organization.