Got Storage? Splunk It

Data Storage Infrastructure: The Bedrock Of Today's Infrastructure Data storage requirements are growing. Recent Gartner estimates put this growth rate at 800% over the next 5 years for the average enterprise. Digitisation and widespread server virtualisation are just some of the forces driving the proliferation and centralisation of data. The result - storage infrastructure becoming ever more ubiquitous, ever more critical.


Monitoring Data Storage Infrastructure: Not To Be Done In Isolation

Given the ever increasing emphasis on performant, available storage arrays, the  importance of gathering, analysing, visualising and correlating storage array performance and capacity data has never been more important. However, anyone who has ever managed enterprise storage technologies knows that this is easier said than done. Whilst storage arrays typically gather quite detailed data about its operations, this data is typically captured and presented for relatively short time periods and is presented in isolation to other components in the data centre such as upstream servers or hypervisors. If you require capacity and performance data to be kept over a longer period of time to help in capacity planning, storage vendors are more than happy to sell you their enterprise storage management suites. In my experience these tend to allow the gathering and trending of performance and capacity data over a long period but are still array-centric (and expensive!).

Enter Splunk. Splunk has been used to monitor, analyse and correlate storage array metrics for a variety of storage platforms. If you look on Splunkbase, there are apps for a range of storage technologies including Symmetrix (e.g VMAX) as well Isilon (both these apps were written by K1's very own Luke BTW) and more recently, Splunk's very own Splunk App for NetApp Data ONTAP


Splunking Your Storage: 5 Key Advantages

Splunking your storage environment makes sense for the following reasons:

  1. Universal Indexing - Splunk is built from the ground up to ingest, analyse and present machine data. If you've ever had to analyse the output of storage array utilities like symcli, you know that while this output is rich in detail, it is quite arcane and in a format not particularly suited for traditional monitoring platforms. Splunk, however, eats this stuff for breakfast. Literally!
  2. Powerful Visualisation - Whether its graphing IOPs over a 24 hour period to measure changes in array activity, or showing a capacity breakdown, snap pool utilisation,  storage tiering activity, Splunk is able to turn streams of data in rich visualisations to give powerful actionable insights into the storage environment.
  3. Analytics - Traditional storage monitoring and management platforms can certainly retrieve data from a given array and store it centrally. However, these platforms are typically siloed in what capability is provided to work with the data once it has been retrieved. Splunk, on the other hand, is armed with an array of statistical and analysis functions that will keep even the most discerning data scientist or IT professional happy.
  4. Lookups - At the end of day a lot of the metrics gathered from storage arrays are capacity, performance and throughput data for a bunch of LUNs, volumes or disks. Interesting for the storage administrator no doubt but this data can be  given  broader appeal by augmenting it with application or business-level data. Lookups tables or real time lookups against data repositories like CMDBs immediately allow the answering of a whole bunch of far more interesting questions, like "what is my storage capacity allocation between physical and virtual environments", "which application landscapes or business functions are consuming the most storage resources", or what's my "utilisation split between non-production or production", or even "what application or business unit is responsible for the greatest amount of data growth". The augmentation of storage data delivers a business-centric view of data storage.
  5. Correlation - One of Splunk's most powerful capabilities is correlation. Splunk is able to deliver a consolidated view of storage activity all the way from application to OS to hypervisor to storage array in a single screen - an absolute must in today's hypervisor-driven infrastructure stacks.


Splunking NetApp And VMware: A Real World Use Case In Correlation

Splunk's latest offering in the storage space is the  Splunk App for NetApp Data ONTAP. K1 have worked with a number of their clients to deploy this app to instrument detailed visibility of their business critical NetApp storage arrays. On its own it delivers an impressive collection of performance and capacity statistics as well as centralised logging. However, K1 has really been able to make this  app sing when deploying it alongside Splunk's Splunk App for VMware. There is out-of-the-box correlation baked into the VMware app which automatically correlates VMware datastore views with NetApp volume statistics to give the much coveted ability to drill down from hypervisor-level storage into the array itself.

This the one of the great value propositions that Splunk can bring to IT Operations - the ability to tie disparate infrastructure components together to provide a unified view of the infrastructure landscape - the modern day IT holy grail. It is this correlation capability that consistently sets Splunk apart from other solution providers in the IT operations management space.