Splunk wants a bigger piece of the Big Data pie
Big data integration firm Splunk has updated its software to make it faster and increase its reliability.
The firm, which specialises in collecting and monitoring large amounts of machine generated data, believes that reliability will cut users’ costs, by obviating the need to keep multiple copies of huge datasets, and encourage more people to give Splunk a bigger role in their Big Data projects.
Do ya feel lucky, Splunk?
“If you can turn it into data, you can put it into Splunk,” said D J Skillman, Splunk’s European technical director. “We are very flexible on the inject side, and solve the massive problem of data collection.”
“Resilience used to be built in by making copies and using a SAN [storage area network], but if you are collecting 5Tbyte a day in a SAN environment, the cost is astronomical,” he said. Building in resiliency in the software means the same performance and reliability can be done on commodity hardware, he said.
Splunk is not trying to take on relational databases – Skillman says they are still best for old-style transactional data like customer records, and it also leaves a lot of the “unstructured” data such as voice and video to other players, but concentrates on users who generate masses of text data from machine sources.
Splunk also now has a bi-directional connector to Hadoop, which Skillman says is, again, about allowing for use of the best tool for the job. “Hadoop is very inexpensive, and massively flexible for large dataset batch processing, but you have to program it.” Users need to have big data scientists to get the most out of Hadoop, he said.
Splunk is not open source, but operates on a “freemium” model whereby users can download and use the software for free, until the amount of data stored each day gets larger than 400MB. Above that, detailed pricing isn’t published, but is understood to start at $10,000 for a licence to store 1GB a day, falling rapidly as volumes rise.
Do you know about Europe’s role in tech history? Try our quiz!