Another challenge that will only heighten as Hadoop heads for the mainstream is finding people to work with the technology. "There's a lack of skills, and that's definitely a challenge in terms of the continued adoption of Hadoop," Aslett says.
Major players including Cloudera, IBM, Hortonworks and MapR are all investing heavily in training programs to teach IT pros how to deploy, configure and manage Hadoop products. "They're well aware that this is actually an issue that could limit the continued adoption of Hadoop at an enterprise level."
"If you go out there and try to hire, it's incredibly difficult," acknowledges Omer Trajman, vice president of customer solutions at Cloudera. A more feasible approach is to look internally for candidates ripe to learn Hadoop, he suggests.
"The most successful companies aren't necessarily going out and trying to hire aggressively. They have people who have the basic skills required, individuals who have backgrounds in statistics, science, data processing, Java development and analytics," Trajman says. "It's really about looking inward into an organization, finding people who already have familiarity with the business and domain expertise, and teaching them how to use these tools."
On the positive side, as awareness of Hadoop grows, the number of IT pros learning Hadoop is growing, too.
"Every time I've talked to a recruiter for the last two years, I've asked if they have anybody with Hadoop experience. Usually the answer was 'ha-what?' Increasingly it's maturing, so you are seeing more people in the field," says Concurrent's Lazzaro.
Figuring out what kind of person is best to hire can be a challenge in itself.
"We originally thought we needed to find a hardcore Java developer," Return Path's Sautins says. But in reality, the talent that's best suited for working with Hadoop isn't necessarily a Java engineer. "It's somebody who can understand what's going on in the cluster, is interested in picking up some of these tools and figuring out how they work together, and can deal with the fact that pretty much everything in the Hadoop ecosystem is not even a 1.0 release yet," Sautins says. "That's a real skill set."
Sign up for MIS Asia eNewsletters.