"Once the results have been put into the database of what was found and where, human analysts can then fire up a Tor browser and check out what the crawler found. The more keywords you have, the more results you'll get, and the more people you have to rifle through it all, the better the chances of finding the needles in that haystack."
Of course, indexing the Dark Web is not static. As McAleavey notes, sites on the Tor network, "often change their address every few hours or every few days, so you need to crawl again looking for those sites of interest because they probably moved since the last time you crawled."
Michaud agreed, but said it is possible to keep up with address changes. While he wouldn't discuss the techniques his company uses to do it, "we do it really well," he said.
Whether it is worth the time and expense to conduct Dark Web surveillance is also a matter of debate. Gong contends that while it is helpful as a "layer" of security, it is not easy to do well. "It requires both sophisticated infrastructure and technical skills that are not trivial to establish," he said, adding that, "it is not very crucial or affordable for an enterprise IT to pull off by itself."
And he believes there is, "nothing that can replace direct monitoring of your own networks and assets."
But Michaud said as it becomes easier and cheaper, it will be a necessary part of a security operation. "Enterprises are scared," he said, "because they know they will be held responsible for data breaches if they aren't proactive.
"If you're just being defensive, you're going to have a bad day."
Sign up for MIS Asia eNewsletters.