Apache Spot: open source big data analytics for cyber

An open source program that analyzes IT network flows and packet data at cloud scale to let defenders identify anomalous behavior has been donated to the Apache Software Foundation.

The program, called Open Network Insight when Intel first developed it earlier this year, has been accepted into the ASF incubator and is being rebranded Apache Spot, Intel and co-sponsor Cloudera announced at the Strata+Hadoop World show in New York this week.

According to its website, the program “uses machine learning as a filter for separating bad traffic from benign” by first building “a model of the machines on the network and their communication patterns,” and then using that to isolate potentially suspicious network flows.

“Context enrichment, noise filtering, whitelisting and heuristics is also applied to network data to produce a shortlist of most likely security threats,” states the website.

“Running on Apache Hadoop, Apache Spot can analyze billions of events in order to detect unknown threats, insider threats, and gain a new level of visibility into the network,” concludes the site.

Spot also provides open data models for network, endpoint, and user — “a standard format of enriched event data that makes it easier to integrate cross application data to gain complete enterprise visibility,” the program’s GitHub page says.

“Spot’s Open Data Models helps organizations quickly share new analytics with one another as new threats are discovered.”

“The idea is, let’s create a common data model that any application developer can take advantage of to bring new analytic capabilities to bear on cybersecurity problems,” Cloudera co-founder Mike Olson told the audience at the New York show Wednesday, according to IDG News Service.

“This is a big deal, and could have a huge impact around the world,” he said.