Suricata

Eoin Miller wrote:

Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.

Wrt the ticket, some work is being done here https://github.com/inliniac/suricata/pull/922

Victor Julien wrote:

Eoin Miller wrote:

Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.

You do need u2+barnyard because there isn't a way to just dump packet payload info out into the JSON (unless using patches below it would appear).

Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html

Eoin Miller wrote:

Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:

No, I use logstash-forwarder to ship the logs from my sensor to my ELK server. Easy to set up, encrypted comms and lightweight. Check https://github.com/elasticsearch/logstash-forwarder

Is this feature available as part of some beta build/package or I should build it myself to test it?

That is available in 2.1beta2 (and 2.1beta1) - JSON Output Enhancemen Include Payload(s)

Victor, would you mind sharing your logstash and forwarder configuration you're using with suricata?
I've got problem with json being escaped and not properly parsed when I'm trying to use such setup with suricata.

There is one in our wiki that you can find here:
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

hope it helps

Thank you, but I've already looked into wiki link - it refers to case where eve.json is read by logstash directly. My use-case involve logstash-forwarder and lumberjack protocol which is different.