Project

General

Profile

Actions
Copy link

Feature #1201

closed

Feature #2303: file-store enhancements (aka file-store v2): deduplication; hash-based naming; json metadata and cleanup tooling

file-store metadata in JSON format

Added by Peter Manev over 10 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Jason Ish
Target version:
4.1beta1
Effort:
Difficulty:
Label:

Description

Currently we write metadata for filestore like so:

root@LTS-64-1:~# cat  /var/log/suricata/files/file.2.meta 
TIME:              06/08/2014-14:15:08.392536
SRC IP:            31.186.225.23
DST IP:            10.0.2.15
PROTO:             6
SRC PORT:          80
DST PORT:          53064
HTTP URI:          /a/11016/26510/105352-2.js?&cb=0.15413070828462816&tk_st=1&rf=http://edition.cnn.com/&rp_s=c&tg_i.site=cnn_international&tg_i.rollup=homepage&tg_i.pagetype=main&p_pos=btf&p_screen_res=1680x945
HTTP HOST:         optimized-by.rubiconproject.com
HTTP REFERER:      http://ads.cnn.com/html.ng/site=cnn_international&cnn_intl_pagetype=main&cnn_intl_position=728x90_bot&cnn_intl_rollup=homepage&page.allowcompete=no&params.styles=fs&Params.User.UserID=53944fdb05ba670a3c6b805990008512&transactionID=14022297068343779055472671&tile=895079222045&domId=6c5b4c103152e6e3&kxid=ojke0w8tp&kxseg=
HTTP USER AGENT:   Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0
FILENAME:          /a/11016/26510/105352-2.js
MAGIC:             HTML document, ASCII text, with very long lines
STATE:             CLOSED
MD5:               2a5d49f36faaf44d1e115f01bee3f499
SIZE:              2175
root@LTS-64-1:~#

It would be beneficial if we can do JSON format logging as well for the meta files.

Actions
Copy link
 #1

Updated by Victor Julien over 10 years ago

  • Assignee set to Anonymous
  • Target version set to TBD
Actions
Copy link
 #2

Updated by Andreas Moe about 10 years ago

Was just about to write an issue about this problem. Seeing that the general direction (with the eve-format) is trending towards JSON this would be a good thing to move to the JSON format aswell.

Actions
Copy link
 #3

Updated by Andreas Moe about 10 years ago

Was looking through the source of suricata to see if this ticket could be solved quickly. Then i say the function "LogFileWriteJsonRecord" in log-file.c. Isnt this an implementation for this feature ticket?

Actions
Copy link
 #4

Updated by Victor Julien about 10 years ago

LogFileWriteJsonRecord() isn't being used for creating/writing an 'meta' file. It's used to generate files-json.log lines. It could be used to generate the meta file with relatively little effort I think.

Actions
Copy link
 #5

Updated by Andreas Moe about 10 years ago

I rewrote the log-filestore.c file to give a json formated output (not using the jansson json object but just changing the fprintf contents).

Actions
Copy link
 #6

Updated by Andreas Moe about 10 years ago

https://github.com/inliniac/suricata/pull/1161

This is with the current regular format and with option for JSON format of the data.

Actions
Copy link
 #7

Updated by Victor Julien almost 10 years ago

  • Status changed from New to Assigned
  • Assignee changed from Anonymous to Andreas Moe
  • Target version changed from TBD to 3.0RC2
Actions
Copy link
 #8

Updated by Victor Julien almost 9 years ago

  • Target version changed from 3.0RC2 to 70
Actions
Copy link
 #9

Updated by Victor Julien over 8 years ago

  • Target version changed from 70 to TBD
Actions
Copy link
 #10

Updated by Victor Julien almost 7 years ago

  • Assignee changed from Andreas Moe to Jason Ish
  • Target version changed from TBD to 70

The meta record should probably simply be the 'fileinfo' record written to a .json file?

Actions
Copy link
 #11

Updated by Jason Ish almost 7 years ago

  • Parent task set to #2303
Actions
Copy link
 #12

Updated by Victor Julien almost 7 years ago

  • Status changed from Assigned to Closed
  • Target version changed from 70 to 4.1beta1
Actions
Copy link

Also available in: Atom PDF