Suricata

It seems datasets have some limit in terms of writing on exit.
Something I've tested in a number of ways and for a while with a couple of diff Suricata versions, mainly:

This is Suricata version 7.0.0-dev (60ad658f3 2021-01-21)

cherry picked from PR https://github.com/OISF/suricata/pull/5754

getting

wc -l /var/log/suricata/*.intel
  187671 /var/log/suricata/dnsseen.intel
  187671 /var/log/suricata/httphosts.intel
  146258 /var/log/suricata/httpuseragents.intel
     395 /var/log/suricata/sshsoftware.intel
    5196 /var/log/suricata/tlscert_issuer.intel
  184183 /var/log/suricata/tlscert_subject.intel
   39648 /var/log/suricata/tlsja3.intel
    4039 /var/log/suricata/tlsja3s.intel
  187671 /var/log/suricata/tlsserial.intel
  187671 /var/log/suricata/tlssni.intel
 1130403 total

and git master

This is Suricata version 7.0.0-dev (8ac363c34 2021-01-23)

getting

wc -l /var/log/suricata/*.intel
  1000 /var/log/suricata/dnsseen.intel
  1000 /var/log/suricata/httphosts.intel
  1000 /var/log/suricata/httpuseragents.intel
   354 /var/log/suricata/sshsoftware.intel
  1000 /var/log/suricata/tlscert_issuer.intel
  1000 /var/log/suricata/tlscert_subject.intel
  1000 /var/log/suricata/tlsja3.intel
  1000 /var/log/suricata/tlsja3s.intel
  1000 /var/log/suricata/tlsserial.intel
  1000 /var/log/suricata/tlssni.intel

Few rules examples I've just tried while testing with different separate runs with the rules below respectively:

alert dns any any -> any any (msg:"Datasets domains -1 "; dns.query; dataset:isnotset,dns-seen,type string,state  /var/log/suricata/dnsseen.intel; dataset:set,dns-seen,type string,state /var/log/suricata/dnsseen.intel,memcap 900mb,hashsize 6000000; sid:111; rev:1;)

alert dns any any -> any any (msg:"Datasets domains -2 "; dns.query; dataset:isnotset,dns-seen,type string,state /var/log/suricata/dnsseen.intel,memcap 900mb,hashsize 6000000; dataset:set,dns-seen,type string,state /var/log/suricata/dnsseen.intel,memcap 900mb,hashsize 6000000; sid:222; rev:1;)

alert dns any any -> any any (msg:"Datasets domains -2 "; dns.query; dataset:isnotset,dns-seen,type string,state /var/log/suricata/dnsseen.intel,memcap 150mb,hashsize 1000000; dataset:set,dns-seen,type string,state /var/log/suricata/dnsseen.intel,memcap 150mb,hashsize 1000000; sid:333; rev:1;)

Seems some limit of 187671 in one case and 1000 in the other.

the machine has plenty of RAM and disk available.