Security Onion won't delete old Suricata Alerts automatically?

[GeorgePatches]

I had an issue today with our SO 2.4.110 at work today. Elasticsearch on the ManagerSearch node kept faulting. Dug into the logs a little and it seemed like a disk space problem. This seemed odd to me because until now it seemed like SO did a good job of taking care of itself and deleting the oldest whatever to make space. The logs seemed to indicate it had deleted everything it could and was still at the high watermark. So I started poking at the elastic indices a little and discovered that basically everything had been deleted, except my Suricata Alerts indices. All my firewall logs, zeek, strelka, etc had be cleared in an attempt to make space for more suricata alerts. Once I finally identified the issue I was able to go into the Kibana dev console and delete old suricata alerts to free up disk space. Everything is functioning normally again.

Is this intended behavior to never delete suricata alerts automatically? The behavior I was expecting was deleting a rolling window from all the indices, or deleting all the oldest indices before deleting anything newer. Maybe I'm overlooking something, but it seems weird to me.

[GeorgePatches]

OK, it's been about a week and logged back in to see what's going on.

Security Onion is still deleting all my logs because it's exceeding the LOG_SIZE_LIMIT of 163GB. Unless someone tells me this is intended, I'm going to open an issue because this seems pretty broken to me.

[dougburks]

Is this intended behavior to never delete suricata alerts automatically?

Most folks don't want their alerts to be deleted automatically by default.

Security Onion is still deleting all my logs because it's exceeding the LOG_SIZE_LIMIT of 163GB.

Seems like your storage might be a little on the low side. You might want to revisit the hardware requirements page (https://docs.securityonion.net/en/2.4/hardware.html) and consider increasing storage. Disk is cheap!

If you can't increase storage and you don't care about your alerts, then you could modify the default index management:
https://docs.securityonion.net/en/2.4/elasticsearch.html#index-management

[GeorgePatches]

Most folks don't want their alerts to be deleted automatically by default.

That was basically what I was thinking was going on...however as the defaults are currently set a machine will basically fill all space until elasticsearch no longer functions.

If you can't increase storage and you don't care about your alerts, then you could modify the default index management:
https://docs.securityonion.net/en/2.4/elasticsearch.html#index-management

Looking through this, I seem to be having issues what the so-elasticsearch-indices-delete. It doesn't explicitly say that suricata is exempted in the documentation, but if I check the code I see that it is. ILM seems to be the only mechanism to prune suricata alerts.

[dougburks]

however as the defaults are currently set a machine will basically fill all space until elasticsearch no longer functions.

This is not a commonly reported issue so perhaps others are doing one or more of the following:

• deploying with more storage
• tuning their alerts so they receive fewer alerts in the first place
• tuning the default index management
• using a distributed deployment so that Elasticsearch is not competing with full packet capture for storage

ILM seems to be the only mechanism to prune suricata alerts.

Yes, from https://docs.securityonion.net/en/2.4/elasticsearch.html#index-management:
Starting in Security Onion 2.4.70, you have the option of disabling so-elasticsearch-indices-delete and just managing indices using ILM. This may be useful for production deployments since ILM provides more granular index management.