so-elastalert Missing from Distributed Installation

[squirtle-turtle]

Version

2.4.110

Installation Method

Cloud image (Amazon, Azure, Google)

Description

upgrading

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

65.8 GB

Storage for /

270.5 GB

Storage for /nsm

2146.4 GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

We have a distributed environment, including 2 search nodes and 7 forward nodes. There is a mix of on-premise and cloud forward nodes; search nodes and our manager node are hosted in AWS.

On our manager node, so-elastalert is missing, and the Docker container cannot be found. Manual reinstallation of the Docker container (docker pull) fails.

The issue first arose on a prior manager node when we spun up and accepted the second search node. We redeployed the manager and the search nodes. It worked for a little while - a couple of hours. The issue repeated itself on the second manager node after we tried to do a scorched-earth rebuild. We reviewed other posts here (anything similar) and didn't have any luck.

The errors we get on the Elastic side are tied to unallocated shards. How can we rectify this?

Errors when running "sudo salt-call state.highstate":

[ERROR   ] Command 'so-elasticsearch-wait' failed with return code: 1
[ERROR   ] stdout: Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)
Server is not ready

----------
          ID: so-kibana-dashboard-load
    Function: cmd.run
        Name: /usr/sbin/so-kibana-config-load -i /opt/so/conf/kibana/saved_objects.ndjson.template
      Result: True
     Comment: Command "/usr/sbin/so-kibana-config-load -i /opt/so/conf/kibana/saved_objects.ndjson.template" run
     Started: 05:48:41.377991
    Duration: 15.806 ms
     Changes:   
              ----------
              pid:
                  3780687
              retcode:
                  0
              stderr:
              stdout:

----------
          ID: elastalert_sbin
    Function: file.recurse
        Name: /usr/sbin
      Result: True
     Comment: Recursively updated /usr/sbin
     Started: 05:48:41.428598
    Duration: 208.07 ms
     Changes:   
              ----------
              /usr/sbin:
                  ----------
                  /usr/sbin:
                      ----------
                      user:
                          933
                  user:
                      933

----------
          ID: wait_for_elasticsearch
    Function: cmd.run
        Name: so-elasticsearch-wait
      Result: False
     Comment: Command "so-elasticsearch-wait" run
     Started: 05:48:43.129805
    Duration: 324077.949 ms
     Changes:   
              ----------
              pid:
                  3780940
              retcode:
                  1
              stderr:
              stdout:

----------
          ID: wait_for_elasticsearch
    Function: cmd.run
        Name: so-elasticsearch-wait
      Result: False
     Comment: Command "so-elasticsearch-wait" run
     Started: 05:48:43.129805
    Duration: 324077.949 ms
     Changes:   
              ----------
              pid:
                  3780940
              retcode:
                  1
              stderr:
              stdout:
                  Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)
                  Server is not ready

Clues in logs:

[root@angband onion]# so-elasticsearch-query _cluster/allocation/explain?pretty
{
  "note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
  "index" : ".internal.alerts-ml.anomaly-detection-health.alerts-default-000001",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "REPLICA_ADDED",
    "at" : "2024-10-28T16:01:16.215Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "throttled",
  "allocate_explanation" : "Elasticsearch is currently busy with other activities. It expects to be able to allocate this shard when those activities finish. Please wait.",
  "node_allocation_decisions" : [
    {
      "node_id" : "IVseaez8RuSyFBehC9qlVw",
      "node_name" : "angband-search-1",
      "transport_address" : "10.x.y.z:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0"
      },
      "roles" : [
        "data",
        "data_hot",
        "ingest",
        "transform"
      ],
      "node_decision" : "throttled",
      "store" : {
        "matching_size_in_bytes" : 0
      },
      "deciders" : [
        {
          "decider" : "throttling",
          "decision" : "THROTTLE",
          "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
        }
      ]
    },
    {
      "node_id" : "TI0wfNmfS_ORWazF7HQgtg",
      "node_name" : "angband-search-2",
      "transport_address" : "10.b.c.d:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "transform.config_version" : "10.0.0",
        "xpack.installed" : "true"
      },
      "roles" : [
        "data",
        "data_hot",
        "ingest",
        "transform"
      ],
      "node_decision" : "throttled",
      "deciders" : [
        {
          "decider" : "throttling",
          "decision" : "THROTTLE",
          "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
        }
      ]
    },
    {
      "node_id" : "1h8ZDdvpQ2qfUlIPIvu9gQ",
      "node_name" : "angband",
      "transport_address" : "10.q.r.s:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0"
      },
      "roles" : [
        "data",
        "master",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "no",
      "store" : {
        "matching_size_in_bytes" : 249
      },
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "a copy of this shard is already allocated to this node [[.internal.alerts-ml.anomaly-detection-health.alerts-default-000001][0], node[1h8ZDdvpQ2qfUlIPIvu9gQ], [P], s[STARTED], a[id=oHSk5wjKR52y47HK6-e-8A], failed_attempts[0]]"
        }
      ]
    }
  ]
} 

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[rh-ops]

Can you provide the output of the following commands?

so-elasticsearch-query _cat/shards | grep -i una

salt \* cmd.run 'df -h /nsm'

For the salt command we're more focused on the results from the manager and the search nodes.

[squirtle-turtle]

Thank you for the help!

Command results:

Unassigned shards command:

[root@angband onion]# so-elasticsearch-query _cat/shards | grep -i una
.internal.alerts-ml.anomaly-detection-health.alerts-default-000001 0 r UNASSIGNED                                          
.ds-ilm-history-7-2024.10.18-000001                                0 r UNASSIGNED                                          
.ds-ilm-history-7-2024.11.01-000003                                0 r UNASSIGNED    

Salt command results from manager and search nodes:

[root@angband onion]# salt \* cmd.run 'df -h /nsm'
angband-search-1_searchnode:
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/nvme1n1     12T  153G   12T   2% /nsm
angband-search-2_searchnode:
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/nvme1n1     12T   95G   12T   1% /nsm
angband_manager:
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/nvme0n1    2.0T   87G  1.9T   5% /nsm

[rh-ops]

Alright so you have 3 replica shards that haven't been moved. What is the output of the following command?

so-elasticsearch-query _cluster/health?pretty

[squirtle-turtle]

Here's the output:

[onion@angband ~]$ sudo so-elasticsearch-query _cluster/health?pretty

{
  "cluster_name" : "securityonion",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 88,
  "active_shards" : 137,
  "relocating_shards" : 0,
  "initializing_shards" : 4,
  "unassigned_shards" : 4,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 94.48275862068965
}

[rh-ops]

Okay so I have 2 commands to run, and then report back the number of replicas
so-elasticsearch-query ilm-history-7/_settings | jq | less -S
so-elasticsearch-query .alerts-ml.anomaly-detection-health.alerts-default/_settings | jq | less -S

Looks like the shards you have unassigned are just replica shards that don't even have any data in them. So if you want to just go ahead and delete those, you can use this command
so-elasticsearch-query {index} -XDELETE

[rh-ops]

Okay if you haven't done anything yet, just to verify if those shards have no data, log into Kibana, navigate to the dev tools page and run the following command:

GET _cat/shards?v

[squirtle-turtle]

The replica shards are indeed empty; however, the primary shards have data. If I remove the shards by specifying the index, will I also remove the primary shard and its data? Is this okay to do?

Outputs from the specified commands are below:

[onion@angband ~]$ sudo so-elasticsearch-query ilm-history-7/_settings | jq | less -S
{
  ".ds-ilm-history-7-2024.10.25-000002": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".ds-ilm-history-7-2024.10.25-000002",
        "merge": {
          "policy": {
            "floor_segment": "100mb",
            "merge_factor": "16"
          }
        },
        "creation_date": "1729892666645",
        "number_of_replicas": "1",
        "uuid": "NTkbFOxSQwS7OfgvIHtbnQ",
        "version": {
          "created": "8505000"
        }
      }
    }
  },
  ".ds-ilm-history-7-2024.10.18-000001": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".ds-ilm-history-7-2024.10.18-000001",
        "merge": {
          "policy": {
            "floor_segment": "100mb",
            "merge_factor": "16"
          }
        },
        "creation_date": "1729287676720",
        "number_of_replicas": "1",
        "uuid": "yV2p_JI-QhaW8El4fD5jIg",
        "version": {
          "created": "8505000"
        }
      }
    }
  },
  ".ds-ilm-history-7-2024.11.01-000003": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".ds-ilm-history-7-2024.11.01-000003",
        "creation_date": "1730497575978",
        "number_of_replicas": "1",
        "uuid": "Hu2TxejGRya9YDplWoD7WQ",
        "version": {
          "created": "8505000"
        }
      }
    }
  }
}

[onion@angband ~]$ sudo so-elasticsearch-query .alerts-ml.anomaly-detection-health.alerts-default/_settings | jq | less -S
{
  ".internal.alerts-ml.anomaly-detection-health.alerts-default-000001": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": ".alerts-ilm-policy",
          "rollover_alias": ".alerts-ml.anomaly-detection-health.alerts-default"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "mapping": {
          "total_fields": {
            "limit": "2500"
          },
          "ignore_malformed": "true"
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".internal.alerts-ml.anomaly-detection-health.alerts-default-000001",
        "creation_date": "1729287670770",
        "number_of_replicas": "1",
        "uuid": "sxa7vBpaQQ6Mr3SSqABkug",
        "version": {
          "created": "8505000"
        }
      }
    }
  }
}

Angry shard values from the GET command:

UNASSIGNED                                          
elastalert_silence                                                 0     p      

UNASSIGNED                                          
.ds-logs-zeek-so-2024.10.20-000001        

INITIALIZING                          10.x.x.x angband-search-2
so-detectionhistory                                                0     p      

UNASSIGNED                                          
.internal.alerts-observability.uptime.alerts-default-000001        0     p      

INITIALIZING                          10.x.x.x angband-search-1
.apm-agent-configuration                                           0     p      

UNASSIGNED                                          
.ds-logs-elastic_agent.metricbeat-default-2024.10.20-000001        0     p      

INITIALIZING                          10.x.x.x angband-search-2
.ds-logs-system.auth-default-2024.10.20-000001                     0     p      

INITIALIZING                          10.x.x.x angband-search-1
logs-ti_recordedfuture_latest.threat-2                             0     p      

[rh-ops]

Use the following command to get rid of the replica shards and ensure more won't be produced.

so-index-list | grep yellow | awk '{print $3}' | xargs -I % so-elasticsearch-query %/_settings -d '{"index": {"number_of_replicas": 0 }}' -XPUT

[squirtle-turtle]

Hi there,

Thank you for your help. I've removed all the replica shards and ran the command recommended. Unfortunately, the health of the cluster is still yellow.

[onion@angband ~]$ sudo so-elasticsearch-query _cluster/health?pretty
{
  "cluster_name" : "securityonion",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 61,
  "active_shards" : 83,
  "relocating_shards" : 0,
  "initializing_shards" : 1,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 98.80952380952381
}

Kibana now has a "404 page not found" error when I try to access it. Open to ideas - I'll post more information in a moment.

[rh-ops]

Hmm try restarting the Kibana container with so-kibana-restart. Also lets look at whats causing the left over shards to remain unassigned with sudo so-elasticsearch-query _cluster/allocation/explain?pretty.

[squirtle-turtle]

We wound up blowing it away and keeping one search node instead of two; adding the second search node was where things kept breaking, and it was taking us too long to dig in. I'm sorry for the delayed reply, that's where we landed!

Thank you for all of your help, I appreciate the assist. So far, health is good after the rebuild.