We are running Elasticsearch 8.2 on Ubuntu 20.04. Machine has 4GB of memory. Less than 1MB of data has been indexed as we are still in testing. However, Elasticsearch gets killed by the OS after a few hours.
Jun 24 01:53:27 industry-new kernel: [1443806.368450] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom
,task_memcg=/system.slice/elasticsearch.service,task=java,pid=1366207,uid=112
Jun 24 01:53:27 industry-new kernel: [1443806.368567] Out of memory: Killed process 1366207 (java) total-vm:4775580kB, anon-rss:2350732kB, f
ile-rss:0kB, shmem-rss:0kB, UID:112 pgtables:4976kB oom_score_adj:0
Jun 24 01:53:27 industry-new kernel: [1443806.495334] oom_reaper: reaped process 1366207 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0
kB
We tried limiting the heap size by creating a file called es.options in /etc/elasticsearch/jvm.options.d and setting:
--Xms2g
--Xmx2g
With that setting in place, we could not even restart elasticsearch – it timed out.
Any suggestions on how to address these kills please?
2
Answers
Was able to "solve" this by setting the 2 Xm values to 1g each. This is on a machine with 4GB of RAM, additionally running only a small python instance served by Gunicorn. The data indexed in Elasticsearch does not exceed 1MB, yet if left unchecked it will eventually be killed because of memory. Even with current settings, Elastic is constantly taking up just under 2GB of RAM. Why I cannot even begin to comprehend. As I noted in my question, each machine has 4GB of memory, so one needs to set your jvm allocation at no more than 1gb to prevent oom kills.
I think we were on Elasticsearch 4 or 5 prior to the upgrade and that ran fine on a 1GB machine. In 8+ Elasticsearch also deprecated the "type" variable when indexing, which makes tracking the origin of data within the index unnecessarily complex. Documentation on Elastic for both Python and Ubuntu can only be described as chaotic and incomplete. I doubt we will be recommending Elasticsearch for any customer going forward.
You need to put below lines in the
jvm.optionsnot thees.optionsfile, and this file needs to be in theconfigfolder of Elasticsearch.Note: Refer jvm options official document for more info.