Hey friends,
Ever since I started working with ElasticSearch, CrateDB and Logstash, I have faced plenty of issues and exceptions that maybe had a clear reason and fix but were not documented well anywhere. I had to research through tons and tons of documentation and StackOverFlow pages to atleast get some hints as to what might be causing the issue and how to fix it.
It was then I decided that I will document such problems for easy access and quick solution for others. One such exception that I got multiple times was:
3. Now restart ElasticSearch/CrateDB/Logstash. The problem should be gone.
I hope that your problem is solved now, feel free to reach out to me if you face any similar issue. Adios till my next post!
Ever since I started working with ElasticSearch, CrateDB and Logstash, I have faced plenty of issues and exceptions that maybe had a clear reason and fix but were not documented well anywhere. I had to research through tons and tons of documentation and StackOverFlow pages to atleast get some hints as to what might be causing the issue and how to fix it.
It was then I decided that I will document such problems for easy access and quick solution for others. One such exception that I got multiple times was:
09:46:07.741 [main] ERROR
io.crate.shade.org.postgresql.util.PSQLException: ERROR: ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
at io.crate.shade.org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2458)
at io.crate.shade.org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2158)
at io.crate.shade.org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:291)
This was due to:
[WARN ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] high disk watermark [90%] exceeded on [Piz Roseg][/crate/ [WARN ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] high disk watermark [90%] exceeded on [Piz Roseg][/crate/ [WARN ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] rerouting shards: [high disk watermark exceeded on one or more nodes] [WARN ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] high disk watermark [90%] exceeded on [Piz Roseg][/crate/ [WARN ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] high disk watermark [90%] exceeded on [Piz Roseg][/crate/ [INFO ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] rerouting shards: [high disk watermark exceeded on one or more nodes] [WARN ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] high disk watermark [90%] exceeded on [Piz Roseg][/crate/ [WARN ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] high disk watermark [90%] exceeded on [Piz Roseg][/crate/ [INFO ][o.e.c.r.a.DiskThresholdMonitor] [Piz Roseg] rerouting shards: [high disk watermark exceeded on one or more nodes]
The Cause:
This particular exception is caused by a very simple problem - Disk Space. You are definitely running out of disk space on the disk where ElasticSearch/Logstash/CrateDB is installed.
The Solution:
The solution comes in three parts:
1. Clean up your disk. You need to have atleast 25-30% disk space free.
2. For every index you have, you need to run the following command individually:
PUT /your-index/_settings
{
"index.blocks.read_only_allow_delete": null
}
In case of CrateDB, you need to run the following SQL query for each table:
ALTER TABLE table-name SET ("blocks.read_only_allow_delete" = FALSE);
3. Now restart ElasticSearch/CrateDB/Logstash. The problem should be gone.
I hope that your problem is solved now, feel free to reach out to me if you face any similar issue. Adios till my next post!
Do you like this post? Please link back to this article by copying one of the codes below.
URL: HTML link code: Forum link code: