org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of [org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]

I noticed that replica shards were sometimes becoming unallocated under high write load. Manually re-allocating them usually worked for a little bit, but after a little while they would become unallocated again. Looking at the logs, I found errors like these:

[2013-04-28 19:52:08,944][WARN ][action.index             ] [hostname] Failed to perform index on replica [logstash-2013.04.28][5]
org.elasticsearch.transport.RemoteTransportException: [otherhost][inet[/10.0.0.1:9300]][index/replica]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of [org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]
        at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:35)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        ...

(The stack trace is truncated in the output above; see this gist for the full output.)

The fix turned out to be relatively simple (set threadpool.index.queue_size to -1), but the explanation of the fix is a bit more involved.

Thread Pools

elasticsearch operations such as index, search, bulk, and merge, are all done in threads. Each type of operation has its own dedicated thread pool which can be configured independently. Prior to elasticsearch 0.90.0.RC2, thread pools defaulted to type cache (this changed in 0.90.0.RC2):

The cache thread pool is an unbounded thread pool that will spawn a thread if there are pending requests.

Note the word unbounded. From prior experience, I knew this was problematic — if a bunch of requests come in at once, elasticsearch spawns hundreds of threads to service the requests. It spends more time spawning threads than actually doing work, and effectively locks up the machine. In extreme cases the node even drops out of the cluster because it does not respond to pings from other nodes in time.

To prevent this behavior, when I first configured the cluster I followed the advice of this blog post (and the instructor at the elasticsearch training I attended) and set the type of each thread pool to fixed. From the documentation:

The fixed thread pool holds a fixed size of threads to handle the requests with a queue (optionally bounded) for pending requests that have no threads to service them.

I also set limits on the size and queue_size of each thread pool. All was working well until write load started increasing and I started running into the exceptions described above.

Bounded Queue

Although I was correct to set the thread pool type to fixed, I should have paid more attention to the queue_size parameter. Again, from the documentation:

The queue_size allows to control the size of the queue of pending requests that have no threads to execute them. By default, it is set to -1 which means its unbounded. When a request comes in and the queue is full, the reject_policy parameter can control how it will behave. The default, abort, will simply fail the request. Setting it to caller will cause the request to execute on an IO thread allowing to throttle the execution on the networking layer.

With this behavior in mind, the EsRejectedExecutionException made perfect sense. Under high write load, not only was the thread pool exhausted, the queue was too. When this happens, the abort reject_policy kicks in and rejects the write to the replica shard. elasticsearch then marks the replica as failed and it becomes unallocated.

Capping the size of the thread pool keeps the machine from locking up, but capping the size of the overflow queue causes writes to replica shards to be rejected if the queue grows too long. The solution is to keep the thread pool at a fixed size, but to set the queue size to unbounded, i.e., threadpool.index.queue_size: -1. Indeed, according to the documentation, newer versions of elasticsearch set the thread pool size to 5 times the number of cores on the machine and the queue_size to -1.

Tagged

5 thoughts on “org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of [org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]

  1. Thanks for this posting – I was running into the same problem, and knew it was related to my thread pool settings, but it was comforting to read a blog posting that confirmed my suspicions.

  2. Great post, I’ve been seeing similar issues.

    > Indeed, according to the documentation, newer versions of elasticsearch set the thread pool size to 5 times the number of cores on the machine and the queue_size to -1.

    I’m not sure this is true. According to http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html , fixed queues have their size default to -1, but the index queue specifically defaults to 200 (written at the top of the page).

    Have I misunderstood?

  3. Great post. Thanks for the clarification, However I hav been running into a problem which has some bizarre side effects (or perhaps not).
    My setup is : Run a Spark process which (is not using the ES hadoop jar for access to ES) searches through the complete results set of a query and for each result doc performs another search. Since his is a spark process and the task gets splitup/distributed, there are multiple search requests to the ES instance. (my ES instance is just a single server with no sharing (read 1 shard) and no replication (read 0 replicas).

    If I run the Spark process with 50 cores assigned, meaning there are atleast 50 concurrent requests for search, I ended up getting NoNodeAvailable exceptions. and this was from random nodes in the Spark Cluster. (and in fact, the ES instance did not have any lag/was pingable from those Spark nodes)

    The Elasticsearch google group suggests its either a thread pool issue or a a connection time out issue. I am unable to determine which is right. with default configurations, I am wondering what the limits are on search thread pool/queue_size) (on a ES version 1.0.2 instance)

    Any thoughts ?

    Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *