Question 202182 in Debian Question posted in
The oficial documentation can be found here.

Debian – How can I increase the maximum number of threads that Java can spawn?

I am writing a TCP server application, and I was bench testing how many concurrent TCP sockets I can manage.
After acceptance the socket is passed to a client class which spawns a separate listener thread to do work.

I was able to get 9462 active connections before the JVM failed to spawn more threads and therefore dropping any new connections.

[1805.442s][warning][os,thread] Failed to start thread "Unknown thread" – pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[1805.442s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Thread-15915"
Exception in thread "Thread-0" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:802)
at socketservertest.TCPServer.run(TCPServer.java:139)
at java.base/java.lang.Thread.run(Thread.java:833)

The system still have 40% available RAM, I suspect this is a limit on the JVM settings or perhaps a restriction on the debian OS itself?
If so, how can I configure the system to allow more memory?

Regards Dominick

What I have tried:
Look in the java documentation and other forums for solutions, nothing helpful found.

Desired outcome:
Be able to handle as many threads as possible (clients), starting goal maybe 50K connections?

Tags:

2

Answers


  1. 0

    The maximum number of threads you can create will be system dependent. Please check your system’s capacity:

    Windows: no limit

    Linux: cat /proc/sys/kernel/threads-max

    The stack-trace also suggests the issue may be related to memory. Every thread needs its own stack, and each thread will have data on the heap while executing. The JVM has default memory limits. Look into setting the stack and heap sizes.

    Set stack size usage (-Xss)

    Usage: java -Xss1024k -jar app.jar

    In the error, you see pthread_create failed. Your jvm is using pthread to create native OS threads. The total stack size will be defined by your host system. It may be configurable through the OS. This property (-Xss) sets the stack size per thread. To avoid running out, try to make the size as small as possible.

    Set heap size (-Xmx)

    Usage: java -Xmx1024m -jar app.jar

    Assuming you’re holding variables in these threads, you’ll also want to keep an eye on the heap. Each thread will be running some procedure, allocating memory along the way. That memory will be allocated on the heap. 50,000 threads is a lot concurrent threads. You may be using a lot of memory. Use this property (-Xmx) to increase the heap size.

    Login or Signup to reply.
  2. 0

    On Linux (wsl2) I can get 30,000 threads with my default settings and reduced stack size. (-Xss150k) with this program:

    import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;
import java.util.concurrent.CountDownLatch;

public class App 
{
    public static void main( String[] args ) throws InterruptedException {
        CountDownLatch latch = new CountDownLatch(1);
        int numberOfThreads = 33_000; // OK at 30_000
        Thread[] threads = new Thread[numberOfThreads];
        for (int i = 0; i < numberOfThreads; ++i) {
            threads[i] = new Thread(() -> {
                try {
                    latch.await();
                } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                }
            });
            threads[i].start();
        }
        ThreadMXBean bean = ManagementFactory.getThreadMXBean();
        System.out.println(bean.getThreadCount());
        latch.countDown();
        for (int i = 0; i < numberOfThreads; ++i) {
            threads[i].join();
        }
    }
}

    The error that I see at 33,000 threads is:

    [7.274s][warning][os,thread] Attempt to protect stack guard pages failed (0x00007f6bb46a2000-0x00007f6bb46a6000).
#
# A fatal error has been detected by the Java Runtime Environment:
# Native memory allocation (mprotect) failed to protect 16384 bytes for memory to guard stack pages
# An error report file with more information is saved as:
# /home/tgdavies/dev/so76628100/hs_err_pid395685.log
...
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6bb467c000, 16384, 0) failed; error='Not enough space' (errno=12)

    Which I interpreted as running out of swap. I had 4GB of swap configured. I configured 32GB of swap, but that didn’t help.

    The error report file has some useful tips, including:

    #   JVM is running with Zero Based Compressed Oops mode in which the Java heap is
#     placed in the first 32GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 32GB virtual address.

    This blog post looks relevant: https://poonamparhar.github.io/out_of_memory_on_64_bit_platform/ but using uncompressed oops didn’t help.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search