Unable to login to kill runaway process

Unable to login to kill runaway process


Sorry I've been posting so many messages. We're working all the kinks out of the system; that's for sure.

I inadvertently pasted a command and started a Java program spawning too many threads. The program spat out a message about being unable to create enough threads, and I tried to kill it with CTRL-C. Then it said something to the effect of "kill signal received out of memory response; may have to forcefully terminate the VM". Unable to do anything, I logged out and tried to log back in.

I now receive the following message whenever I attempt to log in.
Connection to closed by remote host.
Connection to closed.

I'm assuming you only allow one login per user, and that's why I can't get back in. I tried logging in from another account (the account that this transpired on was a student account) to kill the runaway Java VM, but I can't kill it using the other (instructor) login. The runaway process' pid is 1561, and it is running on acano01.

EDIT: since then the process appears to have disappeared and I can log in again. I'll be more careful in future, but is there any mechanism whereby I can expedite the process?

Thanks again,

7 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Just out of curiosity how may threads did you attempt to spawn?

We have a peruser imposed limit, to ensure that runaway programs don'tbring downthe system (acano01) - Java may react in a different way when it can't spawn a thread - I'm not knowledgeable on Java.

There is no limit to the number of logins per user - but if the thread/process limit is still valid, it may cause indeterminate results like you saw.

In the future, I would suggest that you login withyour other account and "su" to the account that has the problem and try tokill the problematic process.

I had attempted to start 128 threads (2 per logical core). It stopped successfully spawning at 100, I believe. I'm guessing that's the thread limit. I don't know if the limit is set in stone, but it seems like a good idea to let users run 2 threads per logical core plus a few for shells etc.

Thanks for the tip. Hopefully I won't make that mistake again but, if I do, then I'll try the fix you suggest.

I'm not sure if Java has an inherentthread limit.

We have imposed a hard limit of150 processes/threads (per user) - along the samelines of # oflogical cores*2+ some for shells, etc.

I wound up experiencing the same issue again; it seems rather arbitrary in when it strikes. I successfully ran this code not 5 minutes before.

I thought it might have something to do with Java's heap size, since the error I get is:

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at Test.(Test.java:88)
at TestSL.(TestSL.java:9)
at MultiTest.(MultiTest.java:24)
at MultiTest.main(MultiTest.java:48)

At this point, I'd expect to see it terminate, but it keeps running. If I press CTRL-C, I then receive:

Java HotSpot 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGINT to handler- the VM may need to be forcibly terminated

I'm not sure what to make of that. I'll try expanding the heap size (and shrinking the thread stack size) in future (to avoid running out of memory). I think it might be a power punch combo: simultaneously running out of memory, exceeding some thread limitation, etc.

I thought it might also be worth mentioning that the "su" fix doesn't seem to work. When I attempt to "su" to the username that caused the failure, I receive the message:

su: cannot set user id: Resource temporarily unavailable

I suspect you're hitting the Max Process/thread userlimit - particularly based on the su response. Again I don't know the Java VM, but I suspect that Java creates a significant number of processes/threads as part of the VM, that then exceeds the max (per user) thread count when you add your 128 Java threads.

I can temporally raise your max thread/process count, and you can see if you experience the same issues.

Thank you very much for the offer. I think I might have found a solution to the problem, though; at any rate, I haven't hit the problem since my last post. My new theory is that Java was simply running out of memory trying to allocate space for the threads' stacks. I tried increasing the initial heap size and reducing the space allocated to thread stacks, and it seems to have alleviated the problem.

To limit the thread stack size to 512KB (down from the default 2MB), I used:

ulimit -s 512
java -Xss512K

I also increased the heap size (in my case, to 8GB):

java -Xss512K -Xms8192M

I'll post an update if I experience the problem again; Both theories are incomplete, but I'm hoping that this is the solution. (I'm not sure why Java would spawn too many threads during some executions; it's easier for me to see how memory requirements could differ across executions.)

Leave a Comment

Please sign in to add a comment. Not a member? Join today