Monday, July 7, 2008

Tomcat Paralyzed in Linux

I have gone through some experience installing tomcat in a Linux box. Just like to share some problem that I encountered.
When I start up the tomcat, one thing that I notice is, after a while, and then tomcat will hang. First it looks very strange to me. I start to question about the performance of the tomcat, which I don think that is the cause of the problem. But luckily after many rounds of try and error, we manage to figure out what went wrong.

1. The setup is: Apache, Tomcat, mod_jk, MySQL.
2. When I start the Tomcat, I monitor the activity vis:
netstat | grep 8009 | wc –l
and the number of instances increase until 512, then the tomcat hang.
3. Something I overlook which also tells me what went wrong:
netstat | grep 3360 | wc -l
4. Finally I conclude that the problem was caused by the SQL connection that is not closed.

When a request comes in, Apache call Tomcat via mod_jk (port 8009). Then in tomcat, my application call MySQL (port 3306) but no response (due to wrong setting. I was doing a migration). So the request keeps on waiting, waiting, waiting. . . and finally all the available ajp connection used up. Then tomcat was paralyzed.

A very strange scenario, as when I run top command, I don’t see java using high CPU or memory.

So, a good experience in deed.
Next time if your tomcat paralyzed too (touch wood), try look at this solution.

No comments: