It’s not really dying..
Turns out it was a “rogue” address trying to connect to the server via WASTE. The address would connect to port 1337 (creating an open “file” of a socket). Then, WASTE would need to open two files to check if they key is listed in its public keys table and also open the server’s private key file to see if the authenticating client’s public key copy matched. So, there are at least 3 open files per connection. And, with tens and hundreds of connections, these “files” are eaten up pretty quickly. Apparently, WASTE does not close these properly (at least in a reasonable amount of time). All of these open files exceeded the max open file limit set in the OS (around 4096, I think). With the max open file limit reached, the OS would not allow any processes (SSH, DNS, MySQL, Apache, etc) to open any files. Therefore denying logins, database reads (and thus DNS queries) and web page requests.
So, to resolve the issue, I simply blocked the address in the server firewall. That is, of course, after writing a simple, yet crude script to get the number of open files and reboot the server if the number exceeded a limit. Fortunately, it only ran once before I figured it all out. The offending address is 68.59.247.150. If that’s you, let me know. I’m not mad, I just want to help fix it. How do you know if it’s you? See the little image below!
