Optimize pop3 for 15-20 clients?

I have a Webmin VPS with Linux.

I have setup Sendmail, which has been working great.

I also have popa3d setup.

We have about 15-20 people that inquiry the pop3 server every 5 minutes or so to check for new mail.

Today everything was working great...then about mid-afternoon we were getting an error saying that we couldn't connect to the pop3 server. I couldn't telnet to it either.

However, we could still use popa3ds and connect with SSL enabled on port 995.

Is there a way to optimize the popa3d server for this many connections throughout the day.

Also, is there something that would block our IP address access to that port since we access it so often? It obviously didn't block our IP address altogether since we could still access Webmin, the apache http site, smtp, and pop3 ssl.