SMTP:
There is the the famous sendmail server. I believe it's the most common email server currently.
Then there is Postfix, which I found on serverwatch.
Doing a google showed Exim also. I've never had experience configuring an mail server but sendmail is generally installed with most Linux distro's I believe.
POP3:
Qpopper says they are the most widely used POP3 server so I'm guessing their pretty good.
Then there is fetchmail which supports both POP3 and IMAP.
From doing a quick search there seemed to be a LOT of mail servers. Lots of them are opensource and should be pretty good.
If I were to setup a mail server I would probably use sendmail and fetchmail
a)because there names sound alike
b)sendmail because it's so popular
c)fetchmail because it supports both POP and IMAP
I would first decide what your planning on doing with this mail server and then choose the right mail servers.
Note. Serverwatch has a great little survey that compares different servers, I would go there to check it out:
http://www.serverwatch.com/stypes/compare/
Good luck.