smtp python
Okay, so you've read a bunch of stuff about email, MTAs, MUAs, SMTP, POP3, IMAP and other acronyms but you still think that you're not getting the whole picture. You sort of know the theory behind all this, but you're not clear as to how email gets from point A to point B in the real world. Why, for example, your email client (MUA) can't connect directly to recipient's mail server? If mail servers use SMTP to communicate with each other, how do they bypass authentication? The days of open relays are over due to spam and you know for sure that you have to authenticate with your mail server to send email. This post will attempt to answer some of these questions.
Introduction
Both historical and security reasons make real life email confusing. Let's say user Alice, who has a GMail account and email address alice@gmail.com wants to send an email to Bob, who has a Hotmail account and email address bob@hotmail.com. Alice's mail client (MTA) is Thunderbird and Bob uses Outlook Express. Then Alice's email will travel according to the following diagram:
[[!img Error: Image::Magick is not installed]]
- Why can't Alice's Thunderbird send email to Bob's Outlook Express directly? Tossing other technical issues aside, Bob might not be online when Alice sends him her email. While many people can stay online 24/7 nowadays, this was unimaginable in the earlier days of the Internet. Therefore, a mail server had to be online all the time to accept and store Bob's email.
- Why can't Alice's Thunderbird send email directly to Hotmail's mail server? Alice, same as Bob, needs a GMail account to hold her incoming messages, but she doesn't really need it to send emails. As we'll see later, it is theoretically possible to send email from your own machine to someone else's mail server, but today there are spam-related restrictions that make it hard doing so. And by having gmail.com send her email, Alice can save some trouble of configuring and running an MTA.
Now, how do mail servers, such as the ones run by GMail and Hotmail, communicate? They do communicate via SMTP, the same protocol that Alice's Thunderbird uses to send emails and they don't have to authenticate with each other. Mail servers can be configured with several restrictions:
- They may allow clients from within their local network to send email without authentication. This might be possible when a company has its own mail server that provides access to everyone in the office subnet without authentication.
- They may accept mail only for users who have an account on the server. Hotmail or GMail servers accept mail for Hotmail and GMail users respectively almost from anybody (more on this later).
- Finally, when someone attempts to send email to an external address from an external network, a mail server may ask for a username and a password.
So, when GMail's mail server sends an email with a Hotmail addressee to Hotmail's mail server, it doesn't have to authenticate with Hotmail's mail server, because Hotmail's mail server will accept the email according to scenario #2. When Alice's Thunderbird connects to GMail's mail server to send Alice's email, it has to authenticate with the server according to scenario #3.
Using Python
That's enough theory for now, I think. Let's fire up Python and try sending some email. In the following example, we're going to reverse the roles and make Bob send Alice an email by connecting directly to one of GMail's mail servers using Python. This will also show you what (roughly) goes behind the scenes when mail servers communicate with each other. First of all, Bob will have to find out domain names of mail servers that accept mail for gmail.com:
$ dig gmail.com MX
; <<>> DiG 9.6.0-P1 <<>> gmail.com MX
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16600
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 5
;; QUESTION SECTION:
;gmail.com. IN MX
;; ANSWER SECTION:
gmail.com. 3048 IN MX 5 gmail-smtp-in.l.google.com.
gmail.com. 3048 IN MX 40 alt4.gmail-smtp-in.l.google.com.
gmail.com. 3048 IN MX 20 alt2.gmail-smtp-in.l.google.com.
gmail.com. 3048 IN MX 30 alt3.gmail-smtp-in.l.google.com.
gmail.com. 3048 IN MX 10 alt1.gmail-smtp-in.l.google.com.
;; ADDITIONAL SECTION:
alt4.gmail-smtp-in.l.google.com. 131 IN A 209.85.135.114
alt2.gmail-smtp-in.l.google.com. 145 IN A 209.85.218.49
alt3.gmail-smtp-in.l.google.com. 257 IN A 209.85.129.114
alt1.gmail-smtp-in.l.google.com. 41 IN A 209.85.199.27
gmail-smtp-in.l.google.com. 145 IN A 209.85.212.99
;; Query time: 39 msec
;; SERVER: 192.168.1.8#53(192.168.1.8)
;; WHEN: Fri Jul 17 18:55:58 2009
;; MSG SIZE rcvd: 230
As you can see, there are several servers that accept incoming email for GMail. Now, Bob launches his Python interpreter:
$ python
Python 2.4.3 (#1, May 24 2008, 13:47:28)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import smtplib
>>> server = smtplib.SMTP('gmail-smtp-in.l.google.com', 25)
>>> server.set_debuglevel(1)
>>> server.ehlo()
send: 'ehlo host.localdomain\r\n'
reply: '250-mx.google.com at your service, [74.125.67.100]\r\n'
reply: '250-SIZE 35651584\r\n'
reply: '250-8BITMIME\r\n'
reply: '250-ENHANCEDSTATUSCODES\r\n'
reply: '250 PIPELINING\r\n'
reply: retcode (250); Msg: mx.google.com at your service, [74.125.67.100]
SIZE 35651584
8BITMIME
ENHANCEDSTATUSCODES
PIPELINING
(250, 'mx.google.com at your service, [74.125.67.100]\n
SIZE 35651584\n8BITMIME\nENHANCEDSTATUSCODES\nPIPELINING')
>>> server.mail('bob@hotmail.com')
send: 'mail FROM:<bob@hotmail.com>\r\n'
reply: '250 2.1.0 OK 31si3999937yxe.40\r\n'
reply: retcode (250); Msg: 2.1.0 OK 31si3999937yxe.40
(250, '2.1.0 OK 31si3999937yxe.40')
>>> server.rcpt('alice@gmail.com')
send: 'rcpt TO:<alice@gmail.com>\r\n'
reply: '250 2.1.5 OK 31si3999937yxe.40\r\n'
reply: retcode (250); Msg: 2.1.5 OK 31si3999937yxe.40
(250, '2.1.5 OK 31si3999937yxe.40')
>>> server.data('Subject: This is test mail from Python\r\n\r\n
This is the body of mail from Python\r\n')
send: 'data\r\n'
reply: '354 Go ahead 31si3999937yxe.40\r\n'
reply: retcode (354); Msg: Go ahead 31si3999937yxe.40
data: (354, 'Go ahead 31si3999937yxe.40')
send: 'Subject: This is test mail from Python\r\n\r\n
This is the body of mail from Python\r\n.\r\n'
reply: '250 2.0.0 OK 1247871480 31si3999937yxe.40\r\n'
reply: retcode (250); Msg: 2.0.0 OK 1247871480 31si3999937yxe.40
data: (250, '2.0.0 OK 1247871480 31si3999937yxe.40')
(250, '2.0.0 OK 1247871480 31si3999937yxe.40')
>>> server.quit()
send: 'quit\r\n'
reply: '221 2.0.0 closing connection 31si3999937yxe.40\r\n'
reply: retcode (221); Msg: 2.0.0 closing connection 31si3999937yxe.40
And that's how he would do it. What if he decided to use GMail's mailserver to send an email to his friend Carl, who has a Hotmail account?
...
>>> server.rcpt('carl@hotmail.com')
send: 'rcpt TO:<carl@hotmail.com>\r\n'
reply: '550-5.1.1 The email account that you tried to reach does not exist. Please try\r\n'
reply: "550-5.1.1 double-checking the recipient's email address for typos or\r\n"
reply: '550-5.1.1 unnecessary spaces. Learn more at \r\n'
reply: '550 5.1.1 http://mail.google.com/support/bin/answer.py?answer=6596 9si4897266yxe.89\r\n'
reply: retcode (550); Msg: 5.1.1 The email account that you tried to reach does not exist. Please try
5.1.1 double-checking the recipient's email address for typos or
5.1.1 unnecessary spaces. Learn more at
5.1.1 http://mail.google.com/support/bin/answer.py?answer=6596 9si4897266yxe.89
(550, "5.1.1 The email account that you tried to reach does not exist. Please try\n
5.1.1 double-checking the recipient's email address for typos or\n
5.1.1 unnecessary spaces. Learn more at\n
5.1.1 http://mail.google.com/support/bin/answer.py?answer=6596 9si4897266yxe.89")
GMail's mail server rejected his request because it does not accept outbound emails from just anybody. What's more interesting, if you were to try this at home, both GMail's and Hotmail's mail servers would likely reject your email. GMail's reply:
...
>>> server.data('Subject: Mail from Python\r\n\r\n
And this is the body of email from Python.')
(421, '4.7.0 [74.125.67.100] Our system has detected an unusual amount of\n
4.7.0 unsolicited mail originating from your IP address. To protect our\n
4.7.0 users from spam, mail sent from your IP address has been temporarily\n
4.7.0 blocked. Please visit http://www.google.com/mail/help/bulk_mail.html\n
4.7.0 to review our Bulk Email Senders Guidelines. 5si2401533qwh.31')
Hotmail's response:
...
>>> server.mail('alice@gmail.com')
(550, "DY-001 Mail rejected by Windows Live Hotmail for policy reasons.
We generally do not accept email from dynamic IP's as they are not typically
used to deliver unauthenticated SMTP e-mail to an Internet mail server.
http://www.spamhaus.org maintains lists of dynamic and residential IP addresses.
If you are not an email/network admin please contact your E-mail/Internet Service Provider for help.
Email/network admins, please visit http://postmaster.live.com for email delivery information and support")
Additional reading
- Telnet - SMTP Commands (sending mail using telnet)
- Stack Overflow: I ALMOST understand how email works, but I’m missing something
- Stack Overflow: Failing to send email with the Python example
Remarks
The above is a true story, only the IP addresses have been changed to protect the guilty.
