Fog Creek Software
Discussion Board




On replacing our mail transfer protocol

The let's-replace-SMTP-article was one of the dumber ones lately. Not only would it be dumb to rewrite this 30-year-old protocol from scratch as it still works fine millions times a day for what its designed to do, but the article failed to actually point out what's bad with SMTP.

He writes that we should have authentication in mail. That's NOT part of the transfer protocol. Tying together them would be bad. Just use PGP. It's been around for at least 10 years and does its job fine. Just tell your MTA (mail server) to drop/bounce all unsigned mail. (If you don't know how there's lots of consultants, including myself, out there who knows the stuff.)

The other thing he wants is a rate limit. Well, d'uh. That's certainly not part of the protocol. Limiting incoming mail will solve nothing since you only get one (or a few) copies of each spam. Given that your mail server is correctly spam won't be queued through it, so you would have to use a stateful inspection firewall decoding your TCP traffic realtime. Now that costs money (a lot if your pipe is a fat one), and *may* reduce spam for *other* people. Not going to happend.

Finally, even Joel falls for this idea (I thought he might know better). This is probably from not understanding the problem. See, most spam are queued *without* the SMTP server's consent. Almost all spam in the world comes from abusing either open unprotected HTTP proxies or SMTP relays!

Even if these services are without passwords, I'd say such malicious use is a crime just like breaking in to their system. So even if the world could agree on a micropayment system (yeah, right...), what would you accomplish? A lot of people would get very large bills just after their computers got broken in to!

Spammers are criminals, and should be dealt with the same way as with other economic crime: large fines.

Jonas B.
Wednesday, April 23, 2003

I saw a really clever idea on /. once, someone has started an opensource project to create an automatic spam identifier plugin to mailservers.

the idea is that if an email is identified as very prolly spam during the transmission using the bayesian thingie then the mailserver can being slowing transmission down....the more likely it is spam, the slower the mailserver would accept the data from that connection, and any other email coming from the same connection.

leading to 2 really good outcomes.
(1) All mail is accepted and nothing is lost...false positives lead only to a slower transmission and do not mean lost email.
(2) anyone using the mail server to send spam quickly gets bogged down...any mail server using the plugin slows down data transfer from a spammer to such an extent that it becomes nearly impossible to use that mail server for spam.....1 email sent a minute is way too slow for a spammer.



And all that without replacing the current protocol :)

best of all worlds
Wednesday, April 23, 2003

Personally, I believe that spam, while a pain to deal with, is a small price to pay for the very best feature of smtp - the LACK of authentication. Because email doesn't enforce authentication or verification, people coming from policially oppressive countries can use it as a communication mechanism.

If the message is important enough, pgp/gpg or some other such tool will be used. I think there is a market to make these tools more user friendly, but the tools for message signing and authentication already exist.

Authentication is really a legal tool - a way of getting someone to own up to what he/she said (thereby, being a target for civil litigation). Besides, authentication can only verify that the email was generated by someone who possessed the key - it cannot verify that the owner is who he says he is.

SMTP is broken by design, and personally, I like it that way.

Wayne Earl
Thursday, April 24, 2003

"even Joel falls for this idea (I thought he might know better)"

The problem is that people get very wound up about spam and tend to knee-jerk react. That's how a lot of bad ideas get adopted (blackholing of dynamic ips for example).


Thursday, April 24, 2003

where is the evidence  that most spam is generated by open relays?

Nearly any mail server with an open relay is blacklisted; there are bots that go round the web identifying them and putting them on the blacklist. the amount of spam has tripled or quadrupled in the past year despite this.

To the best of my knowledge something like 905 of the Spam in the world (Nigerian scams excepted) comes from about 20 commercial mass mailers.

Incidentally you probably don't even need to give your email out to get spammed. Spammers frequently simply send out spam to made up addresses on the grounds that the odds are well in their favour that messages sent to johnsmith'@anyISP or stevejones@anyISP will hit paydirt.

Stephen Jones
Thursday, April 24, 2003

If the corollary of this is that everyone must be identifiable by the email address they use (which is what governments would like), how long would it be until anonymising would come back?

Immediately.

Trying to replace SMTP would be like trying to replace roads with umm roads.  They'd just be more expensive roads with tollgates.

Yes, that's an advance.

Simon Lucy
Thursday, April 24, 2003

"Nearly any mail server with an open relay is blacklisted"

This is patently untrue, if only because many of them are on dynamic ips and therefore are up/down/in/out as the owner connects and disconnects. The relay raper doesn't care - probe for port 25 access... open? good... send a few spams whose (b)cc lists will be exploded on that server, from where they will be sent on, in a few minutes or hours... job done.

"bots that go round the web identifying them and putting them on the blacklist"

1) Bots cannot "go around the web" (port 80 etc.) identifying open mail relays (port 25).

2) So what? Not everyone use "the" blacklist. And which blacklist were you thinking of exactly? See http://rhols66.adsl.netsonic.fi/era/rbl/rbl.html


Thursday, April 24, 2003

From http://www.theregister.co.uk/content/6/30368.html

---"Linford believes 90 per cent of the world's spam is down to 180 recidivist spammers. Deal with them and you've cracked the spam problem, he believes.

"If these 180 were somehow spirited off internet - we'd be left with the Nigerians, and companies spamming by mistake. The spam problem would simply disappear," he said. ®"---

The list of the 180  is at
http://www.spamhaus.org/rokso/index.lasso

Can you provide a source for your belief that Spam comes from dial-up end users' open relays. Your method appears to be a pretty efficient way of sending a mass mailing compared to paying a minimal amount to a commercial mass mailer.

Stephen Jones
Thursday, April 24, 2003

Just because 180 supposedly does 90% of the spam doesn't result in 180 SMTP servers sending 90% of the spam. Who is doing it and how they are doing it are two very different things. If you check the Spamhaus Block List (SBL) listsings for each of the spammers you see that most of them have used several different mail servers.

Removing the people beind the spamming would of course be a good solution, but until the legal systems take such issues seriously it would be good if everyone setup mail servers properly to fix the problem.

Kjartan Mannes
Thursday, April 24, 2003

Ehm..? Evidence? How could one possibly provide evidence? The above is my *experience* . I have at least worked with abuse on a large ISP a while.

Do you have other experience? Study the received-headers on your own spam and see where you end up! If it was true that most spam originates from a few (a couple of thousand, perhaps) servers, you could just blacklist them and get rid of it, right?

I believe the figures about a couple of hundred spammers responsible for most spam may be correct, *but* please understand that these people queue their spam through unsuspecting sys admins.

And of course they do! No ISPs endorse spam. It is a great pain for everyone -- and a great cost for ISPs as they employ lots of people *only* to take care of abuse, which mostly consists of spam nowadays.

These malconfigured servers, open relays, formmail.pl and open proxies come and go on millions of IP addresses as we speak. The open relay search bots do a good job of alerting the sysadmins but they are not nearly fast enough compared with the spammers.

As an example, we have had *several* customers that has found their newly installed Microsoft Exchange being abused to send many tens of thousands spam just while they are installing the thing and haven't configured it properly yet. They go home for the night to start working again in the morning and during the night their machines are busy ...

(Again we find bad Microsoft products causing great pain and cost around the world! Just like with the poor in-addr.arpa-servers for 192.168.x. Hands up everyone! How many of you are running MS XP with private address space and have not disabled "register in DNS"? You are causing extra work for people!)

Jonas B.
Thursday, April 24, 2003

Dear Jonas,
                From reading links on the subject it does appear that much spam is sent by people paying an ISP to send it. That is why the 180 people on the list keep changing their names and registering new domains, or even setting themselves up as ISP's. And ISP's will take them because they may not know that the mass mailing is spam (I  get about twenty-five newsletters a week that are perfectly legit mass mailings) and in more cases they don't really care because they are getting the money anyway.

                I've been checking the headers for the messages and it does appear that the latest  avalanche of spam by the penis enlargement stuff and the growth hormone and the anti-viruses are from open relays. The amount of this rubbish I am receiving has tripled over the last three months, so maybe the SpamHaus stats are out of date.

                    The question is how are those that have an open relay to know about it. Looking at some of the headers from todays spam I see Xmail being Outlook or even Outlook Express. Are these simply dial up users having their ports scanned? How can they be told?

Stephen Jones
Thursday, April 24, 2003

Without out a doubt, the SMTP should be replaced and fixed to something that does not allow sending of a email without a valid return address.

It is also clear that it is not practical, or even feasible right now to replace SMTP.

I also believe that SMTP should be replaced.

However, right now that is not practical.  The fact of replacing SMTP not being practical has nothing to do with the fact that it should be replaced.

We just can’t replace it right now, but that don’t mean it should be replaced!!

Joels idea makes perfect sense, but can't be implemneted right now....

Albert D. Kallal
Edmonton, Alberta Canada
Kallal@msn.com

Albert D. Kallal
Thursday, April 24, 2003

Most spam I get has a valid return address. How else would the spammers get my custom.

What they don't have is the return address being the same as the sending address, but there are often very legit reasons for not wanting that.

Stephen Jones
Thursday, April 24, 2003

The problem is, even if you trace back past the open relays to the actual ISP the message came from, the big ISPs have ceased to care.  They "Forward the complaint on to the individual customer" who then does.. well.... nothing.

I'd wager that the spam problem right now is the large ISPs who sign pink contracts with general spammers and spammers who claim to be opt-in networks.

The problem is that spamming is like money laundering.  On one end is a person in the US getting a spam message.  On the other end is a company in the US paying to have a spam sent.  There is probably a open relay and/or a company in China/Korea/etc. in the middle, perhaps a few layers of indirection.  In order to really take down the company at the top, you have to prevent them from claiming blamelesness.

We need a law like the treason law.  Fine any ISP who knowingly provides comfort to a spammer $10/message.  Make a legal standard that makes sure that it doesn't apply to an ISP with a rouge customer, but only as long as they cut off the customer when spam reports flow in.

Incidentally, all of the spammer address gatherers, or at least, the ones that violate the norobots.txt specification, are in China now.

Flamebait Sr.
Thursday, April 24, 2003

Of course just having a valid return address does not solve everything, but it helps. This also does not mean the “end” of having a different return address form a sender (we just need a mechanism to prevent someone from not using other peoples email id without permission).

However, look how incredible the phone system is since we now have caller id.  It reduces a lot of hassles, and having consumers know who/where the phone call is coming from is a rather nice feature.

Interesting, as we are now in the middle of change of the phone system to a IP based phone system, this might cause some real traceable  and “accountability” problems. We could see a real rise in phone based abuse that caller ID put a end to as IP phone technology takes off…

Albert D. Kallal
Edmonton, Alberta Canada
Kallal@msn.com

Albert D. Kallal
Thursday, April 24, 2003

Ummm at the same time as you have caller id, you have the ability to disable it, its rarely enabled on PBX's or direct extensions (in the UK at least).

Simon Lucy
Friday, April 25, 2003

best of all worlds,

the technique talked about on /. is called a tar pit, and it won't work.

Basically an intentially tuned spam server can just realize that a major ISP on a major pipeline and mysteriously gone from 1 megabyte per second to 20 bits per second.. and react in real time. It can: 1. Cut the connection; 2. Tell other's about the tar pit and attack it.

A tar pit works by maintaining an open tcp connection. And that's not a limitless programmatic resource. You just need to send 50,000 concurrent connections all resulting in a tar pit before a server is overwelmed.

The behavior of an overwelmed tarpit server is probably not damaging.. it won't let email through, but it will exhause some memory and processing power to deal with these fake connections.

Having a real time server that doesn't tarpit.. but basically grade a connection is good though.

Say connection I, J, and K comes from 3 unknown SMPT sources. I is getting a lot of spams according to a live "filter/bays analyzer". J and K is not so bad. You could grade I a "D+", J and K a "B".. and keep track of that average. That way if it's been a D+ for a long time an administrative action is required or you are forced to ban it.
Bs are tolerable to a degree, but you'll always have to waste say 15% of your network traffic to spam.

Another thing you can do is cut off any connections that even dares to send one single spam in real time. That puts the responsibility on the sender to basically figure out what is good email and what is "bad" email and send only the good. Most major ISPs can pull this off. And because this is live you will never ban a major domain. You are just simply not allowing a single spam bit to land on your bit bucket--and allowing all good traffic to pass no problem.

This would be best done with a bays network and a distributed grading network.

I also think a pay to read system would be helpful though someone say it's still completely useless because it forces the ISP to store and serve spam anyway.

Li-fan Chen
Friday, April 25, 2003

Stephen:

I believe the belief that spammers pay their ISP to queue mail is a misunderstanding. Sure, the large spammers may continously start up new companies but they would be shut down very fast if they used their ISPs mail servers. They are not that stupid, they use other means as far as it is possible.

A spammer can get thousands of complaints a day to the ISPs abuse department, something that costs real money. I agree most of the problems today are with dialup users, DSL users and to some extent small Asian ISPs that sometimes don't accept abuse mail at all.

Albert:

Why would you want to redesign SMTP just to require a valid return address? What has this to do with the transfer protocol? Nothing! Just drop incoming mail on your server without a valid return address today, it's not difficult at all.

Jonas B.
Friday, April 25, 2003

Whenever you want to send out a newsletter, like the Joel on Software newsletter for example, you shop around for an ISP that will send the mass mailing for you.

Many won't ask too many questions, out of ignorance or cash flow problems. Remember, when the complaints come in the spam is already out. The spammer then simply chooses another mass mailer.

Stephen Jones
Friday, April 25, 2003

Having a widely accepted protocol for authenticating the identity of the sender does not preclude anonymous email.

It could and should be configurable so you can setup your account for various options:

- Accept all email
- Accept all certified email, reject everything else
- Accept all certified email, and uncertified mail for which the apparent sender is on your whitelist; reject everything else
- Accept only certified email from people on your whitelist
- Reject certified email from named senders

People who want to communicate with each other without revealing their identity to third parties could continue to do so.  The only ones who would have a real problem are those who want to send you unsolicited mail without a verifiable identity. And once they reveal themselves, you can then block them.

T. Norman
Saturday, April 26, 2003

*  Recent Topics

*  Fog Creek Home