Fog Creek Software
Discussion Board




What's wrong with SMTP ...

... that can't be fixed with a client that would upload whitelists to your e-mail server?

Anonymity should exist, no?

E-mail programs need to "innovate".

Scott Fitchet
Tuesday, May 13, 2003

That assumes white lists are a perfect fix. And it also assumes a perfect white list system would fix SMTP.

A perfect white list of ip addresses or email addresses?

A perfect white list of email addresses does little for the current MTA network.

You are attacker X, attacking MTA Y, observing innocent transactions Z

X sniffs network, finds a pool of Z hosts and Z addresses that makes it to MTA Y because MTA Y took the request, accepted the transfer of email.

X knows what's not acceptible email, because a really smart MTA Y would reject any hosts or email addresses it doesn't like.

X then spoofs using good Z hosts and Z addresses, MTA Y is defeated.

Use secure SMTP, helps a little, but threat model changes little.

However, if you start to digest every little 2 words that passes through MTA Y, and do bayesian or heterogeneous scoring, and castrates any connection that starts to do really well on the spamming score on a live connection monitoring basis.. you might get somewhere.. assuming you are being updated correctly on what is considered spam.

For example, you get a SMTP connection from www.hotmail.com.. hotmail wants to give you 26 emails going to email users A-Z on your host in this SMTP transaction. You gotta use A's bays scoring for one of those 26 emails and not for everyone. Or your scoring will lower and you'll cause false positives.

Li-fan Chen
Wednesday, May 14, 2003

But hell don't give up. Using white list is basically a policy fix. One of the problem with email as a networking "technology" is that it's too raw, which is just perfectly good for the anarchic difference in policy among organizations (or even individuals) and in fact key to its success. But when what you want is a reliable simple to use email that has no spam, you need a widely (enough) used policy that will stick--this part is always hard.

Policy examples include: electronic stamps (using wasteful computation or e-pennies--er..that's one and the same lol); white lists; PGP or human authentications; sending spammers to jail or charging them fines; previewing an email first few lines of text (or most interesting features) in the preview pane plus ordering by scoring in the preview pane with elimination of validating hidden gifs; all of these are policies that found their way into actual softwares.


If the emailing world is never friendly enough to give rise to a few serious death-blowing policies I think all hell would break loose and email will really have to worry about being eclipsed by some other communication tool.

Li-fan Chen
Wednesday, May 14, 2003

Here's one of my favorite (but I didn't invent this) challenges that fits the policy: you gotta be human or you must be a mass mailer; so proof you are a human.

Have a Outlook and webmail plug in that autoreplies within a reasonable amount of time upon the reception of any email that the sender's email will almost certainly be deleted if they don't have an account say at
www.iamhumandammit.org network..

What do you do with this account?
When you log in and you see a "recognize the following 4 characters or numbers" test and you gotta fill that out. Everytime you fill the bloody thing out you get karma.

Whenever an email is sent a digest is associated with the karma, spending it..

But when the email is actually read by the receipient.. his plug-in sends the email digest to the website and actually "spends" the karma debt.

And your karma lowers like that.

And karma can be ridiculously strong.. or weak.

So most software will have a gray list...

Spammers have very weak karmas...

Permission-mailers will have to pay for increasingly strong karmas.. (or users -- depending on their feeling.. can let certain permission mailer groups email them while spending little karma). Maybe the permission-mailers pay for something (like the way they pay for Friends on Fox network or offer yahoo mail for free) to buy karma.

And friends and families have ridiculously strong karma.

Now the problem is what would happen if people hack into a strong karma account and sends off a million email?

Well I guess we can't play favorites then, then I guess some Open Source dood will help communism reins and create a email torrent where everyone has same karma.

Li-fan Chen
Wednesday, May 14, 2003

Email has become next to useless.

What do we need email for that wouldn't be better served using other methods?  Seriously, there is this attachment to email as if it THE killer application of the Internet, but IMO it's redundant and annoying.

We have forums that cover the need for delayed messaging and message lists. Instant messaging, for more immediate needs.  Perhaps sending attachments might be useful, if most accounts didn't have a pitiful 5 meg limit, but that too could be done through a forum or some form of personal "webspace".  Why do we need a bloated, insecure, spam infested email system? 

I only bother checking my email once a day if that and I tell people not to bother sending me email.  I know I'm not the only one.

just an opinion
Wednesday, May 14, 2003

IM or other transports of messages will become the new ground for spam if email usage tails off.  Throwing email away because of spam will only move the problem to the next medium.

Strategically, I can't see stopping spam.

Tactically, I can see several ways to mitigate it.

Exposed open relays and socks proxies are obviously evil.  And most open relays are completely ignorant.  When you receive an email from a blacklisted open-relay, don't deny it, just send it a temporary failure to deliver.  So that the open relay keeps storing the junk mail and trying to resend.  I think this is called a tar-pit, or something.

Egress filtering by ISPs would also help the integrity of the internet greatly, maybe most in DOS etc but I'm sure it would help everything a little.

Nice
Wednesday, May 14, 2003

Nothing is wrong with SMTP.  Its a protocol not a moral contract.

Simon Lucy
Wednesday, May 14, 2003

Oh no. Not again.

The Petunias
Wednesday, May 14, 2003

E-mail is the killer application. What do you see people doing in internet cafes? Checking their hotmail.

The only thing that rivals email in popularity is SMS messaging. Chat rooms come third.

Stephen Jones
Wednesday, May 14, 2003

All "spam fighting" suggestions first have to define spam. And doing this is surprisingly difficult. 

For example is any form of mass-emailing spam? I think not. There are plenty of valid mass-emails that are desirable to some people. Supplier price lists, or announcements of "specials" spring to mind.

Is all unsolicited email spam?  Not really.  I sell software. I regularly get pre-sales type questions. I don't really solicit them, but I certainly get them, (and I certainly WANT them)...

Can a mail be Spam to one person and not to another? Certainly. I might want the IBM pricelist every day, my mother might not. (Which is why server based spam fighting is more difficult than client based spam fighting.)

Plus what is the cost of a false positive? Assuming I lose 2% of my support emails. What does that cost me? Assume I lose 2% of replies from support people I asked for help.. how much does that cost me?

Legislative solutions are frankly not much good. Remember there are many many countries out there. Email can come from anywhere. (you can't even blame the product provider for it since then all I'd need to do is send spam with my competitors product to get him shut down...)

Most technical options I've seen are simplistic to the point of stopping nothing but legitimate emails. For example reverse-dns is a complete waste of time, since it's perfectly legitimate to have separate paths for outgoing, and incoming emails. checking for "valid from" addresses is equally useless since it's trivial to get a valid (hotmail) from address.  ISP's routinely prevent multiple emails (more than say 50) being send on a single "connection" to the server.  This stops a spammer for about 40 nanoseconds while he reprograms his sender to send 49 mails per connection. Meanwhile Outlook Express users sending "newsletters" to all their friends and family suddenly can only have 50 friends...

Do I have the ultimate answer? Well no (otherwise I guess I'd be rich <g>) - but I suspect the solution is less "how to erradicate spam" and more "how to block most spam"...

Bruce Johnson
Wednesday, May 14, 2003

"Email has become next to useless."

Funny. Just this morning I saw a report that 80% of businessess now prefer email over more traditional communications channels (phone, fax, ...).
There are some infrastructural problems related to spam. Some of these are legit, but some are also a result of shops continuing to act like its 1989, operating stone-age mail technology that expects volume to be puny and message size to be tiny.
This is 2003. People need to send each other files. Email attachments are the natural way to do this, but I know several places where the BOFH (signature feature: types mail at the command prompt) balks at anyone "daring to send a 2Mb file to 20 recipients"! After that they are outraged that people dare to instal IM clients.
Oh pulease!

Just me (Sir to you)
Wednesday, May 14, 2003

[I only bother checking my email once a day if that and I tell people not to bother sending me email.  I know I'm not the only one]

No not the only one, but you are in the minority. I dislike cellphones so I don't have one. I am in the minority.

Ian Stallings
Wednesday, May 14, 2003

[Most technical options I've seen are simplistic to the point of stopping nothing but legitimate emails]


My favorite is attemtping to use the VRFY keyword to make sure the mailbox is a real mailbox. I guess they don't realize that *99% of SMTP servers have this disabled Why? To make sure spammers don't verify your address of course ;-)

*(That percentage figure is based on my experience writing SMTP software for the last 3 years, so yes it's an opinion dressed as fact).

Ian Stallings
Wednesday, May 14, 2003

Obviously some people are not ready to conceive of an Internet without email.  How could they? After all *email* was the huge flashing neon sign of the wired world.

IM is for communicating with people you already know.  Working from that premise it's not hard to see why spam wouldn't be a problem.  SMS is analogous to instant messaging, who has SMS converstations with people they don't know?

The killer app of the Internet is this feature we are using right now, the forum.  Email has a lot of detractors for a "killer app".

Hotmail is a web based application, it could be a personal forum for all we know or care.  The technology of how the messages get there is irrelevant.  To services like hotmail is how I see things evolving, it's much more extensible any can be detached from an anachronistic legacy protocol like smtp.

Who cares what 80% of businesses think?  80% of companies probably didn't even know about the Internet 10 years ago.  Email is the fax of the 21st century, it is crap but will take a long time to die.

As for being in the minority, so what?  The first people to ever use email were in a minority once too, what does that prove?

I will refrain from writing a long list of reasons why email, at least in it's present form, sucks and deserves to die.  But I do have two questions.  Name one feature of email that isn't better served by other technologies?  Which comon Internet technology has proven to be the biggest headache, not only for administrators, but for all of us?

just an opinion
Wednesday, May 14, 2003

I love email. It's my primary method of communication and I don't think there is anything "wrong" with email or SMTP.

I have 5 email addresses (different domains) and check them most days. I get SPAM, quite a bit actually, but I boot most of it. I scan for emails from people I know or those that "look" legit and read them. Everything else gets booted.

To me, it's no different from the analog world. I get SPAM in my physical mailbox everyday. Usually it outnumbers real mail. I just toss it. I understand the difference in cost, but it's really not that different for me. I have to walk to the mailbox, bring the spam home and then toss it.

I think the technical solutions that allow people to control what is "junk" mail are fine. The rest is annoying and just deal with it. It's the price of a free market. The only things I find really annoying are the porn emails. Those are preventing me from giving my 11 year old an email address.

We could go the route of regular mail and force everyone to use the USPS or some limited number or orgs for all email. That would allow "some" more control, but I'm not sure I like that idea. Not sure I want the Chinese government limiting what emails get sent. Not sure I want the USPS to potentially block my emails.

Leave it alone and deal with spam individually. You will never get it to go away.

Steve Jones
Wednesday, May 14, 2003

Verification of origin would get rid of 95% of spam, and could be done without having to modify email client programs.

It could work like this:  When an email is sent, its header will include a huge random number or a long number that is a hash of the message content.  When the message arrives at the destination server, if you have configured your account to only accept verified senders, the destination server will send a message back to the originating server, asking "did the account xxxx@xxx send a message to the address yyy@yyy with a code number beginning with XXXXXXXXXXXXX?"

The originating server must then respond with the rest of the long number to confirm that it indeed was the sender.  If it replies correctly, the next time you check your mail the message will be in your inbox.  Otherwise, after a certain number of unsuccessful retries over a period of a certain number of hours or days, your ISP's mail server will discard the message.

Once you can verify where it is coming from, "block sender" will be much more successful because it won't be so easy to fake the message's origin.  Of course it won't stop 100%, but neither does the lock on your front door.

T. Norman
Wednesday, May 14, 2003

T Norman you the bomb.

Li-fan Chen
Wednesday, May 14, 2003

I think the problem with every anti-spam solution is false positives.

It is bad if I don't receive an email from my customers (which is why I don't block anything)

It is usuallyworse, when a customer doesn't receive email from me.  Some examples:
1. They don't get annoyed because they don't get an email receipt for a payment because the email from the payment system gets blocked as spam (it's "unsolicited" as they didn't email it first). These systems which request a number codes typed in, are horrid in this kind of situation, as there may be a number of parties involved (e.g. payment processor) who are not particularly concerned about reading the number and typing it in.
2. They keep sending me a question, asking why didn't I get your reply? when I've already replied 6 times in the past 3 days
3. Their ISP decides it doesn't like mine.  I used to a lot of a problems with users from a large particular US ISP... they had decided to "help" their customers by blocking spam - which included blocking UK's large ISP (which definitely isn't spam-friendly) and much more email from outside the US! Eventually I discovered they hadn't blocked hotmail.com so for these folks I had to communicate via hotmail!

I think much so-called spam is in the eye of the beholder... which isn't to say I wouldn't be happy if I never received another message about enlargement or the former dictactor of Nigeria.

S Tanna
Wednesday, May 14, 2003

The "quality" of spam has really gone downhill over the last year or two. Once it was merely irritating, now it's actively offensive.

I think Bayesian filtering is the way to go. I use this, and it works well. See http://www.paulgraham.com/spam.html

Better than being unemployed...
Thursday, May 15, 2003

Mr Normon,

I don't think your verification system will work. I'm not wanting to pick on you personally (so please don't be offended) but the solution does demonstrate some of the problem...

In your scheme the receiver asks the sender if "account xxx@xxx sent a messge."  This is a problem because currently the _sending_ of emails isn't related to the _receiving_ of emails.  Sending is SMTP. Receiving is POP3.  It is very common for ISP's to have different machines for outgoing mail, and for incoming mail.  Fundamentally outgoing mail doesn't have "an account" (yes you may have to login to send, but there's no "sending mailbox".)

Secondly the receiver has only 2 things to go on - the IP number the message came from, and the "from" address in the email.  Connecting to the "from address" will get you to my "receiving" email server, not my sending email server. (So in that case you're talking to the wrong machine) - if you talk to the IP of the sender,  well that's where the spam came from, so yes it did send the email...

Another problem is the issue of "setting your account to only accept verified senders".  This would mean lots of your mail getting lost if sent from people who's ISP's didn't support this feature. 

But most of all, it wouldn't stop the spam. Sure it'd stop the _existing_ spam (ie what you got this morning) but it wouldn't stop spam as a concept.  Spammers will quickly work around this system, and all it'll end up doing is blocking legitimate emails.

Spam exists in any communication system simply because it's another way for companies to reach markets.  Billboards on the side of the road are a form of Spam. Adverts in the paper are spam. People phoning you to sell you something is spam.  What makes email different it the costing model. Email is basically free, hence it's an effective marketing tool.  If emails cost money then spam would be reduced (not vanished).  But the whole reason email is so nice is that it's free. That's a social issue, not a technical one...

Technically a "closed" email system (ie basically whitelisting who you can receive from) is fundamentally not usefull to most (albeit not all) folks who use email.  Just as a closed Phone, or Fax system would radically impact their usefulness.

Bruce Johnson
Thursday, May 15, 2003

Naturally, such a system would require additional information in the headers, and would require authentication (which is already supported by SMTP clients) when the mail is sent so the message can be tied to an account.

And I never said that SMTP itself would remain unchanged.  The server (or cluster of servers) that sent the email message would have to maintain a log of messages it sent out, so it can respond to a verification request.

As long as your ISP provides configurable options, perhaps via a web interface, most of the problems you state won't be problems.  You could set your account to only tag unverified mail rather than discard it, and then you can filter it with your email client.  You could configure your account to accept mail from verified senders plus a whitelist of unverified senders.  The spammer would then have to know and fake your friends' email addresses in order to get through, which is something they do not have the time to figure out for all the 1 million people who they are spamming.

Some spammers will get around it, but not 100%.  The point is not to make it impossible, but to make it difficult or expensive enough that the vast majority will find that spamming is not profitable.  Locks on doors never made burglary impossible, but think about how much more frequently it would happen if locks didn't exist.  Building an impenetrable fortress is not practical.  But placing locks on the front door is practical and has meaningful effectiveness.  We just need the equivalent of a door lock on our email accounts, where we are the ones who choose who should be let into our "homes".

T. Norman
Thursday, May 15, 2003

Actually, what I find most annoying about spam is that (without filtering) it gets listed hand in hand with real email, so I have to spend time and efford picking out what I want to read.

I don't seem to have that problem with billboards or postal junk mail.

Better than being unemployed...
Thursday, May 15, 2003

Hi T.

While I don't disagree that building a better email system would be a "nice to have" anyway, I still see flaws in this approach.

Starting with the analogy - while it's true that locks certainly reduce burglaries I would suggest that the biggest reason burglaries aren't worse than they are is simple because burglary is illegal.  Imagine if it were legal - then you would need a fortress. as I tell folks, "glass isn't a security device" <g>...

The other point where the analogy is false is the skill set required to perform a successful burglary.  The "entry level" burglars go after the residential houses, the "pros" go after mueseums. To go from being an entry-level burglar to being a pro takes a lot of (learned) skill, and I suspect a fair amount of luck. Not to mention a lot of brains. The barrier to being a successful burglar, long-term, is fairly high. (Remember the cops are waiting <g>)

Unfortunately with Spam the barrier to getting skilled is non-existent. Anyone can simply buy (for next to no money) everything they need. Software, addresses, know-how is all available in abundance. And of course it's legal (so if you fail you can try again.) 

Thus it's not enough to say "some spammers will get around it". If any of them get around it, then they'll all get around it in double quick time.  The speed with which the email network can respond to "attacks" is pitiful when compared to the speed with which spammers can change attacks.  And the legacy of each of these "attempts" is quite simply to block legitimate email.

Spam is a social problem, not a technical problem. To attempt to solve it via technical means is like taking an asprin to fix the problem of someone beating you on the head.  Yes it helps, but if the hammerer notices you don't care anymore then he simply beats you somewhere else.

I know it's easy to pick faults, and harder to actually come up with the solution (which I don't have). I'm not wanting to be negative, but any solution has to pass the following 2 tests;

a) do I still get any email that (given the choice) I would have chosen to get.
b) does "most" of the email I would have chosen not to get, get filtered out.

I'm inclined to accept a "reasonable success" rate in (b), but nothing less than 100% of (a) will do...

Unfortunately in the long run though, _any_ failure to filter (b) ultimately ends in the complete failure of (b).

Bruce Johnson
Thursday, May 15, 2003

Ok, go ahead and leave your doors and windows open when you leave your home unattended, and rely on the fact that burglary is illegal.  Or leave your car door unlocked and keys in the ignition, and rely on the fact that auto theft is illegal.

Technical preventative measures are significantly effective, even if they aren't perfect. They turn automated and brain-dead tasks into endeavors that require skill and time.

Merely using a little dinky firewall like Zonealarm is sufficient to immediately deter 99% of the 100+ people every week who attempt to break into my home PC.  Breaking into a computer that has a firewall up does some skill, luck, and determination and cannot be done at will by any fool.  But faking the "From" address of an email message can be done at will by any fool who wants to do it, and can be done en masse by a trivial script.

Most spammers spam for profit, not for fun.  With a sender-verification system in place, the increased expense of spamming combined with the reduced returns resulting from fewer people reading the messages would drive the majority of spammers out of business.

T. Norman
Thursday, May 15, 2003

I pretty much agree with the "a) do I still get any email that (given the choice) I would have chosen to get." has to be 100%

A problem with anti-spam solutions, is apart from requiring 100% accuracy for a lot of people is that people simply do NOT know until they read them, what they would choose to receive.  100% accuracy is therefore not only a high bar - but it is also one that requires the computer to read the user's mind.

I do not wish to receive unsolicited email from businesses? Ooops, I do in one case - I forgot the receipts and flight numbers, from the travel service where I booked online. I do not wish to receive email about raising my credit limit, oops the Amazon delivery information for the books I ordered got filtered because it mentioned when my credit card will be billed, and so it goes on.

S Tanna
Thursday, May 15, 2003

Only one real way to fix it:
money.

make people pay. or better yet, put up a bond. $1 a message, if you decide it's not spam they get it back, if you decide it is, they pay out.

or bonded large whitelists for large orgs.

mb
Thursday, May 15, 2003

New: put your competitor out of business

Write a script to sign up numerous addresses to their opt-in mailing list. Report them all as spam.  Or post to every thread on their message board asking to be emailed when new posts are made. Report them as spam.

I'm half-joking... but people expect free email. Don't expect them to spend a $1 every time some idiot hits the wrong button, or forgets they requested email, like an opt-in list, message board update, etc.  You know what would happen - message boards etc. would turn off automatic email features...

S. Tanna
Friday, May 16, 2003

I know this is ancient, but I'm waiting for the washing to dry...

SMS and IM are both used to spam subscribers, IM is a little easier in most systems to block unwanted messages but some of them less so.  SMS adverts are common.

Simon Lucy
Friday, May 16, 2003

*  Recent Topics

*  Fog Creek Home