Fog Creek Software
Discussion Board




Re: "Bad Spam Filters" spiel

Joel,

In the "Bad Spam Filters" article on November 14, 2002
( http://www.joelonsoftware.com/news/20021114.html ), you made a number
of dicey assertions. Most notably, the assertion that
WhatCounts ( http://www.whatcounts.com/ ) is a legitimate email
delivery company.

A quick search for "whatcounts" on news.admin.net-abuse.sightings
shows enough examples to indicate that their so-called "double opt-in"
system is not remotely perfect:
http://groups.google.com/groups?q=whatcounts+group:news.admin.net-abuse.sightings&hl=en&lr=&ie=UTF-8&sa=G&scoring=d

BTW, "double opt-in" is a spammer phrase. Please don't use it. I mean,
think about the meaning of the phrase - it implies that you've opted
in... twice? Huh? What would be the point of that? You should only
need to opt in _once_, but (and this is the important part) it needs
to be confirmed that it's you - the owner of the supplied email
address - opting in. So a CONFIRMATION email should be sent to the
given email address, requesting (surprise, surprise) confirmation that
it was really you that "opted-in" - hence the appropriate phrase
"confirmed opt-in".

Another collection of clues as to the nature of WhatCounts is revealed
in their so-called "100 Tips, Tricks and Insider Secrets For
Successful Permission Email" document, linked on their website:

http://media.whatcounts.com/insider_secrets.pdf

Consider page 8, with the header "Ten Ways to Sidestep Being Perceived
as a Spammer". Umm... sidestep?? WTF??

Consider the frequent references to "double opt-in" (with occasional
secondary references to "permission email" and _once_ to confirmed
opt-in). The suggestions under "Expand Your List" (page 3) are
particularly revealing, but the real killer is the advice to "purchase
opt-in lists"... as is further detailed on page 9, "Twelve Steps to
Purchasing a Quality Email List".

Here's a tip: it's _not_ _possible_ to purchase an opt-in list. It
simply doesn't make sense. It's ludicrous. And if you have trouble
grasping this concept, think about this: if your spouse or significant
other has "opted in" to having sex with you, can you then sell that
opt-in to others? Hmmmmm.

The only person that can give (or sell) permission to market to them
(or fuck them, or perform any other kind of action on them) is that
person themself (and sometimes not even then if, eg. they're a minor).
You can't claim rights to sell or otherwise distribute permission just
because you have it, any more than someone can claim rights to sell or
give away copies of Fogcreek's software just because they own a
licensed copy.

And just to put the final nail in Whatcounts' coffin: "Whatcounts.com
offers list confirmation as an OPTION only, so any mailing list they
host is likely to be an abuse magnet." Quote from:
http://groups.google.com/groups?selm=u3busqia4o70ea%40corp.supernews.com&output=gplain

You can probably verify this yourself, Joel. When you started using
WhatCounts as a mailing list manager and supplied it with a list of
already-subscribed (to your mailing list) email addresses, did it
insist on doing a confirmation run on all those addresses? Or did it
allow you to avoid confirming?

Anyway - WhatCounts looks like they are at _best_ a not entirely
conscientious email list manager who will avoid putting in any serious
checks and balances on their system because it might reduce the amount
of spammer money they receive. At worst, they're deliberately courting
more "respectable" (ie. wealthy) spammer customers while doing
everything they can to "sidestep" (their own word) being perceived as
a spammer or spam supporter themselves. In any case, I'd say it's
perfectly reasonable for ISPs to refuse to accept email from their
servers until they clean up their act. Disowning that embarrassing
document referred to above and forcing confirmed opt-in for all client
mailing lists they manage would be a good start.

Finally, with regard to your comment "On the other hand, overzealous
system administrators are causing serious damage to the connectivity
of the Internet by imposing draconian spam filters" - I'd simply state
that UNDERzealous system administrators (not to mention the management
giving them their orders) are doing far far far more damage to the
Internet email system by continuing to provide support for their
spamming clients. I'd also suggest that, despite you not being a
spammer yourself, you're also helping to damage Internet email in your
own small way by continuing to fund organisations like WhatCounts.

BTW, were you also aware that the upstream provider of WhatCounts is
Level3? One of the worst spam-supporting ISPs in the US?

Pete.

Peter Wright
Saturday, November 16, 2002

Well, the CEO of Whatcounts.com is a reader of this discussion group so it will be interesting to hear his response...

Joel Spolsky
Saturday, November 16, 2002

I had a look at the WhatCounts.com web site (I trust Joel's recommendations), as I have a mailing list that I'd like to move from a free server to a more professional service. I looked all over their web site, and couldn't find any detail about what they actually provide, or what it costs. The site seems to be just one big brochure.

If the CEO does read this forum, I'd like to let him know that a potential customer just walked out of the shop. I'm not going to do the '100 question dance' with a sales rep just to find out how I can give you money.

Darren Collins
Sunday, November 17, 2002

Not related to the company, but about spam filtering: I am happily using Cloudmark SpamNet ( http://www.cloudmark.com/products/spamnet/ ). Its blacklist is formed uniquely by other users - and it blocks the message, not the sender - so it is really hard for it to take legitimate e-mail as spam. Along with pobox filters, it reduced my spam to a tiny, manageable level.

Chester
Monday, November 18, 2002

A healthy debate about spam is almost always entertaining and often educational. Like everyone else, I receive lots of spam.

Regarding Cloudmark - I use it too. Great product. I actually started using their first beta and then switched to MailFrontier's Matador, which I like more or did until recently. For some reason Matador stopped working and would crash Outlook. Talk about web sites with few information points - try finding help from them. So, I'm back to CloudMark. In general, collaborative filtering is a good idea provided the population is large enough and the mechanism are truly democratic and can prevent ballot stuffing.

Someone had an comment about our web site being just one big brochure. Well, that visitor is right. In our space, few companies provide pricing online - for a variety of reasons. The biggest is probably that every customer has unique requirements and, thus, obtains custom pricing for their work. Since we're primarily focused on servicing customers with many thousands of recipients we don't attempt to compete directly with services like bCentral, Constant Contact, etc. They'll all reveal pricing. But firms like ours, Digital Impact, etc. tend to prefer more personal dialogs with customers. We understand and respect people that are looking for alternative solutions. Still, Darren's comments are appreciated. We'll certainly try and improve the usefulness of the site.

As for dancing with a sales rep - we're too small for that. Most prospective customers end up talking with principals of our company - and we're not at all aggressive when it comes to sales. But there are always questions that are useful and often required. If a customer is sending out 5,000 messages to their customers on a monthly basis the price structure is certainly different than the customer sending out 250,000 order confirmations. Do they need to use one of our three APIs? Will they be creating their own content? Without knowing these things the sales cycle is a waste of everyone's time. And for those prospective customers that think they know all that they need to know and just want raw pricing - experience has shown that they either don't know all that they need to know or would be attempting to simply find the lowest cost provider for blasting services.

I read Peter's comment too and he had lots of good points, though I disagree with him on some of them. Incidentally, he's right about seeing some notes about WhatCounts in the news groups. Check the dates, though. It's not all timely.

I stronly disagree about "double opt-in" being a spammers phrase. That's simply wrong. It's a perfectly legitimate mechanism for insuring that people that want to signup and receive email get a chance to confirm. Take Joel's list. That's double opt-in (also known as confirmed opt-in). When you sign up for his list you get a confirmation message. This insure that someone else doesn't join you to a list without your permission. It also helps insure that only valid email addresses are added to his list. What method is preferred? With Joel's list you won't be added unless you confirm via a message sent to your specified email box. What if a friend signed you up and you didn't want to join? Well, you don't have to confirm. What if you signed up and you put in the *wrong* email address. Mistakes happen. So, given those two scenarios - what approach would you prefer? Please don't suggest Passport! <g>

As for "Ten Ways to Sidestep Being Perceived as a Spammer" I can most definitely assure you that our goal here was to promote good email marketing practices. Did you read the section? No apologies here. We want our customers to be responsible emailers. We've had some customers that didn't "get it" and we helped forcefully educate them.

As for purchasing opt-in lists, some people simply fail to realize that there are legitimate lists available for resale - though we tend to suggest to our customers that they engage those firms directly (the legitimate ones won't give you their lists - just allow a vendor to email to them). Under no circumstances would we ever support or allow a customer of ours to buy a CD or email addresses and use our resources to email them.

As for our allowing customers of ours to use confirmed opt-in as an option - that's simple. Suppose Joel had come to us (as he did) having used another solution (perhaps a self-built one). Would he be advised to email his entire list seeking new permission to email them simply because he changed his tool and was using another resource? There are lots of cases like this - customers coming to use with their existing list of customers. These are certainly examples where a forced confirmation process isn't appropriate.

My note is likely already too long. I'll happy answer any questions directly through email. And, anyone visiting the Seattle area is welcomed to stop by and have a substantive chat. Coffee is on me.

David Geller
Monday, November 18, 2002

Sorry about all the typos and grammatical errors.

David Geller
Monday, November 18, 2002

"As for purchasing opt-in lists, some people simply fail to realize that there are legitimate lists available for resale."

Okay, I'll bite.  Define legitimate lists available for resale.  What specific measures does your company have in place to ensure that these are legitimate?

Secondly, what specific measures does your company have in place for ensuring that the non "legitimate" lists are not used by your customers?

"Suppose Joel had come to us (as he did) having used another solution (perhaps a self-built one). Would he be advised to email his entire list seeking new permission to email them simply because he changed his tool and was using another resource?"

If Joel (or other customers) were unable to provide proof of confirmed opt-in messages for each email address they wished to use, then yes!

As mass email sender, your company has a responsibility to ensure that every recipient actually wants that email.  Otherwise you are wide open to abuse and very likely contributing to the problem.  The minor inconvenience of requiring another confirmation process far outweighs the benefits of knowing that you are not unwittingly allowing someone to abuse your service (and honestly - "accidentally" being spammed versus getting asked for permission a second time - it's a no brainer as to which I'd prefer!)

If your company is really serious about not contributing to the spam problem, you'd reconsider your policy on confirming all email addresses.

Failing that, you'd have a mechanism so that upon request from the owner of that address (or domain), you could refrain from sending emails to a particular address (and ideally domain) unless you had a confirmation email.

Can your company do that?

Just a thought
Monday, November 18, 2002

We monitor opt-out activity and bounce activity - often a strong indicator of stale lists and inappropriate sending. Also, all our agreements require customers of ours to insure that their lists are permission-based and be able to prove it. We've suspended activity before and have also helped educate customers with regard to good citizen practices. Is our solution perfect? Probably not. Do we consciously try to remain on the good side of email publishing? Absolutely. In fact, our core focus and compentency is in on-demand and data/event-driven publishing. You know, the kind that happens when you order a product and need a confirmation message. Or your membership is about to expire from your favoriate NPO and you need a reminder (and gave that NPO express permission do to so). Or you need a solution to tie web-based publishing with email (breaking new delivery, etc.). We're not in the spray-and-pray business of email sending. The numbers we're dealing with are so small.

As for legitimate list providers - I'm no expert in this space. We often refer customers to groups like the DMA. We don't offer any list rental, leasing or selling services. Additionally, we don't aggregate any data between our customers nor, through our agreements, take ownership of any of their email addresses. But there are legitimate firms out. Again, they'd never provide lists that we could email through regardless. All our customers could do would be to give them content and have them email on their behalf.

David Geller
Monday, November 18, 2002

"Double opt-in" is a term used by spammers to justify their not doing confimations by making it sound unnecessary (why make them opt in twice?) and difficult.  Please use the term "confirmed opt-in" and avoid sounding like a spammer.

On purely literal terms there are indeed lists of confirmed opt-in lists that the subscribers have given permission to be sold.  When one of my users reports spam to me and requests that the sender be blocked, I investigate the sender.  I read their privacy policy, and I may try out their signup method.  Exactly once I have found a sender that does confirmed opt-in *and* makes it very clear to subscribers that by signing up, they give permission for their email address to be sold.  All other such lists that I have investigated do not confirm and/or fail to seek permission to sell the email addresses and/or hide the fact that the subscriber's email address will be sold in very obscure marketese/legalese.

This does not bode well for someone who wants to buy a list and not be a spammer.  They have an extremely low chance of actually getting a list that is confirmed and which truly has the informed consent of the subscriber to have the email address sold.  I could not responsibly advise *anybody* to buy a list at all under those circumstances.  It's even impossible to tell if a list which now has responsible practices had those practices when the signed up *all* of those email addresses.  If you take your chances and buy a list, and then get listed on a spam blocklist, well, you gambled and lost.  Me, I don't feel like gambling with my business.

Now, I personally don't think that WhatCounts needs to reconfirm all addresses as long as the customer brings proper confirmations with them, but WhatCounts should have access to those confirmations.  If a customer brings in an unconfirmed address or didn't keep the original confirmations, then WhatCounts definitely should send out confirmations for that list.

Mary Conner
Tuesday, November 19, 2002

"Also, all our agreements require customers of ours to insure that their lists are permission-based and be able to prove it."

But do you check this before the problem surfaces? Even at random? Is the requirement hidden in the fine print, or is it written in large letters so that no customer can miss it?  How likely do your customers think it is that they will be required to provide proof of permission?

Don't you think that waiting for the problem to surface (when it is easily avoidable) is akin to not checking a doctor's credentials (for example) before letting him practice medicine?

Don't get me wrong.  I think it's great that you have a monitor in place to attempt to pick up on abuse after the mail has been sent - but that isn't good enough!  After all, places like Yahoo are pretty good at cancelling spam accounts after the mail has gone out.  This doesn't exactly reduce the amount of spam we all receive, however, because the spammer simply moves on.  Once the email has been sent, it's really too late, and you are likely to end up on a black list for letting the spam happen (as in fact, you have - for repeat offenses)

Also, I do wonder a bit what your tolerance level is.  How many bounces and/or complaints does it take before you decide there is a potential problem?

"As for legitimate list providers - I'm no expert in this space. We often refer customers to groups like the DMA. "

What is your process for evaluating whether they are legitimate or not before you make the referral?  If you are not an expert in this space, perhaps you should not be making the referral...  In my experience, quite a number of the unsolicited email we receive comes from self-styled "legitimate list providers".  Usually prefaced with "you asked one of our partners for this" - an assertion that is completely false.  It is possible that somewhere in the privacy policy at one time someone somewhere buried a clause allowing them to sell the email address.  However, let me assure you that I (along with many) agree with the first poster to this thread in saying that if I share my email with Company A to get support (or even just to post thoughts on a message board) - that does not mean that I want to become part of a mailing list for Companies B-Z to learn about seminars, vacations and other ways to spend my money.  No matter what the privacy policy says.

And also in my experience, once on a "legitimate list" of that type - it is extremely difficult to get off!  Do you keep track of how easy it is to get off the lists belonging to the companies you refer your clients to?  Do you ask them where those addresses come from?  (As an aside, while attempting to clean up their act, at least in public, my personal experience with the DMA is five requests to remove and counting...)

If you don't know the answer to these questions, and can't find out - perhaps you shouldn't be make those referrals!

Really, I suspect many of your problems come down to a reluctance to cause any barriers to entry for your customers, because this could potentially lead to loss of sales (a position I can understand, btw). 

But isn't ending up on a black list worse than losing customers who refuse to prove their lists are permission-based? 

Just a thought
Tuesday, November 19, 2002

Plenty of newsgroups started up before SPAM became a problem. You would often find that didn't even get an e-mail saying that you had subscribed to them, let alone be expected to reply.

Anyway are all these holier than thou anti-spam vigilantes seriously suggesting that a mail server should employ people to read through each confirmation e-mail to check that explicit permission was given. I find it difficult to tell exactly what I have signed up for just for me - let alone for 100,000 other addresses. Microsoft and CNET know that because they regularly spam users who have subscribed to one newsletter with other newsletters they never subscribed to.

You would be surprised at the number of people who regulary give permission to receive e-mails from all the world and his dog. All you need to do is the same as Hotmail, that is check the "please receive occasional offers from our partners" box as the default.

Stephen Jones
Tuesday, November 19, 2002

Hello Stephen,

I'm a bit concerned about your terminology.

"Plenty of newsgroups started up before SPAM became a problem. You would often find that didn't even get an e-mail saying that you had subscribed to them, let alone be expected to reply."

You don't have get newsgroups via email (normally, although there are newsgroups-to-email gateways available, IIRC).  You subscribe to newsgroups via your newsreader and do not need to give an email address out at the time of your subscription to a particular newsgroup (if using a regular news feed).  You may, however, have to give out an email address when signing up for a news service, ala the old dejanews.com.


"Anyway are all these holier than thou anti-spam vigilantes seriously suggesting that a mail server should employ people to read through each confirmation e-mail to check that explicit permission was given."

I'd suggest finding a software solution to handle email confirmations rather than manual confirmations.  There should be solutions available which email a potential subscriber a confirmation email which cannot be forged (emails with a nonce[1] or other confirmation string in the subject/body come to mind).  When the email user replies to the confirmation email with the nonce/string intact in the message, confirmation is complete and the email user has confirmed that they want to receive the email.  That's the way confirmed opt-in email works.

Web bugs work, unfortunately, in a similar way by associating www.someURL.com/theImage.jpg?some_unique_string_2435 with an email address.  When the server logs show that www.someURL.com/theImage.jpg?some_unique_string_2435 has been requested, the server admins/software know that the email with that image/string has been viewed.


"I find it difficult to tell exactly what I have signed up for just for me - let alone for 100,000 other addresses."

This is not my problem.  I (think I) have a pretty good idea of what I signed up for.  If you really want to know where your mail is coming from, use disposable address services (like www.sneakemail.com/ or just Google for "disposable email addresses").


"Microsoft and CNET know that because they regularly spam users who have subscribed to one newsletter with other newsletters they never subscribed to."

Do you have proof of this, or are you just trying to throw company names around?


"All you need to do is the same as Hotmail, that is check the "please receive occasional offers from our partners" box as the default."

This is known as "opt-out" because the user does not actually click on the box enabling third-party email.  If they ignore (or miss) the checked box saying "spam me!", the user must opt out of receiving further email.

Bottom line is, opt-out is bad, confirmed opt-in (as per Peter's definition above[2] and other similar definitions) is the best responsible email marketing practice.

-Chris


[1] Nonce.  See http://searchsecurity.techtarget.com/sDefinition/0,,sid14_gci775421,00.html for a definition

[2] Confirmed Opt-in: "You should only need to opt in _once_, but (and this is the important part) it needs to be confirmed that it's you - the owner of the supplied email address - opting in. So a CONFIRMATION email should be sent to the given email address, requesting (surprise, surprise) confirmation that it was really you that "opted-in""

Chris
Tuesday, November 19, 2002

I have to believe that there are solutions that can work. I've found that my use of MSN Messenger has been completely free of annoying interuptions (hopefully that won't change).

What I suspect is needed for email, if we're to remain tied to an inherently difficult to control SMTP protocol, is a certificate-based model where emailing firms, like mine, could only send email to folks that joined via a difficult to forge certificate and that we sent back a similar cert with each message - allowing far greater (and more accurate) white list mgmt by the client. This doesn't solve the bandwidth glut problem - but it would help address the mess associated with umpteen different spam detection and cleanup programs/methods/models/etc.

Of course, anything remotely similar to this would require broad buy-in from legitimate emarketing/publishing firms, private companies doing their own emarketing/publishing and mail client developers. The last one being represented by a rather narrow group...MSFT, Kapor's new company set out to build a better Outlook - and the various clients built for Linux (Evolution, etc.). But, I fear that's a daunting task measured, in terms of adoption, in years.

There is some hope on the horizon and it's related to MSFT and TRUSTe's TrustedSender program. Strong certificate creation/mgmt and built-in Outlook support. Look for it soon.

David Geller
Tuesday, November 19, 2002

Sorry, I meant newsletters, not newsgroups.

Microsoft's speciality was to send you newsletters you hadn't subscribed to, and then claim when challenged that you must have opted in on their web site. I knew what MS newsletters I had subscribed to because I made a special point with MS of creating a new folder for each newletter. Whether they continue to do this I don't know as I unsubscribed from the lot. There was a big stink about it a couple of years ago, so I suspect they may have held back.

CNET are atrocious. When ZDnet started Smart Planet they took every ZDnet subscriber and started to send him Smart Planet offers and newsletters (to add insult to injury I could never buy any of the special offers anyway because I didn't have a credit card). It took three emails to the head of ZD Community to get myself removed from the list (the first two times even he couldn't get me removed). I still get special offers from CNET even thoiugh the only thing I am signed up for is Tech Republic, and I have steadfastly refused to ever submit a form with a box checked asking for extra stuff.

We can argue the terms opt-out and opt-in. When you have to check a box to receive anything is definitely opt-in, and when you have to check a box NOT to receive anything is definitely opt-out, but what about the most common case of late, where a box must be checked to opt-in to emails, but that box is checked by default for you? Opt-out in all but name, but the name is what we're talking about.

You've missed the point about checking for confirmed opt-in. The point, as David Geller said, is that many clients come with an exisiting customer base, often set up three or four years before. A couple of posters seemed to think the email sendimg company should ensure that each individual recipient had agreed to being sent the mail. Impossible to do without starting with the confirmed opt-in process all over again, which would irritate large numbers of users. After all neither the phone nor the mail company insist the caller/sender has your permission before they send the message.

I can't help but feeling that the quality of many anti-spam solutions is little different from the quality of the stuff advertised on the spam they are blocking.

Stephen Jones
Tuesday, November 19, 2002

" You've missed the point about checking for confirmed opt-in. The point, as David Geller said, is that many clients come with an exisiting customer base, often set up three or four years before. "

So what?  I think the point you've missed is that the minor inconvenience to those customers (having to reconfirm if they don't have proof that their list is legit) is far  less important than the major inconvenience to those same customers when the service gets blacklisted after another customer (or in this case, customers) doesn't have a legit list and sends out spam.  You could argue that the blacklist shouldn't exist in the first place, but frankly, I think that's a losing argument.  Why? Because  blacklists do exist and are getting more popular rather than less, as the volume of spam increases.  So, whether you agree with them or not, if you are a company that sends out messages in volume - it is your responsibility to your customers to ensure that you won't end up on a blacklist.  And a start would be to avoid the use of non-confirmed lists.

"Impossible to do without starting with the confirmed opt-in process all over again, which would irritate large numbers of users. "

I would challenge this assertion.  Especially if the end result was that it becomes more difficult to send out spam.  Especially if you last signed up 3 or 4 years ago.. Legitimate lists are not usually hopping from mail sender to mail sender after all.  So if a year or two later, you get a message saying:  "We're moving list providers and we don't have a record of your request to receive this newsletter on file.  Please confirm that you would like to keep receiving this newsletter." What's the big deal?  After all, you'd only need to send it to those users for whom you don't have proof of confirmation!

"After all neither the phone nor the mail company insist the caller/sender has your permission before they send the message."

Do you receive 100s of unsolicited phone calls a week?  It's a question of volume.  If people abused the phone and mail system to the degree they abuse email, it wouldn't surprise me if some of the same techniques were implemented. 

Furthermore, with physical mail, you can get the Post Office to quit delivering bulk mail and put up a no flyers etc sign on your mailbox.

With the phone, there is clear legislation to follow. Telemarketers are easier to trace and usually located in the same country as you are.  You can only contact as many people as you have staff to call.  Telemarketing is relatively expensive so the outfits calling tend to be more professional / legitimate (barring crooks).  (Email on the other hand, is open to anyone and their dog)

And both the phone and the physical mail system cost the sender exclusively, while the cost of email is distributed (and often approaches nothing for the sender)

So your observation, while true, does not automatically apply to email.

After rereading your comments, I'm actually not quite sure overall what you are trying to argue.  Let me sum up:

First you say that you get lots of spam, from newsletters that you haven't signed up for.  And you further argue that as a result you have no idea what you have or have not signed up for.  Then you say that the people sending out those newsletters shouldn't have to ask their users to reconfirm if the senders switch mail senders and can't prove their list is legit. You add that this is because "most" newsletters date from a time when spam wasn't such a problem, and that people sending phone or physical mail don't have to confirm they have permission first. And finally you conclude that you "think" anti-spam solutions are of dubious quality.

It seems to me that your arguments don't quite add up.

Do you use any anti-spam tactics at all?

Just a thought
Wednesday, November 20, 2002

I think my arguments are quite clear.

A mail service cannot reasonably be expected to keep a check on the relationship between the list sender and the client.

It's unreasonable to expect somebody with a mailing list to have to send off for a reconfirmed opt in every time he changes mail provider.

The indiscriminate blocking of whole sets of URL's and all mail sent from certain mail providers, is a straigtforward fraud to the customer who can't get his legit mail, and doesn't even know it is being deleted, and has much more to do with penny pinching by the ISP than any desire to provide a service to the customer,

Letting the message through the first time, with a message from the ISP to the client saying, "we think this is spam - unless you say any diffferent next time we will block mail from this address" would block all spam but that from the one off guys. If the ISP would co-operate with the customer, instead of trying to second guess what he wants, then the problem would be much closer to solution.

Stephen Jones
Wednesday, November 20, 2002

As stated, I disagree on your first two points.  If a mail service is sending bulk mail out on behalf of someone else, they'd better be sure that they aren't sending messages to addresses that have not requested the bulk mail.

Using your (flawed) comparison between the phone system and email - telemarketing firms are actually required (in many countries there is legislation) to ensure that people who don't wish to receive unsolicited phone calls, don't. 

People who send out mailing lists via mail (in my experience) usually have some basic validation in place to ensure that the mailing list is accurate - especially if they are sending out newsletters.  Mainly because they don't want to waste money, but still.

Given that checking that some one with a mailing list has a confirmed opt in for their addresses is not difficult in the email world, I see no reason why it is unreasonable to expect that the mail service do so, ESPECIALLY because if mail services continue to take your attitude, the medium will change for the worse. 

If there is anything that is clear, it's that if spam continues to increase at its current rate, either paid email or whitelist only or both are going to become the email of the future.  And that is going to cost the mail services far more than it would to check confirmed opt-ins.

"It's unreasonable to expect somebody with a mailing list to have to send off for a reconfirmed opt in every time he changes mail provider.

This is missing the point yet again.  People with the mailing list only have to send off for a reconfirmed opt in *if and only if they don't have the original reconfirmed opt-ins*.  And honestly, I don't see what is unreasonable about having to send off again.  It's not going to inconvenience the mailing list owner at all (in fact, it allows them to eliminate any recipients who have long since abandonned that mailbox - for instance - reducing costs).  Now, this is assuming that the mail service doesn't charge for reconfirming (which they shouldn't - since it is to their benefit to avoid sending unsolicited messages and as a result ending up on a blacklist).

Re: your rant about penny pinching by ISPs.
That's a very creative way at looking at things.  If it bothers you that much, why aren't you running your own mail server?  It's not that expensive!

Mail providers (who, btw, are not always ISPs) have to make choices in order to deal with the spam problem.  Personally, I don't blacklist domains except in very special circumstances, but as someone who follows the current spam/anti-spam techniques very closely - I certainly can see why some people do.  Quite frankly, it seems to be the single most effective method in getting mail services to clean up their act.

No, the blacklists aren't perfect.  But they exist and it's my opinion that screaming at your mail provider not to use them (unless you have some better mechanism) is not going to be effective.  Redirect your anger at those who ended up on the list - if they clean up their act - not only will the world be a better place - but their mail will get through to you again...

Incidentally, reducing spam is much more effective at the server level.  And yes, by handling it at the server level, it is possible to reduce the bandwidth required.  Given that you made a tradeoff to have someone else manage your email - it's a bit unreasonable to then say that you want entire control over spam filtering, unless you are willing to pay extra for the privilege.

Just a thought
Wednesday, November 20, 2002

Dear just a thought,
                              How are you going to check that the guy bringing the list has a set of confirmed opt-ins? if you can find a way fine, mail it off to David Geller.

                                  What about the lists that were set up two or three years ago when scarcely anybody used confirmed optins?

                                    What I find disturbing is that you and your ilk are all in favour of measures that help the administrator of the mail server, but any  benefit to the customer appears to be coincidental. A couple of genuine messages trashed for a hundred spam blocked may seem great to the guy running the mail server. It's not so great to the customer.                                                                                                                                                 

                                    Would you really defend your phone company blocking all calls from Los Angeles exchanges to you for example because some list had received reports that there were too many heavy breathing calls from that area.

                                      If there were some way to ensure that when I send a message I get to know if it's blocked or not, and also that the recipient gets to know that I've been sending him a message, then I would be more sympathetic to your case. Mind you when you get ISP's that block the whole of Saudi because everything goes through one proxy server and thus it obviously must be a mass mailer (as Hotmail did for a couple of days) then things get a little difficult. Let's just hope that unified IM and SMS messaging is in place!

Stephen Jones
Wednesday, November 20, 2002

"How are you going to check that the guy bringing the list has a set of confirmed opt-ins?"
Ask him to provide the emails



"What about the lists that were set up two or three years ago when scarcely anybody used confirmed optins?"

As stated above, in this case, send them out again.  A) It's not that big an inconvenience (it has been years, after all).  B) After a couple years, esp on lists set up w/o any confirmation, it's likely that many emails on that list are no longer monitored.


"What I find disturbing is that you and your ilk are all in favour of measures that help the administrator of the mail server, but any  benefit to the customer appears to be coincidental. "

I find it disturbing the assumptions you make about my "ilk" :)  Seriously though.  I got involved in learning about spam and its prevention after calculating that I was sending more time just deleting email than I was doing anything else (literally hours a day).  As a user, the volume of non-genuine email was incredible.  I often missed my important email because it got buried in the noise.  If anything like that ever happened on the phone system - I would disconnect my phone line.

We decided to do something about it.  Based on my research and experience, I believe spam filtering is most effective at the server level.  And sure, reducing bandwidth is one of the benefits.  But not just for the administrator, but also the users - since they end up paying for the bandwidth!  Yes, the user ends up with less control over how the filtering is done.  That's a trade off the user makes.  After all, unlike the phone system - there is an alternative, where the user can have full control!

My motivation is to reduce spam on a technical and social level, because I never want to receive the kind of volume I was getting again.  Talking to friends, clients, vendors etc - most have no clue about spam filtering (even that it is possible).  So they continue to receive spam, and complain to their administrators.  Who then implement something to stem the tide.

No, the system is not perfect.  In many ways, this is a new and evolvng field. As I stated above, I personally don't agree with blacklists, although I understand their motivation.  I further think that given their existence, administrators (and mass mailers) have a responsibility to do what they can to stay off the blacklists.  Incidentally, I also happen to agree with you that false positives are a big problem.  These tend to happen if your administrator has over-aggressive text-matching as a major part of their spam strategy (not the best way to handle spam, imho).  If that's the case, you should complain to your provider.  Give examples (mail message x got blocked when it shouldn't have.  It was legitimate because ...).  Don't rant at your provider about how they are committing "fraud".

"If there were some way to ensure that when I send a message I get to know if it's blocked or not, and also that the recipient gets to know that I've been sending him a message, then I would be more sympathetic to your case."

Try sending a message receipt. (Outlook has this feature built in, but if you use a different email client, you can simply ask for a confirmation).  If you don't receive confirmation, don't assume that it went through.  (Reminds me on those LM Montgomery books where the letter that never got delivered/sank on a u-boat etc was endlessly used as a plot device for ensuring misunderstandings)

Don't forget that email is not reliable!  It's become fashionable to blame filters on messages that don't get delivered - but in many cases of non-receipt the messages just don't get delivered due to network issues, gremlins etc. 

Cheers.

Just a thought
Thursday, November 21, 2002

Delivery confirmation doesn't work in most cases; either the recipient has turned it off for security reasons (aftar all the spammer can use it to know his message is geniune, and more importantly you want to be able to tell the boss his message didn't get through), or the software on the other end doesn't allow it. It works for Outlook to Outlook mail but not much else.

In my experience the most common cause for "missed mails" is that the guy at the other end couldn't be bothered to read them. The second cause is that you forgot to send them! Hotmail accounts that fill up and delete mails automatically whilst you are hysterically attempting to log on to them are the third cause.

To say you can't expect reliability from email is a self-fulfilling prophecy. In my experience messages nearly always get thtough, or you get a reasonable explanation if the network/server is down, in all cases except those where the mail server has decided to put some kind of restrictions in place.

And it certainly better start becoming reliable because world busniness is dependant on it! Just to bive you a simple example, in my field , which is teaching, the number of overseas job offers with a fax number was around 50% ten years ago, up to 90% five years ago. The percentage of job offers with an email address was 0% ten years ago, 5% five years ago, and 98% now.

Stephen Jones
Thursday, November 21, 2002

I know this will be a hot one but... what are the real economic impacts placed on the end-user vs. the ISP related to the receipt and mgmt of spam?

If Hotmail, Yahoo, AOL, et al. are willing to absorb storage and bandwidth costs (they appear willing), what are the costs the average end-user has to pay?

Clearly I'm not defending "spam", but I often hear that there's a huge cost from spam that has to be absorbed by the end-user. Is it really huge?

Personally, I pay a fixed rate for AT&T broadband at home. So, the cost, to me, for dealing with spam is my personal time. How do I place a value on that? There are no hard bandwidth costs I have to consider. Nor are there any significant storage costs. Most consumers don't pay a metered rate.

And, for everyone working for "someone else" and receiving spam at work.... is there really a justification to sue a spammer based on the cost of receiving that spam? Isn't it the corporation they work for that's absorbing *all* of those costs (storage, bandwidth, admin)? How often are these corporations willing to seek damages?

Frankly, I think the amount of paper spam (advertising circulars, credit card offers, AOL CDs, etc.) I receive at home is far more damaging. It's environmentally costly and it *does* impact what I pay for garbage/recycling collection. My guess is there are few hard parallels to email spam in terms of cost.

Also - I remain convinced that software will fix the spam problem within a few years. The current model of everything going into an "inbox" and then having to get filtered out is just poor, right now. I'm quite sure our mail clients will get smarter and spam will simply become noise we rarely see or have to deal with.

Asbestos suit on.

David Geller
Thursday, November 21, 2002

Stephen:

You could also have a manual confirmation process (as I noted).  Email, as implemented at present, is not reliable.  Period.

In order to get it to be reliable, it would need to be entirely redesigned (and many people think that it will come to that eventually).  In the meantime, it's dumb to pretend that it is reliable.  Fact is - if it's really important that the message get through, you'd better have some way of checking that it actually got through (be it sending a separate "confirm" email, phone or whatever).

I continue to be bemused by your logic though, because while you continue to argue that spam filtering is bad because you miss email - in your last message you turned around and stated that the three most common reasons for email not getting through are: "guy didn't read his mail", "you forgot to send" and "auto-delete due to full mailbox".

None of which have anything to do with spam filtering and certainly aren't arguments against it.  Two of them are social engineering problems, and the third could actually be greatly reduced by spam filtering.  So what are you actually trying to say?

David:

I agree that so far (in North America - don't forget that in other parts of the world metered usage is more common) the user has not had to bear the bulk of the cost for spam, keeping aside the question of time (and the problem of noise to valid email ratio).

However, I don't think that ISPs are going to absorb bandwidth costs indefinitely.  Certainly here in Canada most ISPs (particularly corporate ones) have bandwidth caps, over which you must pay.  The "free" email providers are slowly but surely introducing fees and restrictions on all of their accounts.  Hotmail, Yahoo and AOL can't front that bill forever, and this will become more true (not less) as the volume increases. Local ISPs such as Bell and Rogers are also working on introducing bandwidth caps.  I think it's a matter of time, especially given the current difficulties making a profit in that market.

You might pay a fixed rate for broadband - but is that rate increasing? (Mine has gone up ~25% in the last year, and that's without bandwidth caps) Certainly the cost of bandwidth impacts what you pay for broadband long term.  After all, bandwidth is a limited resource, and your provider has to have sufficient capacity.  Increasing capacity as demand increases is expensive and someone is going to pay for it. This is a kind of interesting point, because I don't pay for garbage collection directly (sounds like you do?).  The cost of garbage collection is thus not something that I think about much - it's hidden somewhere in the general coffers of the municipal budget.  Broadband, however, is in my face every month, and though I don't pay for junk email - I'd like to keep it that way!

With respect to the spam at work issue.  Being a small company - there is not a lot of difference between my "corporation" absorbing the cost and the workers absorbing it.  Every dollar I spend on stupid stuff like receiving email that I'm not even going to read - doesn't go to the employees.  But even if we were a giant corporation - I've never bought into the argument that just because "a corporation can afford it" suddenly it's okay to gett it to bear your costs (be it petty theft in terms of supplies or bandwidth). 

I can assure you that as a company we do what we can to reduce our volume at the source.  Granted, it's unlikely that we will sue anyone (not being American <g>) - in part because tracking down die-hard spammers is difficult and they are usually operating out of places difficult for us to reach.  Let me assure you that when we receive local spam, the president of that company hears about it promptly and is asked to cease and desist (Among other measures).

Incidentally, don't you think that it is environmentally costly to produce computers (and the electricity to run them?)  Paper spam is recyclable at least, and often biodegradable.  Adding new machines (which have a limited life cycle) in order to handle increasing spam volumes may not seem like pollution - but don't forget one major "benefit" of cars was the reduction of pollution (horse manure).  It's important to look at the entire life cycle of a product before deciding that one has less environmental impact than another!

I do agree with you that filtering at a client level is poor.  And I also agree that software will fix the spam problem (after all - although the volume of spam I receive is higher than I'd like - the actual stuff that gets through is very rare.  I think we're sitting at a 99+% accuracy rate right now.  And false positives have not been a problem, since we don't use aggressive phrase matching.)

But I'm not happy to just have spam as noise, because someone has to pay for the resources it uses - and it ain't going to be the spammers.  I'd rather use those resources for useful things. 

So it's equally important to me that sending junk mail becomes more difficult.  And if that means that people with legit mailing lists are minorly inconvenienced by having to reconfirm their lists when they switch providers - so be it.

I suspect that's much more desirable than changing the whole email mechanism to whitelist only or a per email fee (think of the pain that would be!)

Just a thought
Thursday, November 21, 2002

Dear Just a Thought,
                                  My point about the reliability of email is clear. Despite what you say email is a higly reliable method of communicaiton. It doesn't equal the 100% reliability of a voce telephone call but there are problems there with keeping an easily retrievable record of the communication, as well as the difficulty of getting through to the person you want in the first place. It is however more reliable than fax, since not only it is exceptionally common for faxes never to reach the person they were intended for but also many faxes, because of technicall problems, come though totally illegibly although the person at the other end doesn't realize there is a problem.

                      As a result of this, plus the low cost and retrievablity, email has become the defacto method of communication between and to businesses, particularly for international communciations.

                        It is perfectly true that you will often hear that your mail has not arrived, but as long as you addressed it correctly that is unlikely. After all do you consider snail mail unreliable because of the number of times you have heard that "the cheque is in the post", or that the world's supply of paper is in danger because "the dog ate my homework.

                          Your suggestion however that servers should filter out legitimate messages will change all of that.  The fact that you are happy with 99% accuracy is frightening. For the guy whose Spam to legit ratio was 45 to 1, that means you are junking nearly 50% of his legitimate mail!

                          I really disagree think your arguments about bandwidh are contrived. Most broadbandwidth  is taken up with downloading MP3's and porn,  and using a dialup connection I frequenlty find I have downloaded nine or ten magabytes of data in a session, and thiat is just taking part in discussion forums. If you wanted to get save bandwidth let the coike dealers out of thier cells and fill them with those whoe peddle Flash animations and animated gifs instead.

                               

Stephen Jones
Thursday, November 21, 2002

You are very entertaining :)

This conversation has been a bright spot in my week.  I'm never sure what assumptions you will leap to next...

Wrt reliability of various mediums.

A) The phone system is not 100% reliable.  It's not bad (in the first world).  But have you ever tried to contact people living overseas?  Dropped calls and terrible line signals are not as uncommon as you apparently believe! 

Furthermore, the phone system uses a completely different mechanism in order to deliver communications.  As such it is much easier to achieve reliability with phone communications than it would be with email. I'd recommend taking a telecommunications course if you are interested in this kind of thing.

B) I have personally never failed to receive a fax.  My experience with faxes (aside from hating the machines themselves due to them all possessing evil feeders <g>) has been pretty good.  Occasionally the faxes I send will not go through due to disconnections and poor line quality - but this is a function of the phone system (which according to you is 100% reliable...).  And in those cases I always know about it.  If the fax isn't legible (never happened to me) - then the other person presumably knows about it.  So I don't know exactly what you are trying to prove with this comparison.  Your argument that Phone is 100% reliable but email is more reliable than fax is complete nonsense, unless you are talking about faxes spewing out in an office and the addressee doesn't see it.  (The fax still got there...)

I agree with you, by the way, that emails are better than faxes - but for completely different reasons.  Email can send everything a fax can - except you can't easily filter the contents of a fax (Although I'm working on it...)

C) The physical mail system is also not bad - but it does occasionally lose things too.  I'm not talking about the "Cheque is in the mail" or the "I never got that (wink, wink)" - I'm talking about letters and packages that never show up for what ever reason, or that show up months or years later.  In my experience, when this occurs, it is usually when sending letters between continents - although I once sent a letter within my province (to myself actually) that never showed up.

D) Compared to the above three systems, however, email is  much less reliable.  And that's not even considering problems of filtering or overfull mail boxes being deleted.  I am responsible for several system that send notification emails (eg upon purchase) Occasionally I will get a report that an email has not been received from the system.  In quite a number of cases, I am able to trace the email from end to end - and in every case so far, non delivery was for technical reasons. Not filtering.

You on the other hand offer as proof of email reliability that "everyone uses it" (and your "feelings").  Just because something is widely used doesn't mean that it is 100% reliable.  That doesn't mean that one shouldn't use email.  I agree with you that the advantages outweigh the disadvantages. 

It's a stretch (Jumping to conclusions again) to say that filtering is obviously going to change email and make it less reliable.  As I've pointed out (and you've refused to believe) email is already not reliable at the technical level.  The ratio of noise to good mail (spam) and the problem of full mailboxes also means that it is pretty likely that legitimate email will be ignored (this is like the fax that gets sent to the right number, but picked up by the wrong person or ignored).  Many people already do not read more than the first line in their email (there have been studies on this) - because they get too many.  So reducing the noise level by filtering can only increase the probability that your communication sent via email will actually make it to the other person's brain :)

"Your suggestion however that servers should filter out legitimate messages will change all of that.  The fact that you are happy with 99% accuracy is frightening. For the guy whose Spam to legit ratio was 45 to 1, that means you are junking nearly 50% of his legitimate mail!"

Finally, (assumptions again!) I do not junk legitimate mail.  Period.  I can't remember the last false positive. As stated - they are not a problem for us - because we don't get them.    We do actually have someone read the spam filter entries (subjects only), so I am absolutely positive about this. The accuracy I was talking about referred to junk that got through.

And I believe I said "99+%".  That was me being lazy and not looking at the actual statistics.  In fact, in the last 1904 emails received, only two spam messages made it through the filter.  (0 legitimate messages were trapped).  And both spam messages could've be caught if they had been parsed slightly differently (but my spam catcher wizard hasn't gotten around to that yet). 

So yes, I'm happy with that. 

And I bet that your hypothetical "guy" with 45:1 spam ratio (yikes!)  would be happy with that too.  After all, I bet he doesn't read his email particularly carefully right now given that most of it is junk. If the junk was mostly filtered out, the legit messages would get more attention and suddenly - email would become more "reliable".

Nuff said.

Just a thought
Friday, November 22, 2002

Dear Just a Thought,
                                It would be quite nice if you stuck to commenting on what I am saying, not what you imagine.

                                I never said you didn't have dropped calls and I have spent nearly all my working life overseas so I know how difficutl it can be to get through. I have been known to spend up to two days simply trying to get a connection from Saudi to Sri Lanka, but the point I am trying to make is that once you have got through and are speaking to the person at the other end, you know tne info has got through. In that respect it's 100% reliable. To make it clearer where | am going think of SMS messages versus a mobile phone conversation. As the SMS message travels on the system part of the bandwidth you can often send a message even though the other mobile is disconnected; however you do not have the certiainty that it got through, whereas if you do manage to connect by phone you know who you are speaking to,

                    You may not have had dirty faxes but I get them all the time. Part of this is the question of dirty phone lines, and it can be exacerbated by the fact that people try and fax inapropiate color schemes. I would go so far as to say that 15-20% of the faxes I receive are seriously defective.

                      You yourself admit that email is reliable. I quote "OCCASIONALLY  I will get a report that an email has not been received from the system." Most important is the fact that you do get the feedback - as I mentioned before with SMS you don't unless you ask for it, and then confirmation is not always possible.

                      You have however let the cat out of the bag when you say that you have never had notification that a message has been returned because of a Spam filter, and this is my big beef. THEY NORMALLY DON'T TELL YOU. Now you are sending individual emails, and presumably your IP range is not suspicious, but as Joel and others have pointed out many of there newsletters have not got through AND NOBODY HAS BEEN ANY THE WISER.

                    You shouldn't use figures like 99% if you are simply talking off the top of your head. I took it to mean that !% of the mail you trapped was false-positive. You are now stating that you meant that only 1% of Spam got through, and there were no false positives. I am suspicous as to how you arrived at these figures, but I wish to state again my complaint THE ONLY THING I AM AGAINST IS LEGITIMATE MAIL NOT GETTING THROUGH AND NEITHER THE SENDER NOT RECIPIENT KNOWING ABOUT IT.

                        If you can succeed in testing out a spam filter system over three months, that stops say 50% of Spam and never gives a false positive, or that simply tags the mail but allows the recipeint to choose if he wants to download it, then I, like many others, will consider you a benefactor to the human race.

                      But do test very carefullly first. Block out mail sent to mutlitple recipients with the word "free" occuring a lot will block out an attempt to arrange or schedule an important meeting, with the consequent economic and personal disruption.

Stephen Jones
Friday, November 22, 2002

I didn't explain the SMS vocie message difference so clearly.

SMS was originally developed to help systems engineers troubleshoot. So it ran on part of the bandwidth that is reserved for the system.

Often you will find that the signal strength is not enough to allow connection for a voice call, or more often that all the avaliable bandwidth is taken up. You can still send an SMS message however.

This applies to GSM only I believe, but is a useful tip to keep in mind.

Stephen Jones
Friday, November 22, 2002

I urge everybody using spam filters and concerned about false positives to do their spam filtering AT SMTP TIME.

It is best explained by an example: spammer or open relay connects to your mail server, and starts the SMTP dialogue. Once you got the body of the mail, before acknowledging it, you pass it to your spam detection software. Hey, it's a spam. Your mail server either rejects the message right away, or better, hangs the connection.

What's so special about it? If the mail was falsely identified as spam, the sending agent generates a bounce that is very likely to reach the original sender, as the mail was legitimate. The sender will then realize his mail did not reach the recipient.

It does not solve the spam detection problem, however. And false positives are not great either, but with this easy mechanism you can afford some.

I have been using SA-Exim (Spamassassin and Exim) for over a year at home and for many months on a fairly loaded system and it works great. I do teergrubing (stalling the spammer's connection), and even with tens of stalled connections, the system is very responsive. See: http://marc.merlins.org/linux/exim/sa.html

Nicolas Marchildon
Friday, April 09, 2004

*  Recent Topics

*  Fog Creek Home