|
Spam filtering on servers
Let me rant for a minute.
I hate spam filtering on email servers. It may be that it can be properly, but I've yet to see it.
At our company, our IT first tried this about 6 months ago, but were forced to changed their plans when they heard enough complaints about missing emails. They instead just prepended "SPAM:" on subjects of suspicious emails. Fortunately, I have a Bayesian filter on my box, so I could basically ignore those labels (they were very inaccurate).
Just last week, they switched the server to a Bayesian filter. The problem though is that since it's at the server, they have no way of really knowing if they're improperly marking emails. The way it identifies "Ham" is by outgoing email. Well, that's a very flawed concept, as lots of email, like newsletters is uni-directional.
Also recently, my wife's law firm started down the same path with the server automatically junking email it didn't like. My wife had to work from home for a while because we had child care problems. We found out after many days that no one at work had gotten any of her emails explaining the situation, so they had no idea where she was. Needless to say, she was furious. In a less tolerant company, she could easily have been fired. What other emails did their (incompetent) IT person chuck?
How can these people possibly think that is a good idea? Talk about your cost of spam!
Whew. I feel better.
But don't get me started on automated stripping of HTML from emails...
David
Wednesday, February 25, 2004
My similar rant. Removing attachments blindly on the server.
Our IT department strips out ZIP files as attachments on incoming, outgoing, and internal email.
Given that we are a software company and often have to send programs to customers and accept data from customers, this is a PITA.
I've taken to renaming the zip files from foo.zip to foo_dotzip. And then explain to the customer why they have to jump through that extra hoop.
It's even worse when someone sends a zip file. Mostly I have to give instructions on how to put it on our FTP site.
argh.
pdq
Wednesday, February 25, 2004
Yep, that bugs me too. I'll send an email with an attached DLL to someone two cubes down and it'll never make it. You'd think they could discern internal emails and be a bit more lenient.
David
Wednesday, February 25, 2004
Silly rant, you've obviously never spent time as a Sys Admin.
Depolying individual spam filters to X desktops and then training users on it's operation is not very wise. A central solution lets IT people concentrate on other tasks than diagnosing extra problems from installing software to each and every desktop.
Also your anger is misplaced, it's not like the IT department created the SPAMing problem. They are just trying to do thier best to prevent Grow Your Penis emails from wasting your time (while wasting theirs).
Prefers Server Side Filtering
Wednesday, February 25, 2004
As a 'spam admin' where server-side spam detection is done, I have to say that I agree with most of the complaints. There are some things that we don't feel we have much choice but to strip (although we leave zips alone).
We get some users complaining about the fact that some legit mail gets caught in the filter, but the reality is that the owners of the company got sick of dealing with the spam themselves and forced us to put in server-side filtering. Now I spend 1-2 hours a day going through the trapped mail and forwarding the legit stuff. There is no way to keep everybody happy, so I just try to close my ears to complaints.
The biggest beef I have is the quality of the software we're using (not under my control). The Black/White lists that I create should be processed *before* any other filtering takes place. If that one little change could be made I would be able to cut my 'spam admin' time in half.
If I was in charge, everybody would be running SpamBayes in their Outlook and anybody who couldn't figure out 'delete as spam' or 'recover from spam' would be looking for a new job.
Ron Porter
Wednesday, February 25, 2004
OK, Mr Sys Admin, what's the cost of a false positive? What if it costs you a client because you never responded to their frantic email? What if it costs you an employee (see my paragraph above about my wife)? What about a potential sale?
I understand fully the impact of a mailbox full of a spam. It's a complete pain. I also understand that the salesforce whines incessently, because they're on the road and don't want to download the spam, before their client-side filtering can do its work.
That said, is blindly deleting email the answer? No. It's not. If there were a "good" solution on the server, I'd be cool. I'd have to know that it was a properly trained Bayesian filter with very lenient settings. Only throw away stuff that's 99% spam. That would catch the vast majority of what I get.
As for how to properly train at the server level, I'm not sure how to do that. Maybe let certain employees run client-side filtering and then combine their values. The server would need semi-frequent updates to keep it working. Seems doable to me. Anyone know of such a beast?
David
Wednesday, February 25, 2004
The company I work for has been testing, and is very close to, rolling out postini to all of our email users. postini is a service that receives your email, filters it, and sends the rest to your SMTP server.
The nice thing is that users can log into their website and find out what was filtered and have the false-positives delivered to their inbasket - users don't have to worry about losing valid email.
I was originally wary of it, but it does seem to be a good solution. It does look like the service is a bit costly, but considering how mush time the IS department was wasting on Spam it might be worth it.
Standard Notice: I don't work for, get paid by, or know anyone at postini - my company is just using their service.
RocketJeff
Wednesday, February 25, 2004
oops, forgot the url (for the 1% of you who couldn't figure it out).
http://www.postini.com/
RocketJeff
Wednesday, February 25, 2004
The cost of a false positive for us is potentially in the hundreds of thousands of dollars if not higher, but we couldn't deal with all the spam. So we slowly ramped in Spam Assassin on the server - we started with a score of 10.0 and manually reviewed every email that was flagged as spam.
Slowly we developed a white list (all user contacts from address books and anything improperly flagged as spam). Over time we ramped Spam Assassin to a score of 6.0, which is a pretty strict setting. We manually review anything between 8.0 and 6.0 and continue to add to our whitelist.
Additionally we run a bayesian filter on every client so they can further cull their email.
The biggest thing for us was manually reviewing all email within a certain threshold and creating a massive whitelist which circumvents the spam filtering on the server.
Admin
Wednesday, February 25, 2004
"What if it costs you a client because you never responded to their frantic email? " If it's such a f'ing emergency pick up the phone. Email is not all that reliable even without spam filtering.
I run NAV Gateway at our company. I love it. Spam filtering is best done at the server level. If you precious developers think otherwise, then take some of your free time walking from desktop to desktop and set up some filtering for the client if it's that important to you.
Sysadmins fix mistakes. Developers ship them.
Mike
Wednesday, February 25, 2004
Mike, you make me furious.
Sysadmins are a cost. They get in the way. They can be a necessary evil but add no intrinsic benfit.
Developers, testers, tech support, sales, etc are out there bringing revenue. All sysadmins do it tell you why you can't do what you need to do to serve the customer.
pdq
Wednesday, February 25, 2004
Ha hah, Baysian spam filters.
I emailed a president of a local networking services company, using the email address that was printed on his business card. I never got a response, and I did want to buy something he was offering, so I finally called him on the phone.
It turned out that the Baysian filter that his company uses on their incoming email rejected my manually-written, non generic subject lined message as "probable SPAM". He had to whitelist me.
So, here's someone basically in a sales position, very proud of their company's networking expertise, and you can't email the sales guy at his business card's printed email address.
Go figger. False positives can cost a sales person actual opportunities.
Bored Bystander
Wednesday, February 25, 2004
pdq,
I'm a developer, and I'd just like to say "Sit down and shut up." If you think you'd be able to develop for more than a couple of months without a syadmin staff, you're deluded. They might not contribute directly to the revenue stream, but without them you'll be contributing a lot less each month, because you'll be doing your own admin. Pretty quickly everything will go to hell in a handbasket with incompatible development environments on machines because of insistent system administration.
I've worked in shops with an admin staff, and everything tended to run pretty smoothly. I've worked in shops without them, and people tended to loose several days a month to admin issues. Working as a one man operation now, I can assure you that a sysadmin is worth their salt. I've lost contracts simple because I couldn't get my system whipped into shape in time to meet deadlines.
Clay Dowling
Wednesday, February 25, 2004
"All sysadmins do it tell you why you can't do what you need to do to serve the customer."
You mean like "no, you can't send an email with a 60mb attachment"
To be honest. Sales people drive the whole thing. Developers are a supporting role to them, and sysadmins to them.
Also a plus of being a sysadmin, our jobs aren't going to India in such droves.
Mike
Wednesday, February 25, 2004
It seems to me that attempting to block spam at the server (instead of filtering it at the client for example), plus blacklisting domains, causes more problems with email than it solves. I have never sent spam, but I have had my email blocked (blacklisted) for a period of time because someone else at the same hosting service was sending spam, I have had mail undelivered (with no notification) because of spam filtering or blacklisting at the company to whom I was attempting to send email, and I know the owner of a small business who blocks all mail from free email accounts such as Hotmail (and thus potentially misses sales opportunities). There have also been times when the server-based spam filtering software at my ISP or webhost has had problems and just dropped or discarded all emails either being sent or received for a few days at a time. Usually there is no recourse for any of these problems and no practical way to get off blacklists (especially private corporate blacklists). If your email address is spoofed as the (false) "from" email address for spam or virus emails, then it only gets worse for you.
For another opinion on this issue, see Dvorak's latest article at PC Magazine http://www.pcmag.com/article2/0,4149,1537408,00.asp where he complains that his personal email is now blacklisted by the entire att.net domain and he has been unable to find anyone at AT&T to correct it. He "did a Google search on this problem and found numerous complainers who had this happen. All of them said that they could find no mechanism to correct this ridiculous situation." (Yes, I know he is deliberately controversial, but he still highlights an important issue.)
Philip Dickerson
Wednesday, February 25, 2004
Coincidentally, Microsoft has just announced a proposal to reduce spam (Sender Authentication, or "Caller-ID" for email):
http://www.microsoft.com/presspass/press/2004/feb04/02-24RSAAntiSpamTechVisionPR.asp
http://www.microsoft.com/mscorp/twc/privacy/spam_callerid.mspx
"To be more effective in the fight against junk e-mail, filters need additional information that is not available in e-mail messages today. Microsoft believes some relatively simple but systemwide changes to the e-mail infrastructure are needed to provide greater certainty about the origin of an e-mail message [...]"
"It is based on three proposals to better enable effective filtering:
- Establish a verifiable identity in e-mail through a caller-ID approach
- Enable high-volume e-mail senders to demonstrate their compliance with reasonable e-mail policies
- Create viable alternatives for smaller-scale e-mail senders to distinguish themselves from spammers"
Microsoft's proposal has some additional mechanisms for verifying the source of email compared to "Sender Policy Framework" (SPF - http://spf.pobox.com/ ) which is also a proposed mechanism to reduce spam emails sent from spoofed addresses.
The CTO at Sendmail appears to be ready to support this Microsoft proposal in the Sendmail product: https://www.sendmail.com/smi/news/pressrelease.jsp?eventOID=80352&localId=USA
Philip Dickerson
Wednesday, February 25, 2004
>> I know the owner of a small business who blocks all mail from free email accounts such as Hotmail (and thus potentially misses sales opportunities). <<
10% of our clients use hotmail alone.
I would point-blank refuse a request to block hotmail from our PHBs.
Mediocre ASP Monkey
Thursday, February 26, 2004
The impression I get is that the policy tends to come from the PHB level based on the contents of their in-box on any given day. Which obviously doesn't lead to a coherent solution for anyone.
I would prefer to implement some sort of filtering at the server for two reasons - the first being trying to get everyone working on a local filter strikes me as being like trying to nail jelly to the ceiling, the second being that otherwise how to you cover general mailboxes?
As always the devil's in the details.
a cynic writes...
Thursday, February 26, 2004
Mike, your attitude is repulsive.
pdq, yours is almost as bad.
Both of you seem to be stereotypes of the bad side of your jobs.
Bad sysadmins tend to make tyrannical decisions that guarantee their control of the network and reduce any chance that they'll have extra work. I mean, the idea that you'd willingly chuck emails that might be extremely critical to the company just so your won't have to deal with Spam volume indicates that you have no idea why you are employed at all.
And to make the offhanded suggestion that all critical issues are handled with phone calls shows a profound ignorance of the real world. Email has become a Very critical form of both business and personal communication. People do and will continue to send critical messages this way. Phone calls don't generally leave paper trails, so lots of people PREFER to send critical messages via email.
Again, my main point was that bad filtering at the server level is throwing the baby out with the bathwater. Good and careful filtering at the server level, with practically NO chance for false positives, even at the expense of letting some spam through, is good. Unfortunately, it seems that few companies have such a system in place.
David
Thursday, February 26, 2004
What's PHB?
Stephen Jones
Thursday, February 26, 2004
"What's PHB? "
Pointy Haired Boss - a.k.a. the boss in the Dilbert cartoon.
PHB is now (almost) universal shorthand for a bad manager or bad management in general.
RocketJeff
Thursday, February 26, 2004
"Phone calls don't generally leave paper trails, so lots of people PREFER to send critical messages via email." Email doesn't always either. Ask Microsoft. One company I worked at previously purposely did not back up email. They were afraid of data migration. You know, data becoming evidence.
47% of the mail coming into my domain is spam and or viruses, what do you suggest I do. We do audit for false positives and haven't had any problems. Believe me, the users would rather have 20 less spams a day at the risk of losing an email.
To act like one dropped email is the end of the f'ing world is just stupid. Like I said pick up the phone, on a good day, you can't guarantee your mail was read by the recipient without them replying or you calling them.
Having policies and procedurs in place to reduce work and foster reliablity just make sense. Developers do this too, not just sysadmins.
Mike
Thursday, February 26, 2004
----"To act like one dropped email is the end of the f'ing world is just stupid. "-----
Might not be the end of the world but it can be the end of your company if the junked email happened to be a potential customer sending round for quotes.
Normally the dropped email is simply the end of one contract, or another three months trying to hire the right candidate because you junked the resumefrom the only candidate that fitted the qualifications.
----"Like I said pick up the phone, on a good day, you can't guarantee your mail was read by the recipient without them replying or you calling them."----
You don't have any idea, do you? It's quite possible that you and the person you are sending your emai to are never actually in the office at the same time. More importantly, your suggestion does nothing to stop the problems your company will have if your system junks email it should be receiving.
Or are you suggesting your company should put up disclaimers on the web site and all emails it sends saying "'to reduce work and foster reliablity just make sense', and as a result we can't guarantee anybody in our company will ever read the emails you send unless you take the time and expense of phoning the contact person you may not even know the name of to check."
Stephen Jones
Friday, February 27, 2004
What Stephen said.
There is no excuse ever for a company deleting a valid email once it gets inside their walls. None.
Mike mentioned that 47% of the emails that come in are spam or virii. That's actually probably lower than my company, or at least much lower than my personal percentage. That said, it's probably safe to can all the emails with virii. After that, you only have to deal with the spam. If you can't then manage to trim that at the server while guaranteeing no lost emails, then you need to put filters on the clients. In all likelihood, there are only a few client email addresses that get most of the spam anyway.
David
Friday, February 27, 2004
"You don't have any idea, do you? It's quite possible that you and the person you are sending your emai to are never actually in the office at the same time"
You sound like we drop every third legit message. A lot of the business world uses email, but does not consider it mission critical.
Mike
Monday, March 1, 2004
 Recent Topics
Fog Creek Home
| 
