The Old Joel on Software Forum: Part 4 (of 5) - Technical Solution Blocking Scam Mails At Source

Technical Solution Blocking Scam Mails At Source

Hi,

Many Nigerian Network administrators have a problem of getting their Ips blacklisted often because 419 scam mails are often sent through their networks, by people who are difficult to trace because cybercafes do not have records of their customers. Some attempt to cure the problem by heavyhandedly blocking outgoing SmTP in the entire network (sometimes an entire ISP) but most of our scammers use free email addresses anyway.

I and a network admin friend are considering a solution that involves scanning all outgoing HTTP POST requests for "scammy" patterns, and in cases where the message is detected by Bayesian methods to be very likely to be a spam, the post (usually e-mail) has to be approved before it is allowed to leave the network. It seems to obvious that I am surprised I have not come accross a solution before. Do you feel the idea is sound?

Regards,
Seun Osewa

Seun Osewa (afriguru.com)
Friday, March 26, 2004

Osewa, unfortunately Bayasian works best for individual profiles.

It works because it knows keywords that are relevent to your life, and keywords that are very much not. Some training is involved, and the profile you build for it to make smarter decisions in the next iteration of filtering is personalized to your inbox.

So when you are filtering out going email, you have the problem of building a "theoretical" profile of "all the users out there who could possibly receive email from you". And most bayasian tools don't work with this.

Another thing that look so promising, but ofcourse won't work, is digests. There are reservoirs of registeries containing nothing more than MD5 hashes (ordered in a look up table) that you can hit. All you have to do is filter all outgoing messages, take each sentence in the message, do an MD5, and compare this MD5 with the registries, if it matches, it says the likelihood of this sentence existing in a previous sent email (widely accepted by the world as spam) is also in your current email. But spammers have been getting around this by intentionally doping sentences with completely random typose (on an individual message basis)... to chase thsi particular arms race would require very very large reservoirs and too many hits to do unique look ups.

The nicest thing you can do is add a neglegible cost to your emails. And flag some of the more heavy users. If you have the fortune of hosting your own webmail, add captha challenges.

Li-fan Chen
Friday, March 26, 2004

Do you realize how much time it will take? Have you thought of the legal implications? How many customers will sign up to use an ISP that will read all their emails?

Stephen Jones
Friday, March 26, 2004

Incidentally only about one in a thousand 419's get through the Bayesain filters.

The real problem isn't Nigerian scam, it's US spam of the type that an apologist for the scumbags on another thread is all in favour of.

Stephen Jones
Friday, March 26, 2004

Don't mean to shoot you down but you can't filter SSL HTTPds going to Hotmail and Yahoo mail and the like. What you can filter for is really heavy traffic to these sites on a SOURCE IP basis.

If you host non-SSL webmails that's another story, your idea to snoop WOULD work.

Let me show you another trick that may help. Provided most of the junk emails are HTML only, and you are the relaying SMTP server, try this trick:

Add a shim image (non-visible, zero length) to all out going emails. The shim goes to a URL you host. The asp server you are hosting answers these shims by 1) provided the gif file and 2) logging the url encoded code. The code should be the email account's random unique id you use to track accounts.

Whenever people open one of these emails, you will get a hit on your asp web server. Whenever users delete them or ignore them or choose to not open them with images loaded, you'll get no hits.

And then at the end of the month you do a simple count:

1. Who sent tons of email?
2. Of those who sent tons of email, are they usually getting responses in the form of email opens?

By determining this you can pin-point troublemakers pretty quickly. But mind you this assumes spammers aren't smart enough to ditch an account after a month.

If the system is properly designed, you can narrow the monitoring report to weekly or even twice weekly.

Li-fan Chen
Friday, March 26, 2004

For more details try this explanation: http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=32054

Li-fan Chen
Friday, March 26, 2004

>Osewa, unfortunately Bayasian works best for individual profiles. <Chen>
These scam mails are so similar. For a large number of people I probably can't use these tools to determine that probability that the mail is _not_ a scam mail, but I can use it to determine the probability that the mail _is_ scam. The scammers attempt very little customization of the mails, (for now).

>Do you realize how much time it will take? Have you thought of the legal implications? How many customers will sign up to use an ISP that will read all their emails? <Stephen>
The problem is that they don't "sign up". They go to cybercafes to browse. They cannot afford phone lines much less dial-up access, so we all use cyber-cafes. And there are no records of who uses which system in a cybercafe. So unless mails are intercepted in real time, by the time the complaint comes around the criminal is in a different cyber-cafe!

>Don't mean to shoot you down but you can't filter SSL HTTPds going to Hotmail and Yahoo mail and the like. What you can filter for is really heavy traffic to these sites on a SOURCE IP basis.
As far as I know Yahoo and Hotmail only use SSL for authentication (yahoo uses it for authentication only if you spoecifically request for it). Please educate me further if I am wrong.

I'll now look into your solution, but bear in mind that there is no user accounting whatsoever being used in these cafes because they, or we, are afraid if we start asking for people's names or addresses they will be put off. they just get vouchers which enable them to browse for specific periods.

Regards,
Seun Osewa

Seun Osewa (afriguru.com)
Friday, March 26, 2004

Well, Osewa, at the moment very organization has had to pick and choose form the many anti-spam tactics to use for their special situation. You keep pointing out the cybercafes as been the source of anonymous spam and you are also explicitly saying that you cannot change the business decision to make it less anonymous, so you are reduced to using less draconian--and potentially less effective--methods of solving the problem. Good luck anyways :-)

Li-fan Chen
Friday, March 26, 2004

The major problem (from where I stand): To you or the cyber cafe owner there are no monetary incentive to solve this problem. The minute you can determine a significant cost saving or profit increase caused by an anti-spam measure, is the minute you can sell this to a business-minded cyber cafe owner.

Li-fan Chen
Friday, March 26, 2004

First, re: SSL. If you control the client machine, you can read SSL traffic. There is a commercial app whose name I've forgotten that lets you do this. It essentially mounts a man-in-the-middle attack on the SSL session, but it works because you've added there key to your browser. I've forgotten the exact details, but it's definitely a technical possibility. Much harder than sniffing clear traffic.

Second, you have to realize that your filter is never going to learn from its mistakes, otherwise the scammers could poison it. Therefore, some posts will be wrongly blocked, and the user will have no recourse.

Brian
Friday, March 26, 2004

Thanks for the tip, Brian.

The value proposition is simple: For networks currently on various e-mail and e-commerce blacklists all over the web, a solution such as this helps them to get off those blacklists!

Seun Osewa (afriguru.com)
Friday, March 26, 2004

I doubt if you'll get off the ecommerce blackliists.

Most email blacklists are the result of open relays. Check those first and then start writing to the main ones. Persistence will probably pay more than technology here.

Most Nigerian 419's come through Hotmail or Yahoo, The Nigerian ISP will not enter into it.

Stephen Jones
Saturday, March 27, 2004

Ok, I am sure there are open SMTP relay scanners freely available, so I'll publicise them on my site.

With mail sent with free e-mail providers, it is very easy to determine the originating IP, but I'll have to go and confirm if the originating ISPs are still blamed.

Seun Osewa (afriguru.com)
Saturday, March 27, 2004

Seun,

1) One VERY SIMPLE way to blocking Scam emails is to lock down your desktop so that users do not install programs. 95% of the 419 guys use freely available programs to mass mail the 419 emails. Lock your desktop and most of your problems are solved.

2) Second technical solution is proprietary and is something I have thought about for the past 2 years. Contact me privately and maybe we can go into business together.

FN
Saturday, March 27, 2004

how about something like this for controlling junk mail: http://www.rfxtech.com/index.html

ITDirector
Saturday, March 27, 2004

The problem I was trying to solve was "what do I do as a network admin whose network is being used for such activities, and whose ip addresses are getting on various blacklists, assuming I have no cooperation from the free e-mail providers?"

And the answer would seem to be "use technology thatke it very difficult for frausdsters to not leave a trail". In other words, detect and intercept the fraud while the fraudster is still at the cyber-cafe terminal. Is it possible that this is less significant than I am making it seem? I wonder, afterall my particular ISP seems to have never found its way to e-mail blacklists.

Seun Osewa (afriguru.com)
Sunday, March 28, 2004

And then I think I also ignored the fact that replies to these mails have to be *recieved* so be of any use to the scammers, so that might be another avenue for blocking. And at that point seems like a much simpler problem to handle. Because when you broadcast such mails to 1000 people, one of them will be smart enough to report the mail, _before_ you get to recieve any replies.

Seun Osewa (afriguru.com)
Sunday, March 28, 2004

But the replies will be going to the Hotmail accounts over which you have no control.

Stephen Jones
Monday, March 29, 2004

It is very easy for a snooping proxy server to determine what hotmail/yahoo account you have open. So, participating cybercafes can be on alert for these people. Once they log in ... ;-) Its more of a social arrangement than a technical solution though

Seun Osewa (afriguru.com)
Monday, March 29, 2004

Spamming is dificult to prevent. People can always come up with random email addresses which most of the time really exist. And with free email services these scam mails can easily be sent.

Feasible solutions can only be obtained by implementing filters at the SMTP level. Most of this scam mails have a regular pattern. Therefore you can use tools like awk (assuming smtp runs on a Unix variant) to search for those patterns in a mail. Its similar to what Antivirus software does. They keep a list (and sometimes metadata) of known viruses and prevent them from being transferred to your computer. To apply the same technique would involve continuos updates to your pattern file to match the new pattern of this scam mail.

Another option (not a really good one) is to always ping back the domain an email is recieved from by the smtp server. This only addresses those that masqurade their email addresses for fear of being caught. However, not all servers return ping messages.

I must say this website is really growing. I have been lurking around for a while and i must say congrats to you Seun.

Hakeem Ogunleye
Wednesday, March 31, 2004

Article on Bayesian and Nigerian 419

http://www.nigeriavillagesquare1.com/Articles/femi_oyesanya1.html

femi oyesanya
Friday, May 14, 2004

Man in the middile can be prevented by adding digital signatures to the encryted mail during session key exchange. It is not impossible for a 419'ner to encrypt key exchange and then forward a public key. Aslo, there could be undelying privacy issues when you begin to hijack keys
in a man in the middle attack.

femi oyesanya
Friday, May 14, 2004

I seem to get a dozen or so 419 emails a week. I used a script that I customized, and I never hear back!

Found the script here:

http://www.netscientia.com/nigerian_email_scam

Prepare to laugh!

Jack Harris
Friday, May 28, 2004