4 posts about ops

The Three Simplest and Most Effective Anti-Spam Hacks I Have Ever Seen

May 28 2008

Hack zero: Switch to Gmail This is not a joke: Gmail is a fantastic and nearly spam-free platform. Notably, you can hook it up with a custom domain name so no one knows you are part of the Goog machine like everyone else.

Hack one: Greylisting with Postfix on Ubuntu

A mail transfer agent using greylisting will "temporarily reject" any email from a sender it does not recognize. If the mail is legitimate, the originating server will most likely try again to send it later, at which time the destination will accept it. Wikipedia: Greylisting

Assuming that you have your own email server, greylisting is genius. Diabolically elegant, really. If you run an email server (or any server that can receive email) you are probably running the Postfix MTA, in which case their is a main configuration file appropriately named main.cf. A couple of edits to this file and you are on your way.

Here's how this setup looks (not my graph but I have definitely seen this happen on production mailservers):

The really brilliant thing about greylisting is that it it deals with spam way before it ever reaches your inbox, which is the only way to go (I don't use any spam filtering on my mailbox. That's too late, especially from a sysadmin perspective (think of the children cycles!).

Step one: install postgrey.

apt-get install postgrey

Two: edit your main.cf file.

sudo vi /etc/postfix/main.cf

Three: Then open it up and look for your smtpd_restrictions; add the following line:

checkpolicyservice inet:127.0.0.1:60000

Four: Reload Postfix

/etc/init.d/postfix reload

Hack 2: DNS Blocklists

This one is even easier, requiring only an extra line (for each blocklist). The blocklists are Just put it right there in that same block in main.cf. I typically use four of them. (Each has a slightly different purpose and tolerance. Check out the sites to get a flavor for why they exist.) This one is actually my favorite — it was created by the geek premier Paul Vixie and uses a DNS lookup for an extraordinarily light overhead.

Step One:

Open your main.cf file again and add these lines:

  
  reject_rbl_client list.dsbl.org,
  reject_rbl_client sbl.spamhaus.org,
  reject_rbl_client cbl.abuseat.org,
  reject_rbl_client dul.dnsbl.sorbs.net
  

Then reboot Postfix:

/etc/init.d/postfix reload

As with the example above you will also want to watch your mail log to make sure nothings gone wrong.

sudo tail -f /var/log/maillog

Hack 3: Keep Spammers out of Your Forms

This is really the ideal place to stop spam: before it happens. There are a bazillion ways to prove that someone is a human (CAPTCHAs ... sigh), but I think it is instead better to put the burden on the bots.

Step one: Add a hidden field to your form.

< textarea name="comment" class="hidden" >

Step two:

In your handler, ignore anybody that filled out that form (as robots will do). Here's a fragment in php (assumes that the presence of a errors array will prevent submissions):

if (!empty($_REQUEST['comment'])) { $errors[] = "No Spam please."; }

Those are my favorites, let me know if you have any others!

Happy Hosting

Feb 21 2008

If you need a professional application server, particularly for rails, I'm recommending Rimuhost these days.

Their VPS plans are a reasonable deal, but, as always, the support is what is always most important when choosing a host unless it's a super small static site, in which case the big guys like mediatemple, dreamhost, bluehost are fine. (Oh and check wordpress.com if you need a free site.)

Just now I've been getting really clear, friendly, (and super fast) support from Brandon at Rimuhost, and he showed me a great trick for removing a password from a ssl cert:

cd /usr/local/apache2/conf/ssl && mv sitename.com.key sitename.com.key.withpassword openssl rsa -in sitename.com.key.withpassword -out sitename.com.key

Previously one of their techs was chatting to me well into the evening about a SQL issue I was having, free of charge, after he got off of work, just cause he was cool. and I love their VPS control panel, which lets you reboot, power down, and upgrade your RAM/HDD on the fly.

If you need a development and production box, they will clone your production VPS (for free), in under an hour. Fastest demo box I've ever set up!

Who's linking to our website? New tools.

Feb 6 2007

6/25/07 UPDATE: I am obligated to point out that this little script has graduated from interesting to useless — thanks to the new Google Analytics, which is hands down the best tool for understanding web traffic. And it's bloody free. You probably knew this already. But, just in case, here's a great new tutorial. That is all.

It is a pretty basic trick to get an idea of people that are linking to your site. Just google: link:http://mysite.com

But that is an extremely rudimentary technique for several reasons.

  • You will probably get a bunch of internal links, which are pretty useless.
  • You will not get a sense of the total number of links from each referrer — they are not tallied or ranked.
  • You only get referrers for the individual page you type in, not your entire site. Which means that you are getting largely underreported numbers. (even http://www.yoursite.com is different from http://yoursite.com)
You asked, and we listened: We've extended our support for querying links to your site to much beyond the link: operator you might have used in the past. Now you can use webmaster tools to view a much larger sample of links to pages on your site that we found on the web. Unlike the link: operator, this data is much more comprehensive and can be classified, filtered, and downloaded. All you need to do is verify site ownership to see this information. Peeyush, Google Webmaster Central Blog
So yesterday I was super happy to discover via the trusty Google Webmaster Central Blog that there is a new "links view" in the Webmaster Toolkit.

The Webmaster Toolkit is a service from Google that you really should be using. It takes just a few minutes to get started and then you get lots of data, including the new link data. If you haven't already (and, uh, you run a website), check it out and you will be happy to pick up a bunch of free statistics about your site. Notably, you can also create an XML sitemap (not a graphical HTML sitemap, though!) of your site to make sure google is indexing the whole thing. And you can test your robots.txt file (important for keeping those pictures of the last drunken staff party out of images.google.com).

I did have a couple of problems with the data, though — there still is no way to get a good ranking of your referrers, or a ranking of your most popular pages. Luckily, you can download the entire file and do whatever you want with it. (hooray for openness!)

Since we have a bunch of clients I wanted to send this new data, I took the time to write a simple perl script. And I figured a few other people could use it.

It's here:

[unique_addresses.pl.txt]

Instructions for unique_addresses.pl

Prerequisites: Using this script requires that you know how to execute file from the command line (and that you have perl installed). This will only work for Mac/Linux folks (requires perl and the *nix commands for sorting). ... If you are a progressive blogger or organization and can't get this to work, email me your stats and I will process them for you.

  1. Download your entire external links file from the webmaster toolkit.
  2. Use Excel or something to pull out that column of external links, and save this as something like "referrers.txt"
  3. Repeat the above for your "pages" column, but name it something like "pages.txt"
  4. Download script.
  5. Make it excutable in the same directory as your "pages.txt" and "referrers.txt" files
  6. Run "./unique_addresses.pl" and it will prompt you through the rest.

Again, if you are working for a good cause but run into trouble, just email chris at blast dot com or leave a comment.