6 posts about evaluation
6/25/07 UPDATE: I am obligated to point out that this little script has graduated from interesting to useless — thanks to the new Google Analytics, which is hands down the best tool for understanding web traffic. And it's bloody free. You probably knew this already. But, just in case, here's a great new tutorial. That is all.
It is a pretty basic trick to get an idea of people that are linking to your site. Just google:
But that is an extremely rudimentary technique for several reasons.
- You will probably get a bunch of internal links, which are pretty useless.
- You will not get a sense of the total number of links from each referrer — they are not tallied or ranked.
- You only get referrers for the individual page you type in, not your entire site. Which means that you are getting largely underreported numbers. (even http://www.yoursite.com is different from http://yoursite.com)
You asked, and we listened: We've extended our support for querying links to your site to much beyond the link: operator you might have used in the past. Now you can use webmaster tools to view a much larger sample of links to pages on your site that we found on the web. Unlike the link: operator, this data is much more comprehensive and can be classified, filtered, and downloaded. All you need to do is verify site ownership to see this information. Peeyush, Google Webmaster Central Blog
So yesterday I was super happy to discover via the trusty Google Webmaster Central Blog that there is a new "links view" in the Webmaster Toolkit.
The Webmaster Toolkit is a service from Google that you really should be using. It takes just a few minutes to get started and then you get lots of data, including the new link data. If you haven't already (and, uh, you run a website), check it out and you will be happy to pick up a bunch of free statistics about your site. Notably, you can also create an XML sitemap (not a graphical HTML sitemap, though!) of your site to make sure google is indexing the whole thing. And you can test your robots.txt file (important for keeping those pictures of the last drunken staff party out of images.google.com).
I did have a couple of problems with the data, though — there still is no way to get a good ranking of your referrers, or a ranking of your most popular pages. Luckily, you can download the entire file and do whatever you want with it. (hooray for openness!)
Since we have a bunch of clients I wanted to send this new data, I took the time to write a simple perl script. And I figured a few other people could use it.
Instructions for unique_addresses.pl
Prerequisites: Using this script requires that you know how to execute file from the command line (and that you have perl installed). This will only work for Mac/Linux folks (requires perl and the *nix commands for sorting). ... If you are a progressive blogger or organization and can't get this to work, email me your stats and I will process them for you.
- Download your entire external links file from the webmaster toolkit.
- Use Excel or something to pull out that column of external links, and save this as something like "referrers.txt"
- Repeat the above for your "pages" column, but name it something like "pages.txt"
- Download script.
- Make it excutable in the same directory as your "pages.txt" and "referrers.txt" files
- Run "./unique_addresses.pl" and it will prompt you through the rest.
Again, if you are working for a good cause but run into trouble, just email chris at blast dot com or leave a comment.
37 Signals is a super-smart little company known for creating easy-to-use web-based project management tools (namely the Basecamp suite), and they have just announced the latest in their product family: Campfire.
According to their website, "Campfire brings simple group chat to the business setting. Instant messaging is great for quick 1-on-1 chats, but it's miserable for 3 or 4 or 7 or 15+ people at once. Campfire solves that problem and plenty more. "
A little background: By day I work at a university with a nonprofit evaluation team. We work for other nonprofits that are trying (...or are being forced to by their funders) to discover and amplify the best aspects of their program. A major part of our job is finding out what people think works well — so we are typically creating surveys and conducting focus groups or our clients.
Focus groups are a great way to bring together a lot of good people and get a lot of good advice about a program. Among all the fun things we do with nonprofits, they're my favorite. But in-person focus groups are expensive and difficult to arrange, especially when you are working with busy people. Who has time to sit around and talk about making a program better? You've too busy working on the program, dammit!
This problem is what first attracted me to the idea of conducting focus groups online. At the American Evaluation Association's conference last year in Toronto, I was fortunate to sit in on a session that discussed the existing methods for doing this type of work. All of the evaluators in the room seemed to really get excited about it: online focus groups solve so many of of the distance/time/money problems evaluators face, especially in the nonprofit sector. Some people have already been doing this with telephone surveys, but in many cases (depending on your phone company and country) this can be really expensive — and it's unlikely to give you a transcript.
And, unfortunately for most of us at the evaluator's conference, the presenters were from the for-profit world, and their solutions were ridiculously expensive — about $1500 per focus group administration. These are basically managed chat rooms in which invited participants are egged on with questions from a well-trained facilitator. This is why online focus groups have been largely a marketing research arena — big pharma et al don't need no stinkin' open source solution.
After the session I raised the issue of reconfiguring WordPress or a more robust CMS (Joomla or Drupal) with a chat feature. If you could just manage to get all of your participants together online at the same time (or over a 3-day period), you could easily solicit lots of feedback. (And you wouldn't have to transcribe the results). Folks (including focus group guru Richard Kreuger) were excited about the possibility of finding a cheap way to do this: there is a serious need for something way below the $1500 range.
Unfortunately, in my research, I that most server-side free/open source chat software is buggy, poorly designed, and horrible with server resources. I found that people were using hosted solutions like Yahoo discussion groups in a pinch, and having limited success. So, not really able or willing to create a program from scratch, I put the project on hold. The recent release of Campfire brings a number of awesome features to the under-$20 table. Here are my notes on the best features for focus groups with Campfire:
- It's simple, clean, easy, elegant (all synonyms for "37 signals product")
- Real time image previewing (for getting reactions, for example, to a chart)
- You get a unique url for your chat, unlike an IM chat, so you can invite people to a web page
- It tracks previous conversations
- It keeps a record of who said what when (the timestamps are just killer)
- You can have secure chats (essential, I would think, for IRB approval ...)
- No software other than a web browser is needed.
- Repeat: automatic transcriptions!
So I'm really excited about Campfire and I look forward to trying it out, hopefully with a longer post in the near future about the existing options for online research of this nature. Until then, check it out for yourself if you need to do some group instant messaging, I think the possibilities are amazing if you just have a talented coordinator and some people that have valuable things to say.
Educational Testing Services — they're the folks that make the SAT and GRE — has a new test for ICT literacy. Despite a humorously useless Flash intro, the test appears to be fully baked. The sample questions on the ETS website look at the ways a test-taker would represent and evaluate information in an online context.
Test taking is boring. It's a boring subject. But you can't direct change in any environment without having some feedback. So this investment in ICT literacy evaluation is good to see.
Give them a visit:
ETS ICT Literacy Assessment Tests Information and Communication Technology Proficiency and Computer Skills
Evaluation is an important, albeit rarefied, science of promoting nonprofit organizations. Do you need to measure the effectiveness of a specific program — or your entire organization? Well, there's an entire discipline devoted to helping you do just that.
Unfortunately, as with most rarefied, important sciences, the "discipline" part tends to mean something more like "punishment," rather than "a codified mode of study."
At any rate, doing a real, formal evaluation of a program is still better than wasting your time doing something that's useless, fruitless, a waste of energy or just a sham. So here's a buch of resources from the internet to help you get going for free. Thanks for this post go to Joyce B. Morris and David Colton, Ph.D., from the EvalTALK Listserv.
The Program Manager's Guide to Evaluation
Taking Stock: A Practical Guide to Evaluating Your Own Programs
Understanding Evaluation: The Way to Better Prevention Programs
The 2002 User-Friendly Handbook for Project Evaluation
User-Friendly Handbook for Mixed Method Evaluations
Tools for Planning, Implementation, and Evaluation
"Also, there are some good, free resources on the Internet and from state and federal government agencies. For example, you can order "Introduction to Program Evaluation for Comprehensive Tobacco Control Programs" (2001) from the CDC (www.cdc.gov/tobacco). Don't be turned off by the reference to tobacco control as that is used for context. The "W.K. Kellogg Foundation Evaluation Handbook" can be ordered from the foundation or is available as a free download in pdf format
When Nonprofit X goes to county Y and begins handing out seeds and fertilizer to farmers as part of an agricultural intervention during a famine, how does Nonprofit X know that they aren't causing a greater problem or ignoring a better solution? Perhaps it turns out the fertilizer is more valuable when it is sold on the nearby market than it is in the ground. Perhaps microloans or digging new wells are better solutions that the population is looking for.
At any rate, even in this very simple hypothetical situation, it is clear that you have to go there. You have to ask and investigate. You should, in short, behave in a manner very similar to that of a market researcher. In fact, if we undertook all of our development decisions with a rigorous marketing mindset, we'd be going in far fewer circles with misused money.
This is the gist of a thought-provoking article from the Harvard Business School on the concept of marketing being used in human development situations involving poverty. I think it is especially pertinent for folks involved with technology and communications because of the heavy (though often overlooked) influence of advertising and marketing in all media.
I would call this a process of evaluation, but the "marketing" concept works well.
In many poverty-reduction programs, be they governmental, philanthropic or academic, the decision making about method — how to alleviate poverty — comes from the top down. This makes sense in many ways, and I'm certainly thankful for the developed world's research institutions, mulling and muttering in their thinktanks about how to best deal with poor farmers and streetworkers around the world.
But programs have to take all of their stakeholders into account when it comes to crafting policy — and that should mean subjecting proposed solutions to the "market" of consumers — the population in poverty. In traditional markets, this process is essential, because bad ideas don't make money and are necessarily reformed or removed from the market. But in many governmental and philanthropic situations, bad ideas can thrive.
This is a concept of including the population you are serving the in the program decisions that you are making. It's that simple.
Here's a bit:
"We're making the important distinction of replacing the exchange paradigm with an intervention paradigm, where we say we're intervening to change lives, not to change consumer choices.
... Four billion people are outside the exchange network. But it's not as though these folks have nothing to do with marketing. Somebody is approaching them. Somebody might approach a poor subsistence farmer in Uganda to say, 'We want to give you some agricultural aid. We want to give you a loan.'
There again, why is advocacy important? Because even in that case, what we find is that they have already decided what's good for the farmer. They already decided that what's good for the farmer is these kinds of implements, these kinds of equipment, this kind of loan. In fact, the farmer may say, 'Given everything else, that's not exactly the kind of output that is going to enhance my life. I want security and food for the family first, before I become this terrific entrepreneur.'"
Read the entire article at the Harvard Business School website.
Here's a chart of evaluation software packages form the Evaluator's exchange. The list is untested by yours truly, but the Evaluation Exchange (From the Harvard Family Research Project) is an excellent resource for folks working with children.
Here's a bit from the article, the chart is about a 60K PDF.
A wide variety of software packages are available to help nonprofit organizations track program management data and outcome measures for evaluation. One advantage of an "off-the-shelf" package is that these packages are often "tried and true." Another is that the software makers often supply training and technical assistance to users. On the other hand, organizations should carefully consider whether the available packages are appropriate for their needs. In some cases, organizations may find it more useful to design their own tracking systems, although the process can be time-consuming and may require extensive technical training.
The Evaluation Exchange Harnessing Technology for Evaluation: Promising Practices - at the Harvard Family Research Project (HFRP)