Unit Testing
Lamson provides the lamson.testing to help with writing unit tests. It doesn’t do everything for you, but does enough that you can TDD your email interactions with pretend users. It includes features for checking messages get delivered to queues, checking spelling, and running things through a fake or real relay.The Log Server
The first thing you need to do for testing is to run the “log server”:
$ lamson log
The log server acts as the smart relay host you’ve configured in your config/settings.py
file by default. What it does is take all emails that your Lamson application
“sends out” and redeposits them into run/queue
. This queue directory is then also
used by lamson.testing
for checking that emails were sent out. When anything goes
wrong, you can look in this directory and see what is getting sent out.
An alternative to this setup would be to have all of the sending/routing/relaying done internal to the whole testing framework, similar to how
lamson.testing.RelayConversation
works. However, I found that this made testing that your server actually sends proper emails much too difficult.
Test Organization
Lamson organizes tests into directories that match the things you’re testing:
librelist $ ls -l tests/ total 16 drwxr-xr-x 6 zedshaw staff 204 Aug 19 17:01 handlers drwxr-xr-x 8 zedshaw staff 272 Aug 18 15:50 model drwxr-xr-x 3 zedshaw staff 102 Aug 18 15:50 templates
In most of the projects I’ve only rarely used template tests, but I’ll cover
them below. Model tests make sure that any app.model
classes work right, and
handler tests make sure that any app.handler
classes work. Nice and simple.
You can add any other directories you want to this, and you can also use doctests if you want. This comes free with nosetests and is very handy.
Handler Tests
We’ll use an example from the librelist.com code base that validates that a user can subscribe, unsubscribe, and post a message to a mailing list. This test was primarily written in a TDD style, since generally interactions and usability testing works better TDD style.
First you have a common preamble of modules that you need to include:
from nose.tools import * from lamson.testing import * from config import settings import time from app.model import archive, confirmation
Right away you notice we just include everything from nose.tools
and
lamson.testing
so that we can use it directly. Yes, this violates Python
style guidelines, but practicality is more important than dogmatic slavery to
supposed standards.
After that we include the config.settings
, and two modules from app.model
that we’ll
use to check that everything was working.
Notice that we don’t include anything from
app.handlers
directly. These tests are meant to be from the perspective of a user interacting with the handler via emails.
Once we have that we do a little setup to clear set some common variables and clear out some queues we’ll need to check:
queue_path = archive.store_path('test.list', 'queue') sender = "sender-%s@sender.com" % time.time() host = "librelist.com" list_name = "test.list" list_addr = "test.list@%s" % host client = RouterConversation(sender, 'Admin Tests') def setup(): clear_queue("run/posts") clear_queue("run/spam")
Most of these are just variables used in tests later, but the big one is the
client
variable. It’s a
lamson.testing.RouterConversation
class that lets you simulate delivering email to your Lamson project.
There’s also a lamson.testing.TestConversation class that actually uses your real
config/settings.py
to connect to a Relay. This isn’t used so much, but is intended for running “smoke tests” against a newly deployed server.
With that we’re ready to write out first handler test:
def test_new_user_subscribes_with_invalid_name(): client.begin() client.say('test-list@%s' % host, "I can't read!", 'noreply') client.say('test=list@%s' % host, "I can't read!", 'noreply') clear_queue() client.say('unbounce@%s' % host, "I have two email addresses!") assert not delivered('noreply') assert not delivered('unbounce') client.say('noreply@%s' % host, "Dumb dumb.") assert not delivered('noreply')
This is the longest of the tests, and shows all the various things you can do
with the lamson.testing
gear. Here’s what we’re doing in order:
- Call
client.begin
to clear out queues and state and start fresh. - Use
client.say
to send an email from that client to your Lamson application. Notice that you configured the RelayConversation to pretend to be one person with each email getting the same subject line. - Use
lamson.testing.clear_queue
when you want to make sure the queue is clean. - Use
lamson.testing.delivered
to check if a certain message from someone is in the queue.
With that you can do pretty much everything you need to send an email and make sure you get proper replies.
Here’s another example:
def test_new_user_subscribes(): client.begin() msg = client.say(list_addr, "Hey I was wondering how to fix this?", list_name + '-confirm') client.say(msg['Reply-To'], 'Confirmed I am.', 'noreply') clear_queue()
Notice in this example we have a fourth parameter list_name + '-confirm'
and
we get a msg
back from our call to client.say
. This basically combines
client.say
with delivered
to do it in one shot. Very commonly, you’ll want
to say something to your server and make sure you got a certain response, and
then do something with that response. This is how you do that.
We then use this '-confirm’ email message to actually subscribe the fake user.
Finally, here’s two more examples:
def test_existing_user_unsubscribes(): test_new_user_subscribes() msg = client.say(list_name + "-unsubscribe@%s" % host, "I would like to unsubscribe.", 'confirm') client.say(msg['Reply-To'], 'Confirmed yes I want out.', 'noreply') def test_existing_user_posts_message(): test_new_user_subscribes() msg = client.say(list_addr, "Howdy folks, I was wondering what this is?", list_addr) # make sure it gets archived assert delivered(list_addr, to_queue=queue(queue_path))
In test_existing_user_unsubscribes
what we do is call
test_new_user_subscribes
to go through that process again, and then we chain
off that to do an unsubscribe. There’s really nothing new here other than that
little trick.
In test_existing_user_posts_message
we do the usual send a message and expect
a reply, but then we also make sure that this message was delivered to the
archiver queue.
Apart from those methods and techniques, there’s really nothing more to doing a handler test. The only additional thing would be using assert_in_state to make sure that your handler is in a particular state. I’d recommend against doing that too much in a handler test, since it will make your tests brittle. I only do it when the state is very important, such as when checking that they are in a SPAMMING or BOUNCING state that I need to enforce.
Model Tests
There’s less functionality available in lamson.testing
for doing your models. The theory
is that your models will be classes, modules, and ORM that you need to perform the majority
of your storage and analysis. Since has very little to do with email you probably won’t
use lamson.testing
as much.
About the only things you might use are APIs for checking that queues get certain messages in them, and that certain users are in certain states.
Here’s a quick example from librelist.com again that tests how archives work:
from nose.tools import * from lamson.testing import * from lamson.mail import MailRequest, MailResponse from app.model import archive, mailinglist import simplejson as json import shutil queue_path = archive.store_path('test.list', 'queue') json_path = archive.store_path('test.list', 'json') def setup(): clear_queue(queue_path) shutil.rmtree(json_path) def teardown(): clear_queue(queue_path) shutil.rmtree(json_path) def test_archive_enqueue(): msg = MailResponse(From="zedshaw@zedshaw.com", To="test.list@librelist.com", Subject="test message", Body="This is a test.") archive.enqueue('test.list', msg) assert delivered('zedshaw', to_queue=queue(queue_path))
This is the usual initial setup, and then some extras to make sure that the JSON archives is working. You’ll notice that we hand construct various messages, call methods on the app.model.archive
module, and then use delivered
to make sure they’re correctly delivered.
Template Tests
Typically you really can only test that your templates are spelled right, or that your templates render when given certain locals. I’ve found that automated testing of templates isn’t incredibly useful yet, so the only one I’ve written is from the oneshotblog example:
from nose.tools import * from lamson.testing import * from lamson import view import os from glob import glob def test_spelling(): message = {} original = {} for path in glob("app/templates/mail/*.msg"): template = "mail/" + os.path.basename(path) result = view.render(locals(), template) spelling(template, result)
This uses lamson.testing.spelling
to make sure that each template renders and
that it is spelled correctly. This uses
PyEnchant to do the checking, which
turns out to be rather annoying. If you are interested in improving the
template testing setup, then feel free to talk about your ideas on the lamson
mailing list (but bring code, talk is cheap).
Conclusion
Hopefully you’ll be able to develop your application using good testing techniques with
the lamson.testing
API. If you find additional testing patterns that could be included
then talk about them on the lamson mailing list to see
if they’re general enough for others.