mirror of https://github.com/Mikaela/Limnoria.git
284 lines
14 KiB
Plaintext
284 lines
14 KiB
Plaintext
![]() |
Advanced Plugin Testing
|
||
|
-----------------------
|
||
|
The complete guide to writing tests for your plugins.
|
||
|
|
||
|
Why Write Tests?
|
||
|
================
|
||
|
Why should I write tests for my plugin? Here's why.
|
||
|
|
||
|
For those of you asking "Why should I write tests for my plugin? I tried it
|
||
|
out, and it works!", read on. For those of you who already realize that
|
||
|
Testing is Good (TM), skip to the next section.
|
||
|
|
||
|
Here are a few quick reasons why to test your Supybot plugins.
|
||
|
|
||
|
* When/if we rewrite or change certain features in Supybot, tests make
|
||
|
sure your plugin will work with these changes. It's much easier to run
|
||
|
supybot-test MyPlugin after upgrading the code and before even reloading
|
||
|
the bot with the new code than it is to load the bot with new code and
|
||
|
then load the plugin only to realize certain things don't work. You may
|
||
|
even ultimately decide you want to stick with an older version for a while
|
||
|
as you patch your custom plugin. This way you don't have to rush a patch
|
||
|
while restless users complain since you're now using a newer version that
|
||
|
doesn't have the plugin they really like.
|
||
|
|
||
|
* Running the automated tests takes a few seconds, testing plugins in IRC
|
||
|
on a live bot generally takes quite a bit longer. We make it so that
|
||
|
writing tests generally doesn't take much time, so a small initial
|
||
|
investment adds up to lots of long-term gains.
|
||
|
|
||
|
* If you want your plugin to be included in any of our releases (the core
|
||
|
Supybot if you think it's worthy, or our supybot-plugins package), it has
|
||
|
to have tests. Period.
|
||
|
|
||
|
For a bigger list of why to write unit tests, check out this article:
|
||
|
|
||
|
http://www.onjava.com/pub/a/onjava/2003/04/02/javaxpckbk.html
|
||
|
|
||
|
and also check out what the Extreme Programming folks have to say about unit
|
||
|
tests:
|
||
|
|
||
|
http://www.extremeprogramming.org/rules/unittests.html
|
||
|
|
||
|
Plugin Tests
|
||
|
============
|
||
|
How to write tests for commands in your plugins.
|
||
|
|
||
|
Introduction
|
||
|
|
||
|
This tutorial assumes you've read through the plugin author tutorial, and that
|
||
|
you used supybot-plugin-create to create your plugin (as everyone should). So,
|
||
|
you should already have all the necessary imports and all that boilerplate
|
||
|
stuff in test.py already, and you have already seen what a basic plugin test
|
||
|
looks like from the plugin author tutorial. Now we'll go into more depth about
|
||
|
what plugin tests are available to Supybot plugin authors.
|
||
|
|
||
|
Plugin Test Case Classes
|
||
|
|
||
|
Supybot comes with two plugin test case classes, PluginTestCase and
|
||
|
ChannelPluginTestCase. The former is used when it doesn't matter whether or
|
||
|
not the commands are issued in a channel, and the latter is used for when it
|
||
|
does. For the most part their API is the same, so unless there's a distinction
|
||
|
between the two we'll treat them as one and the same when discussing their
|
||
|
functionality.
|
||
|
|
||
|
The Most Basic Plugin Test Case
|
||
|
|
||
|
At the most basic level, a plugin test case requires three things:
|
||
|
|
||
|
* the class declaration (subclassing PluginTestCase or
|
||
|
ChannelPluginTestCase)
|
||
|
* a list of plugins that need to be loaded for these tests (does not
|
||
|
include Owner, Misc, or Config, those are always automatically loaded) -
|
||
|
often this is just the name of the plugin that you are writing tests for
|
||
|
* some test methods
|
||
|
|
||
|
Here's what the most basic plugin test case class looks like (for a plugin
|
||
|
named MyPlugin):
|
||
|
|
||
|
class MyPluginTestCase(PluginTestCase):
|
||
|
plugins = ('MyPlugin',)
|
||
|
|
||
|
def testSomething(self):
|
||
|
# assertions and such go here
|
||
|
|
||
|
Your plugin test case should be named TestCase as you see above, though it
|
||
|
doesn't necessarily have to be named that way (supybot-plugin-create puts that
|
||
|
in place for you anyway). As you can see we elected to subclass PluginTestCase
|
||
|
because this hypothetical plugin apparently doesn't do anything
|
||
|
channel-specific.
|
||
|
|
||
|
As you probably noticed, the plugins attribute of the class is where the list
|
||
|
of necessary plugins goes, and in this case just contains the plugin that we
|
||
|
are testing. This will be the case for probably the majority of plugins. A lot
|
||
|
of the time test writers will use a bot function that performs some function
|
||
|
that they don't want to write code for and they will just use command nesting
|
||
|
to feed the bot what they need by using that plugin's functionality. If you
|
||
|
choose to do this, only do so with core bot plugins as this makes distribution
|
||
|
of your plugin simpler. After all, we want people to be able to run your
|
||
|
plugin tests without having to have all of your plugins!
|
||
|
|
||
|
One last thing to note before moving along is that each of the test methods
|
||
|
should describe what they are testing. If you want to test that your plugin
|
||
|
only responds to registered users, don't be afraid to name your test method
|
||
|
testOnlyRespondingToRegisteredUsers or testNotRespondingToUnregisteredUsers.
|
||
|
You may have noticed some rather long and seemingly unwieldy test method names
|
||
|
in our code, but that's okay because they help us know exactly what's failing
|
||
|
when we run our tests. With an ambiguously named test method we may have to
|
||
|
crack open test.py after running the tests just to see what it is that failed.
|
||
|
For this reason you should also test only one thing per test method. Don't
|
||
|
write a test method named testFoobarAndBaz. Just write two test methods,
|
||
|
testFoobar and testBaz. Also, it is important to note that test methods must
|
||
|
begin with test and that any method within the class that does begin with test
|
||
|
will be run as a test by the supybot-test program. If you want to write
|
||
|
utility functions in your test class that's fine, but don't name them
|
||
|
something that begins with test or they will be executed as tests.
|
||
|
|
||
|
Including Extra Setup
|
||
|
|
||
|
Some tests you write may require a little bit of setup. For the most part it's
|
||
|
okay just to include that in the individual test method itself, but if you're
|
||
|
duplicating a lot of setup code across all or most of your test methods it's
|
||
|
best to use the setUp method to perform whatever needs to be done prior to
|
||
|
each test method.
|
||
|
|
||
|
The setUp method is inherited from the whichever plugin test case class you
|
||
|
chose for your tests, and you can add whatever functionality you want to it.
|
||
|
Note the important distinction, however: you should be adding to it and not
|
||
|
overriding it. Just define setUp in your own plugin test case class and it
|
||
|
will be run before all the test methods are invoked.
|
||
|
|
||
|
Let's do a quick example of one. Let's write a setUp method which registers a
|
||
|
test user for our test bot:
|
||
|
|
||
|
def setUp(self):
|
||
|
ChannelPluginTestCase.setUp(self) # important!!
|
||
|
# Create a valid user to use
|
||
|
self.prefix = 'foo!bar@baz'
|
||
|
self.feedMsg('register tester moo', to=self.nick, frm=self.prefix))
|
||
|
m = self.getMsg() # Response to registration.
|
||
|
|
||
|
Now notice how the first line calls the parent class's setUp method first?
|
||
|
This must be done first. Otherwise several problems are likely to arise. For
|
||
|
one, you wouldn't have an irc object at self.irc that we use later on nor
|
||
|
would self.nick be set.
|
||
|
|
||
|
As for the rest of the method, you'll notice a few things that are available
|
||
|
to the plugin test author. self.prefix refers to the hostmask of the
|
||
|
hypothetical test user which will be "talking" to the bot, issuing commands.
|
||
|
We set it to some generically fake hostmask, and then we use feedMsg to feed
|
||
|
it a private message (using the bot's nick, accessible via self.nick) to
|
||
|
register the username "tester" with the password "moo". We have to do it this
|
||
|
way (rather than what you'll find out is the standard way of issuing commands
|
||
|
to the bot in test cases a little later) because registration must be done in
|
||
|
private. And lastly, since feedMsg doesn't dequeue any messages from the bot
|
||
|
after getting fed a message, we perform a getMsg to get the response. You're
|
||
|
not expected to know all this yet, but do take note of it since using these
|
||
|
methods in test-writing is not uncommon. These utility methods as well as all
|
||
|
of the available assertions are covered in the next section.
|
||
|
|
||
|
So, now in any of the test methods we write, we'll be able to count on the
|
||
|
fact that there will be a registered user "tester" with a password of "moo",
|
||
|
and since we changed our prefix by altering self.prefix and registered after
|
||
|
doing so, we are now identified as this user for all messages we send unless
|
||
|
we specify that they are coming from some other prefix.
|
||
|
|
||
|
The Opposite of Setting-up: Tearing Down
|
||
|
|
||
|
If you did some things in your setUp that you want to clean up after, then
|
||
|
this code belongs in the tearDown method of your test case class. It's
|
||
|
essentially the same as setUp except that you probably want to wait to invoke
|
||
|
the parent class's tearDown until after you've done all of your tearing down.
|
||
|
But do note that you do still have to invoke the parent class's tearDown
|
||
|
method if you decide to add in your own tear-down stuff.
|
||
|
|
||
|
Setting Config Variables for Testing
|
||
|
|
||
|
Before we delve into all of the fun assertions we can use in our test methods
|
||
|
it's worth noting that each plugin test case can set custom values for any
|
||
|
Supybot config variable they want rather easily. Much like how we can simply
|
||
|
list the plugins we want loaded for our tests in the plugins attribute of our
|
||
|
test case class, we can set config variables by creating a mapping of
|
||
|
variables to values with the config attribute.
|
||
|
|
||
|
So if, for example, we wanted to disable nested commands within our plugin
|
||
|
testing for some reason, we could just do this:
|
||
|
|
||
|
class MyPluginTestCase(PluginTestCase):
|
||
|
config = {'supybot.commands.nested': False}
|
||
|
|
||
|
def testThisThing(self):
|
||
|
# stuff
|
||
|
|
||
|
And now you can be assured that supybot.commands.nested is going to be off for
|
||
|
all of your test methods in this test case class.
|
||
|
|
||
|
Plugin Test Methods
|
||
|
===================
|
||
|
The full list of test methods and how to use them.
|
||
|
|
||
|
Introduction
|
||
|
|
||
|
You know how to make plugin test case classes and you know how to do just
|
||
|
about everything with them except to actually test stuff. Well, listed below
|
||
|
are all of the assertions used in tests. If you're unfamiliar with what an
|
||
|
assertion is in code testing, it is basically a requirement of something that
|
||
|
must be true in order for that test to pass. It's a necessary condition. If
|
||
|
any assertion within a test method fails the entire test method fails and it
|
||
|
goes on to the next one.
|
||
|
|
||
|
Assertions
|
||
|
|
||
|
All of these are methods of the plugin test classes themselves and hence are
|
||
|
accessed by using self.assertWhatever in your test methods. These are sorted
|
||
|
in order of relative usefulness.
|
||
|
|
||
|
* assertResponse(query, expectedResponse) - Feeds query to the bot as a
|
||
|
message and checks to make sure the response is expectedResponse. The
|
||
|
test fails if they do not match (note that prefixed nicks in the
|
||
|
response do not need to be included in the expectedResponse).
|
||
|
|
||
|
* assertError(query) - Feeds query to the bot and expects an error in
|
||
|
return. Fails if the bot doesn't return an error.
|
||
|
|
||
|
* assertNotError(query) - The opposite of assertError. It doesn't matter
|
||
|
what the response to query is, as long as it isn't an error. If it is
|
||
|
not an error, this test passes, otherwise it fails.
|
||
|
|
||
|
* assertRegexp(query, regexp, flags=re.I) - Feeds query to the bot and
|
||
|
expects something matching the regexp (no m// required) in regexp with
|
||
|
the supplied flags. Fails if the regexp does not match the bot's
|
||
|
response.
|
||
|
|
||
|
* assertNotRegexp(query, regexp, flags=re.I) - The opposite of
|
||
|
assertRegexp. Fails if the bot's output matches regexp with the
|
||
|
supplied flags.
|
||
|
|
||
|
* assertHelp(query) - Expects query to return the help for that command.
|
||
|
Fails if the command help is not triggered.
|
||
|
|
||
|
* assertAction(query, expectedResponse=None) - Feeds query to the bot and
|
||
|
expects an action in response, specifically expectedResponse if it is
|
||
|
supplied. Otherwise, the test passes for any action response.
|
||
|
|
||
|
* assertActionRegexp(query, regexp, flags=re.I) - Basically like
|
||
|
assertRegexp but carries the extra requirement that the response must
|
||
|
be an action or the test will fail.
|
||
|
|
||
|
Utilities
|
||
|
|
||
|
* feedMsg(query, to=None, frm=None) - Simply feeds query to whoever is
|
||
|
specified in to or to the bot itself if no one is specified. Can also
|
||
|
optionally specify the hostmask of the sender with the frm keyword.
|
||
|
Does not actually perform any assertions.
|
||
|
|
||
|
* getMsg(query) - Feeds query to the bot and gets the response.
|
||
|
|
||
|
Other Tests
|
||
|
===========
|
||
|
If you had to write helper code for a plugin and want to test it, here's
|
||
|
how.
|
||
|
|
||
|
Previously we've only discussed how to test stuff in the plugin that is
|
||
|
intended for IRC. Well, we realize that some Supybot plugins will require
|
||
|
utility code that doesn't necessarily require all of the overhead of setting
|
||
|
up IRC stuff, and so we provide a more lightweight test case class,
|
||
|
SupyTestCase, which is a very very light wrapper around unittest.TestCase
|
||
|
(from the standard unittest module) that basically just provides a little
|
||
|
extra logging. This test case class is what you should use for writing those
|
||
|
test cases which test things that are independent of IRC.
|
||
|
|
||
|
For example, in the MoobotFactoids plugin there is a large chunk of utility
|
||
|
code dedicating to parsing out random choices within a factoid using a class
|
||
|
called OptionList. So, we wrote the OptionListTestCase as a SupyTestCase for
|
||
|
the MoobotFactoids plugin. The setup for test methods is basically the same as
|
||
|
before, only you don't have to define plugins since this is independent of
|
||
|
IRC.
|
||
|
|
||
|
You still have the choice of using setUp and tearDown if you wish, since those
|
||
|
are inherited from unittest.TestCase. But, the same rules about calling the
|
||
|
setUp or tearDown method from the parent class still apply.
|
||
|
|
||
|
With all this in hand, now you can write great tests for your Supybot plugins!
|
||
|
|