Unit testing is the process of creating small methods that test other methods for validity. These methods are usually either named with a test_ prefix or are kept in a special test project. The code quality of these tests must be to the highest level. If your tests are inaccurate, how can you rely on them to accurately inform you if your code has errors. Let’s look at the good and bad associated with unit tests:
The Good
- They provide confidence
- They are easy to write
- Remove reliance on other teams
- The aim of evermore quality
- Run as part of source control checkin
- Run as part of a daily quality check
The Bad
- They take developer time.
- They must be high standard.
- They can make your builds brittle.
- They will add to build times.
- Too much reliance, discounting manual testing.
- They cannot read business requirements.
Unit tests afford confidence for developer, company owners, product owners, auditors and salesmen to mention a few. This confidence is contagious but as mentioned must not be the only testing that is done in a project as unit test do cannot determine how a product should behave. It can only determine if a method given a certain number of parameters returns a certain result.
Your unit tests should be multi-dimensional and cover both the negative and positive scenarios. If you test in this manner, your understanding of how your method behaves in all scenarios will be far greater. For example, you may want your method to error if its parameters are null. You should check this scenario as well as the desired result scenario.
Unit tests are by nature small, testing one method only. Your methods being tested should also be small (Think SOLID). Due to their size you should be able to amass a lot test fairly quickly. If you look at the test triangle unit tests should provide the largest slice of testing in your project.
One of the largest benefits for me as a middleware developer is that I can develop code, test it and pass it on to UI developers with confidence as I have a means of running the code. No one likes to be disturbed by work you completed weeks/months ago. It also gives the user of these methods an indication of how they are intended to be used and what is expected if they are used inappropriately.
You can set your test runner to test your code as part of your Continues Integration system such as Jenkins. This can be enforced so that nothing goes into your master branch in GIT until all test pass. This is your defence barrier to faulty code getting into you master branch.
One drawback is they do take time to develop and must be revisited when the method is altered in anyway. You need to think about what you need to supply the test and what will be returned. Some questions I always ask is:
- Is the output going to be the same every time?
- Does the code rely on a third party, and would a mock work better?
- Have I separated the code enough to make the testing easy?
- Are any arguments to the unit tests the simplest type they can be?
- What data types is the test expecting and can I easily provide this?
The answers to these questions will drive what decisions you make. You may choose to refine your code to make it simpler, use a mock for third party libraries.
Once you start to amass a huge number of tests your build times will start to expand and become an issue. You can mitigate this by splitting your project into smaller projects or by using test tool features such as caching, skipping and asynchronous running.
Over reliance on unit tests can lead to less manual testing than is required. One thing developer’s know is manual testers are irreplaceable for finding bugs and misinterpretations of business requirements. In many cases they act as a bridge between the PO / Project manager. Remember the code could be perfect but if you have not met he expectation of your customers its broken.
Let’s move on to the implementation, as with my other articles I will write these in C#, Python, Golang and Java. The examples will be on the simple side and they will get you started on your unit test journey.
Setup method is a special method that allows you to set up your tests with shared code. It works in a similar fashion to constructor. It allows you to set up global test variables that every test can use. This can lead to test efficiency. The drawback being that you are sharing the same object so you could end up changing that object causing test fails in other tests or even creating a race condition especially if you run your tests using asynchronous.
Clean down Method. The clean down method works in much the same way as the set-up method does but instead of creating items it allows you the option to tear down objects you have created in the start-up method. It also allows you to notify users that your test have run, although you may want Jenkins to that as Jenkins can send test results.
C#
//using ms-test and nunit
private ICoffee wetBeanCoffee;
[TestInitialize]
public void Init() {
wetBeanCoffee = new WetBeansCoffee();
}
[TestCleanup]
public void Cleanup()
{
wetBeanCoffee = null;
}GoLang
//Using Testify
var wetBeansCoffee = WetBeansCoffee{}
func TestMain(m *testing.M) {
setup()
m.Run()
teardown()
}
func setup() {
log.Println("\n-----runs before all tests-----")
}
func teardown() {
log.Println("\n----runs after all tests----")
}
Java
//Using JUnit and Hamcrest
@BeforeAll
public static void Init()
{
wetBeanCoffee = new WetBeansCoffee();
}
@BeforeEach
public void InitBeforeEachTest()
{
System.out.println("Setting it up before every test.");
}
@AfterEach
public void CleanupAfterEachTest()
{
System.out.println("Clean up after every test");
}
@AfterAll
public static void Cleanup()
{
wetBeanCoffee = null;
}
Python
class TestCoffee(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls._wet_coffee = WetBeansCoffee()
@classmethod
def tearDownClass(cls) -> None:
cls._wet_coffee = None
def setUp(self) -> None:
print("runs before every test")
def tearDown(self) -> None:
print("runs after every test")
....A test method is where the magic happens, and these special methods call your methods under test in as many different scenarios as they can. The name of test methods is very important. The name should always describe what you are testing. This tells the next person or indeed your-future-self, briefly what is being tested.
You should have as many test methods as is required to cover all possible input scenarios. For example, you must test passing in nulls, special characters, invalid numbers to name but a few. It’s basically a great way to explore how each method works under as many different conditions as you can. The beauty is that if someone else comes along and uses that method they should know how it is supposed to work by virtue of the tests.
Assertions are a cut down version of an [if statement}. The fundamental difference is that assertions will fail a test if they are not correct. Most languages have similar assertions of which I will try and cover in this article but here are some for a little taster:
- Are the values equal
- Are the types equal
- Value is true / false
- Expectation of an exception
- Value is null
- Value is not null
- Inconclusive test
- outright fail
If you find that you don’t have what you need in the assertions provided, then you can combine an [if statement] and an assertion fail command to get your desired result. This can be super handy and offers ultimate flexibility. What I would say is try and stick to minimum custom code. If you are testing your method using the same logic then it’s pointless if you think about it.
Most assertions allow you to leave feedback. This is normally the last parameter. I don’t have strong options on whether you use these or not. It will help you pinpoint which assertion has failed in a test where multiple assertions have been made.
Right! Let’s get our hands dirty. Below are some examples of the most used assertions. isTrue, areEqual and a straight up fail are what I would call core assertions. Once you have these mastered you can create hundreds of unit tests and may even be all you need for your project.
C#
//using ms-test and NUnit.
[Order(1)]
[TestMethod]
public void TestDryingBeansHasValue()
{
string consistencyValue = wetBeanCoffee.dry_beans();
Assert.IsFalse(string.IsNullOrEmpty(consistencyValue));
}
[Order(2)]
[TestMethod]
public void TestBeanVolume()
{
int beanVolume = wetBeanCoffee.GetBeans();
Assert.AreEqual(beanVolume, 4);
}
[TestMethod]
public void TestDryingBeansForConsistency() {
String consistencyValue = wetBeanCoffee.dry_beans();
String test = wetBeanCoffee.dry_beans();
if (test != consistencyValue)
{
Assert.Fail("the value of dry beans is inconsistent");
}
}
GoLang
//Using Testify
func TestValuesAreCorrect(t *testing.T) {
coffee := NewCoffeeModel("Frapachino", 1, true, "France")
assert.Equal(t, coffee.label, "Frapachino")
assert.Equal(t, coffee.milk, true)
assert.Equal(t, coffee.beanOrigin, "France")
assert.NotEqual(t, coffee.sugarLumps, 2)
}
func TestBeanCount(t *testing.T) {
var beanCount int = wetBeansCoffee.GetBeans()
assert.Equal(t, beanCount, 4)
}
func TestDryingBeansForConsistency(t *testing.T) {
if testing.Short() {
t.Skip("skipping test")
}
consistencyValue := wetBeansCoffee.DryBeans()
test := wetBeansCoffee.DryBeans()
if test != consistencyValue {
assert.Fail(t, "the value of dry beans is inconsistent")
assert.FailNow(t, "the value of dry beans is inconsistent")
}
}
Java
//Using JUnit
@Test
@Order(1)
public void TestDryingBeansHasValue()
{
String consistencyValue = wetBeanCoffee.dry_beans();
assertNotEquals(consistencyValue, "");
}
@Test
@Order(2)
public void TestBeanVolume()
{
int beanVolume = wetBeanCoffee.GetBeans();
assertEquals(beanVolume, 4);
}
@Test
public void testLabelsAreEqual() {
CoffeeModel coffee = new CoffeeModel();
coffee = wetBeanCoffee.assignCoffeeProperties(1, "Frapachino", true, "France");
assertEquals(coffee.getLabel(), "Frapachino");
}
@Test
public void getGranniesSecretReceipt() throws IOException {
assertFalse(wetBeanCoffee.runGetCoffee());
}
@Test
public void testDryingBeansForConsistency() {
String consistencyValue = wetBeanCoffee.dry_beans();
String test = wetBeanCoffee.dry_beans();
if (test != consistencyValue)
{
Assertions.fail("the value of dry beans is inconsistent");
}
}
Python
def test_a_drying_beans_has_value(self):
consistency_value = self.__class__._wet_coffee.dry_beans()
self.assertNotEquals(consistency_value, "")
def test_b_bean_volume(self):
bean_volume = self.__class__._wet_coffee.get_beans()
self.assertEquals(bean_volume, 4)
def test_labels_are_equal(self):
coffee_one = self.__class__._wet_coffee.assign_coffee_properties(1, "Frapachino", True, "France")
self.assertEquals(coffee_one.label, "Frapachino");
def test_drying_beans_for_consistency(self):
consistency_value = self.__class__._wet_coffee.dry_beans()
test = self.__class__._wet_coffee.dry_beans()
if test != consistency_value:
raise AssertionError
Type assertions are something we need to look closely at. They are not always as they seem. We will look at checking type, object and value. Type is the type of object it is, object checks if it uses the same slot in memory and value checks the value held inside of the object. Strings will often only do an are equal check, so do not use the [same methods] for these comparisons, use the [areEqual].
C#
You will see from the below code that we are creating two new coffee objects and assigning the first coffee object to the third, essentially making the first and third coffee variables the same underlying object.
I have deliberately made the content of the strings in the last example the same to show you how these are a match, even although they are different objects. This could lead to invalid tests results. If this is needed use assertEquals command as this is what this is for.
//using ms-test and NUnit
[TestMethod]
public void TestIfTwoCoffeeObjectsAreTheSame()
{
CoffeeModel coffeeOne = new CoffeeModel();
CoffeeModel coffeeTwo = new CoffeeModel();
coffeeOne = wetBeanCoffee.AssignCoffeeProperties(1, "Frapachino", true, "France");
coffeeTwo = wetBeanCoffee.AssignCoffeeProperties(1, "Frapachino", true, "France");
CoffeeModel coffeeThree = coffeeOne;
Assert.AreSame(coffeeOne, coffeeThree);
Assert.AreNotSame(coffeeOne, coffeeTwo);
// watch this gotcha.
Assert.AreSame(coffeeOne.Label, coffeeTwo.Label);
}
GoLang
I was very pleased with GoLang’s interpretation of objects and it was the most accurate at assessing if two objects are the same including at the string level which Java, Python and C# all failed to achieve.
You will see from the below code that we are creating two new coffee objects and assigning the first coffee object to the third, essentially making the first and third coffee variables the same underlying object.
func TestCompareObjects(t *testing.T) {
coffeeOne := NewCoffeeModel("Frapachino", 1, true, "France")
coffeeTwo := NewCoffeeModel("Frapachino", 1, true, "France")
coffeeThree := &coffeeOne
coffeeFour := &coffeeOne
// are pointing to the same object
assert.Same(t, coffeeThree, coffeeFour)
// are not pointing to the same object although are the same type and content
assert.NotSame(t, coffeeOne, coffeeTwo)
assert.NotSame(t, coffeeOne.label, coffeeTwo.label)
// asserts that two objects are equal or convertable to the same types and equal.
assert.EqualValues(t, coffeeOne.label, coffeeTwo.label)
assert.EqualValues(t, coffeeOne, coffeeTwo)
// Exactly asserts that two objects are equal in value and type.
assert.Exactly(t, coffeeThree, coffeeFour)
}Java
You will see from the below code that we are creating two new coffee objects and assigning the first coffee object to the third, essentially making the first and third coffee variables the same underlying object.
I have deliberately made the content of the strings in the last example the same to show you how these are a match, even although they are different objects. This could lead to invalid tests results. If this is needed use assertEquals command as this is what this is for.
//Using JUnit
@Test
public void testComparingObjects()
{
CoffeeModel coffeeOne;
CoffeeModel coffeeTwo;
coffeeOne = wetBeanCoffee.assignCoffeeProperties(1, "Frapachino", true, "France");
coffeeTwo = wetBeanCoffee.assignCoffeeProperties(1, "Frapachino", true, "France");
CoffeeModel coffeeThree = coffeeOne;
//are the same as the point to the same object
assertSame(coffeeOne,coffeeThree);
assertEquals(coffeeOne,coffeeThree);
// are not the same although same type and same values
assertNotSame(coffeeOne, coffeeTwo);
assertNotEquals(coffeeOne, coffeeTwo);
// watch this gotcha. This should return false as they are not the same item, they do have the same value and will fail if they have different values.
assertSame(coffeeOne.getLabel(), coffeeTwo.getLabel());
// should use below for consistency
assertEquals(coffeeOne.getLabel(), coffeeTwo.getLabel());
}
Python
You will see from the below code that we are creating two new coffee objects and assigning the first coffee object to the third, essentially giving them the same underlying object.
Watch out for the string item within the coffee as this is in my opinion wrong as these are different objects and not the same object. It is doing a comparison of the string contents and because they are both frapachino it passes. If this is needed using assertEqual command as this is what this is for.
def test_if_two_coffee_objects_are_the_same(self):
coffee_one = self.__class__._wet_coffee.assign_coffee_properties(1, "Frapachino", True, "France")
coffee_two = self.__class__._wet_coffee.assign_coffee_properties(1, "Frapachino", True, "France")
coffee_three = coffee_one
self.assertIsNot(coffee_one, coffee_two)
self.assertIs(coffee_one, coffee_three)
self.assertIsInstance(coffee_one, CoffeeModel)
# watch out for this as they are both string but not the same object. Looks only to assess the contents
self.assertIs(coffee_one.label, coffee_two.label);Marking tests as ignore or disabled affords you the ability to have the tests excluded from the test run. This can be for many reasons:
They are part of a test-driven development program and although they, they code they rely on is not written yet.
Perhaps they take a long time to run and you want to limit certain tests to only run when you want to perform a certain test scenario manually. Some languages allow you to ignore tests based on a regex of their name or a flag as part of the test run command. This allows you to run a small subset of tests on a pull request and all the tests overnight when the resources are not being used.
I am almost certain there will be other reasons. I am personally not a fan of having redundant code lying about in a project, so i have not specified that as a valid reason. Inside of the decorators for ignoring tests I would highly recommend giving a reason why a test is being ignored as this will help lift the vail of mystery around why the test is being ignored.
I will now show you how to achieve this in the examples below.
C#
The [Ignore(“Do not run this test as we are not ready for it.”)] decorator is used in c# to skip tests. The tests results will have a handy breakdown of how many test passed, skipped and failed. The skipped value is the value that tells you how many tests where ignored.
//using ms-test and NUnit
[Ignore("Do not run this test as we are not ready for it.")]
[TestMethod]
public void DryingBeansForConsistency()
{
string consistencyValue = wetBeanCoffee.dry_beans();
for (int i = 0; i < 5; i++)
{
string test = wetBeanCoffee.dry_beans();
if (test != consistencyValue)
{
Assert.Inconclusive("the value of dry beans is inconsistent");
}
}
}
GoLang
GoLang does not have a decorator to skip tests. You can use the t.skip from inside the test method to indicate that you want to skip this test. To call it form the test runner command line to disallow resource hungry tests you can wrap it in an if statement with the testing.short run variable and pass in –short argument.
// to skip this test run the test call with short arg
//go test -short -v
func TestDryingBeansForConsistency(t *testing.T) {
if testing.Short() {
t.Skip("skipping test")
}
consistencyValue := wetBeansCoffee.DryBeans()
test := wetBeansCoffee.DryBeans()
if test != consistencyValue {
assert.Fail(t, "the value of dry beans is inconsistent")
assert.FailNow(t, "the value of dry beans is inconsistent")
}
}
func TestSkipThisTestAsItsBroken(t *testing.T) {
t.Skip("Skip this test as we know it is going ot fail.")
assert.True(t, false)
}Java
In Junit you can use the @Disabled(“Do not run this test as we are not ready for it.”) decorator to ensure a unit test is not run. The tests results will have a handy breakdown of how many test passed, skipped and failed. The skipped value is the value that tells you how many tests where ignored.
//Using JUnit
@Disabled("Do not run this test as we are not ready for it.")
@Test
public void testDryingBeansForConsistency() {
String consistencyValue = wetBeanCoffee.dry_beans();
String test = wetBeanCoffee.dry_beans();
if (test != consistencyValue)
{
Assertions.fail("the value of dry beans is inconsistent");
}
}
Python
The @unittest.skip decorator is used in python to skip tests and works well. The test run will tell you how many tests have been skipped and how many have been run.
@unittest.skip('Do not run this test as we are not ready for it.')
def test_drying_beans_for_consistency(self):
consistency_value = self.__class__._wet_coffee.dry_beans()
test = self.__class__._wet_coffee.dry_beans()
if test != consistency_value:
raise AssertionErrorModern testing tools are really powerful and you can avoid adding loops in your tests by using group assertions or repeat decorators. Group assertions will analyse a collection, allowing you to check that items are unique, not null or not included in the collection.
Some Languages allow you to specify a repeat decorator. The repeat decorator will run the test as many times as specified. This great if you method is likely to return different results with the same parameters as it means you can make assertions based on this.
There is nothing wrong with loops, however the less programatic code your right into your tests the less that can go wrong with the tests themselves. Here are some examples of tests, one using group assertions and one using the repeat decorator.
C#
The [Repeat(11)] decorator tells the test runner to run the first test 11 times. The collection assertions have rich features such as does contain, all items are not null and all items are unique amongst others.
//using ms-test and NUnit
[Repeat(11)]
[TestMethod]
public void DryingBeansForConsistancyWithException()
{
string consistencyValue = wetBeanCoffee.dry_beans();
string test = wetBeanCoffee.dry_beans();
if (test != consistencyValue)
{
Assert.Inconclusive("the value of dry beans is inconsistent");
}
}
[TestMethod]
public void TestCoffeeGroup()
{
CoffeeModel coffeeOne = new CoffeeModel();
coffeeOne = wetBeanCoffee.AssignCoffeeProperties(1, "Frapachino", true, "France");
List<CoffeeModel> OrderredByGroup = wetBeanCoffee.GroupOrder();
CollectionAssert.AllItemsAreUnique(OrderredByGroup);
//note it is the same type with same values but it is a different object
CollectionAssert.DoesNotContain(OrderredByGroup, coffeeOne);
CollectionAssert.AllItemsAreNotNull(OrderredByGroup);
}
GoLang
Golang is cutting its own shapes with regards to repeat and you need to run it from the test runner using the count argument. I would also recommend combining this with the verbose argument as you will then see the test passes which are normally hidden. The collection assertions functionality is loaded with features such as contains, not contains, empty and subset.
// to test this multiple time to get a failure we will need to run it as a script
//go test -run TestDryingBeansForConsistency -count 10 -v
func TestDryingBeansForConsistency(t *testing.T) {
consistencyValue := wetBeansCoffee.DryBeans()
test := wetBeansCoffee.DryBeans()
if test != consistencyValue {
assert.Fail(t, "the value of dry beans is inconsistent")
assert.FailNow(t, "the value of dry beans is inconsistent")
}
}
func TestCoffeeGroup(t *testing.T) {
var subOrder []CoffeeModel
var newOrder []CoffeeModel
subOrder = append(subOrder, wetBeansCoffee.AssignCoffeeProperties(1, "Frapachino", true, "France"))
subOrder = append(subOrder, wetBeansCoffee.AssignCoffeeProperties(1, "Coffee", false, "France"))
coffeeOne := NewCoffeeModel("Frapachino", 1, true, "France")
hotChocolate := NewCoffeeModel("hot chocolate with marshmallows", 1, true, "France")
orderedByGroup := wetBeansCoffee.GroupOrder()
// list contains
assert.Contains(t, orderedByGroup, coffeeOne)
// list does not contain
assert.NotContains(t, orderedByGroup, hotChocolate)
// list is not empty
assert.NotEmpty(t, orderedByGroup)
// list is empty
assert.Empty(t, newOrder)
// subset is part of list
assert.Subset(t, orderedByGroup, subOrder)
}
Java
The @RepeatTest(11) decorator tells the test runner to run the first test 11 times. The second of the example test methods uses the hamcrest library to assert that items do or do not exist within collections.
@RepeatedTest(11)
@Test()
public void DryingBeansForConsistancyWithException() {
String consistencyValue = wetBeanCoffee.dry_beans();
String test = wetBeanCoffee.dry_beans();
if (test != consistencyValue)
{
Assertions.fail("the value of dry beans is inconsistent");
}
}
@Test
public void TestCoffeeGroup()
{
CoffeeModel coffeeOne;
coffeeOne = wetBeanCoffee.AssignCoffeeProperties(1, "Frapachino", true, "France");
List<CoffeeModel> OrderredByGroup = wetBeanCoffee.GroupOrder();
assertThat(OrderredByGroup, not(hasItem(coffeeOne)));
assertThat(OrderredByGroup, not(nullValue()));
}Python
Python is an outside case and does not have a repeater method, however there is a work around if you create your own repeat decorator. Then you can call @repeat(11) from your test method and this tells the test runner to run the test 11 times. Note this will not work if you rely on setUp and tearDown methods.
The second test method uses assert in and assert not in to determine if the collection has or does not have certain records. Note it will not recognise a different object and therefore you cannot redefine your testable object in the test. You can see i have appended it, you will need to give some thought to this when doing your own tests.
def repeat(times):
def repeat_helper(f):
def call_helper(*args):
for i in range(0, times):
f(*args)
return call_helper
return repeat_helper
class TestCoffee(unittest.TestCase):
@repeat(11)
def test_drying_beans_for_consistency_with_exception(self):
consistency_value = self.__class__._wet_coffee.dry_beans()
test = self.__class__._wet_coffee.dry_beans()
if test != consistency_value:
print("not consistent.")
else:
print("consistent.")
def test_coffee_group(self):
coffee_one = self.__class__._wet_coffee.assign_coffee_properties(1, "Frapachino", True, "France")
# same but not the same object
same_content_only = self.__class__._wet_coffee.assign_coffee_properties(1, "Frapachino", True, "France")
OrderedByGroup = self.__class__._wet_coffee.group_order()
OrderedByGroup.append(coffee_one)
self.assertIn(coffee_one, OrderedByGroup)
self.assertNotIn(same_content_but_not_coffee_one, OrderedByGroup)Test handling exceptions is the last part of the puzzle and is where a test will throw an exception and this is expected. In my opinion this is a valid unhappy path that you want to test. Especially if you only
Handling exceptions is the last part of the puzzle. You will need to know how to make an assertion that a test will fail with an exception and that is fine and a test pass.
If you are testing your methods for unhappy and happy paths you will eventually come across a method that throws an exception and escalates it up through the method. You can event tests for specific exceptions, I believe this will lead to a greater understanding of how your code works and more importantly fails. Let’s take a look at how we will handle exceptions in our test methods.
C#
//using ms-test and NUnit
[TestMethod()]
[ExpectedException(typeof(ArgumentNullException), "To throw an exception when argument is null.")]
public void WeakCoffeeGrindBeansTestWithInvalidArgument()
{
wetBeanCoffee.GrindBeans(null);
}
GoLang
//Using Testify
func TestErrorAssumptions(t *testing.T) {
_, err := wetBeansCoffee.GrindBeans("")
assert.Error(t, err)
assert.ErrorContains(t, err, "cannot have an empty power source")
}Java
//Using JUnit
@Test
public void WeakCoffeeGrindBeansTestWithInvalidArgument()
{
assertThrows(NullPointerException.class, () -> {wetBeanCoffee.GrindBeans(null);});
}
Python
def test_weak_coffee_grind_beans_with_invalid_argument(self):
with self.assertRaises(Exception):
power_source = ""
self.__class__._coffee_model.grind_beans(power_source)and there you have it, a brief guide to unit tests. I will be revisiting this topic in more detail in other articles as I feel there is far more to talk about with regards to decorators, test drivers and assertions but as a basic starting place I think we have done not bad.
