For evaluating exercises written using Python 3, we're using the unittest framework.
Let's start with a Hello World exercise to explain the basics of evaluation in Python3. Following is the solution file named
Evaluation of this simple "Hello World" exercise is also very simple. All we have to do is to create a class extending unittest's
TestCase class and declare a test method inside. The test method will get the string printed in stdout and compare it with expected output ("Hello World").
import sysfrom unittest import TestCaseclass Evaluate(TestCase):def test_hello_world(self):import exercise # Imports and runs student's solutionoutput = sys.stdout.getvalue() # Returns output since this function startedself.assertEqual("Hello World\n", output, "You must print Hello World")
Let's see what we did in the above solution code.
1st line, the
sys module is imported which will help us get the output of student's code
2nd line, TestCase class is imported to define our test class
Evaluate and it is defined on the 4th line
5th line, our first test function is defined
6th line, the exercise module that is implemented by students is imported. Since the solution code is not in a function, it is enough to import the module to run the student's code. We have to import it inside the test function to be able to get the output.
7th line, we get the output of exercise module.
sys.stdout.getvalue() does exactly what we want here. It will only get the strings written to stdout since the test function has started. So you cannot get any outputs written before the
test_hello_world() has begun.
8th line asserts whether the output is correct and returns feedback if it's not.
When what you want to test lives outside of a function or class definition (like our Hello World example above), you will need to re-import the student's module for each of your test cases.
Let's see an example. Consider a very basic exercise which has a solution as below:
import sysarg1 = sys.argvarg2 = sys.argvprint(arg1 + arg2)
This code gets two command line arguments and prints the concatenation of them.
To test this code with various inputs, we will need to import the module several times. The test code will be a bit ugly, however this is the only way we can do such trick in Python. An example test looks like:
import sysfrom unittest import TestCaseimport importlib # module to import other modules# This is necessary to import exercise module successfully the first timesys.argv = ['0', '1', '2']import exercise # Import the module to enable reloadingclass Evaluate(TestCase):def test_exercise(self):sys.argv = ['exercise.py', 'face', 'book']importlib.reload(exercise) # re-import the module and run the code againself.assertEqual('facebook\n', sys.stdout.getvalue(), "Fails for the arguments 'face' and 'book'")def test_exercise_longer_args(self):sys.argv = ['exercise.py', 'randomaccess', 'memory']importlib.reload(exercise) # re-import the module and run the code againself.assertEqual('randomaccessmemory\n', sys.stdout.getvalue(), "Fails for the arguments 'randomaccess' and 'memory'")
Let's see what we did above and why.
1st line imports the sys module which will help us provide arguments to student's solution and get output strings.
3rd line imports the module that provides functionality for loading and reloading other modules. Because we are not inside a unittest at this point, we can't test the student's code yet.
5th and 6th lines are meaningful together. We need to import the module globally to make reloading work. In order to import this module, we have to provide some arguments since the module doesn't check whether there are arguments or not. If we don't fill
sys.argv , importing will fail.
On the first test, 11th line, we add the actual test arguments to
sys.argv , then reload the exercise module on 12th line. It will simply re-run the entire module code again and print the new outputs.
On the second test we do the same thing, pass new arguments and reload the module.
If you try importing the module with
import statements in each test function, Python will ignore the second and other imports. And if you try importing and reloading the module in the same test function,
sys.stdout.getvalue() will return the output of two imports since the code will run twice.
Thus, you have to import the module globally first, and reload it in each test function to be able to run the code several times in different test functions.
Now we will change the Hello World example a little bit and move the
print() statement into a function. The task is still to print "Hello World" but only when the
hello_world() function is called.
Following is the sample solution code in
def hello_world():print("Hello World")
To check whether the student's function is working properly, we need to import the module and call the function.
The code below is a sample evaluator for
from unittest import TestCaseclass Evaluate(TestCase):def test_sum(self):import exercise # Imports student's solutionexercise.hello_world() # Call student's functionoutput = sys.stdout.getvalue() # Returns output since this function startedself.assertEqual("Hello World\n", output, "You must print Hello World")
The only difference of this evaluation code from the original is that we explicitly called
hello_world() function since importing the module is not enough to run the code anymore.
In your evaluator code, you can check whether a variable is defined in the student's code and check its value. Consider the exercise is to declare two variables, namely name and age and assign "Jon Snow" and 29 respectively. A sample solution file (exercise.py) is below:
name = "John Snow"age = 29
In the test code, we will first check if the variables exist and then compare their values with expected ones. Here is an example test code:
import exercise # Import student's codefrom unittest import TestCaseclass Evaluate(TestCase):def test_name(self):self.assertTrue(hasattr(exercise, "name"), "You must declare 'name'")self.assertEqual(exercise.name, "John Snow", "'name' value seems wrong")def test_age(self):self.assertTrue(hasattr(exercise, "age"), "You must declare 'age'")self.assertEqual(exercise.age, 29, "'age' value seems wrong")
You can test student's object oriented knowledge. Let's say the exercise is asking students to declare a class named
Dog and implement a function inside named
bark() . The function will print "woof woof" when it's called.
An example solution is given below:
class Dog:def bark(self):print("woof woof")
Despite the fact that student may submit an empty solution file, we don't need to manually check whether the
Dog class is declared. The only thing we need to do is to import it and test it.
import sysfrom unittest import TestCasefrom exercise import Dog # Import Dog class only (will fail if not declared)class Evaluate(TestCase):def test_bark(self):dog = Dog()dog.bark()output = sys.stdout.getvalue()self.assertEqual("woof woof\n", output, "bark() must pring 'woof woof'")
Now we will get into the details of writing unit tests in Python 3.
In a test case where you would like to test the use of a specific function, you will need to mock that function. How can you test use of functions?
Force students to use a specific function
You can mock a function and assert the call count of the function is more than 0.
Ensure the arguments passed to a function are correct
You can test a mocked function arguments with
Prevent students from using a specific function
You can mock a function and assert the call count of the function is 0.
Change the behaviour of a function
You can change the return value of a mocked function and or completely override it via
Let's see how to use mocking in different situations.
Let's say we want student to call a builtin function in order to solve a problem. We may want to change the behaviour of the builtin function while testing student's code.
Let's say the question is to read a file,
people.txt , from disk and parse it. The file contains "first_name | last_name | birthdate" in each line. Students will implement a function named
parse() and this will return a list of first names. An example way to solve this problem is given below:
def parse():people_file = open('people.txt', 'r')lines = people_file.readlines()return [line.split('|') for line in lines]
In the test file, we will have to mock the builtin
open() function to test different behaviours.
from unittest import TestCase, mockfrom exercise import parse # Import parse function only (will fail if not declared)class Evaluate(TestCase):@mock.patch('builtins.open')def test_parse_single_line(self, mock_open):mock_open.return_value.readlines.return_value = ['Ned|Stark|10/10/10']result = parse()self.assertEqual(['Ned'], result, "Feedback about the mistake")
Let's examine the new concepts in the above evaluation code.
This decorator mocks a function and gives you the ability to change it's behaviour or track if it's called or not. The mocked function is provided as argument to the test function.
This line sets the return value of
readlines() function. Since the readlines is actually on the object returned by
open() function, we had to first reach the
return_value of open.
This evaluation code works with the sample solution code. However, students may use the
open() function in different ways and cause the tests fail. For instance, if a student uses
read() rather than
readlines() on the returned file object, it will return a Mock object since we didn't set return_value for
read() and the test will fail although student's solution might be correct. Thus, using mocks in some type of exercises that students can solve the same problem in several ways is not a good practice.
On the other hand, it is a good way to force students to use a specific method or a specific algorithm. In this example, we may intentionally want students to use
readlines() method if there is a lecture about it before this quiz. Then, we will have to add one more line to the test code to force the usage of readlines:
self.assertEqual(mock_open.return_value.readlines.call_count, 1, "You must call readlines()")
This ensures that the call count of
readlines() is 1.
So the final test file would be something like:
from unittest import TestCase, mockfrom exercise import parse # Import parse function only (will fail if not declared)class Evaluate(TestCase):@mock.patch('builtins.open')def test_parse_single_line(self, mock_open):mock_open.return_value.readlines.return_value = ['Ned|Stark|10/10/10']result = parse()self.assertEqual(mock_open.return_value.readlines.call_count, 1, "You must call readlines()")self.assertEqual(['Ned'], result, "Feedback about the mistake")@mock.patch('builtins.open')def test_parse_multiple_lines(self, mock_open):mock_open.return_value.readlines.return_value = ['Ned|Stark|10/10/10', 'Jon|Snow|11/11/11', 'Cersei|Lannister|10/11/12']result = parse()self.assertEqual(mock_open.return_value.readlines.call_count, 1, "You must call readlines()")self.assertEqual(['Ned', 'Jon', 'Cersei'], result, "Feedback about the mistake")
You can find more information and examples in the official documentation page.
In this section, we will give an example of mocking class methods and other functions to change the behaviour.
Consider the exercise is about Python exceptions and we want students to call a given function in a
try / except block and print the exception message.
A sample solution is given below:
def method_from_instructor():# This method is given by instructor# Students shouldn't change itraise Exception("This is an exception message")def method_for_students():# This method is for students to edittry:method_from_instructor()except Exception as e:print(e)
While writing the test for this exercise, our purpose is to make sure students don't just copy / paste and print the exception message. So one way would be mocking
method_from_instructor() and raising an exception with a different message.
import sysfrom unittest import TestCase, mockclass Evaluate(TestCase):@mock.patch('exercise.method_from_instructor')def test_method_for_students(self, instructor_mock):import exercisedef mock_method():raise Exception("Another random exception message")instructor_mock.side_effect = mock_methodexercise.method_for_students()self.assertEqual("Another random exception message\n", sys.stdout.getvalue(),"You must print exception message")
This evaluation code overrides the
method_from_instructor() and makes sure it raises an exception with a different message. To do that, we basically implemented another function
mock_method() and set it as
method_from_instructor() mock. It means that
mock_method() function will be executed when
method_from_instructor() is called.
Thus, even if students just copy / paste the exception message and print it, it wouldn't work because we are raising the exception with a different message inside
For further information, we strongly recommend you to read the official mock documentation of Python.
How would you mock a class method? It is almost the same with mocking module functions. We need to change the example given in above section but the purpose is the same.
Consider that we ask students to implement a method in a class which calls another method in the same class provided by us. We don't want students to be able to edit our method to pass the quiz.
Let's say the exercise is concatenating two strings retrieved from two different methods. The solution is given below:
class Customer:def get_full_name(self):# This is the function to be edited by the studentsreturn self.get_first_name() + " " + self.get_last_name()def get_first_name(self):# This is provided by the instructorreturn "Arya"def get_last_name(self):# This is provided by the instructorreturn "Stark"
Our purpose in this example is to make sure students don't just copy / paste the given names. So we will mock two methods provided by the instructor and return different names while testing.
import sysfrom unittest import TestCase, mockfrom exercise import Customerclass Evaluate(TestCase):@mock.patch.object(Customer, 'get_first_name')@mock.patch.object(Customer, 'get_last_name')def test_method_for_students(self, last_name_mock, first_name_mock):last_name_mock.return_value = 'Lannister'first_name_mock.return_value = 'Tyrion'customer = Customer()full_name = customer.get_full_name()self.assertEqual("Tyrion Lannister", full_name,"You must return full name")
What is different here? We used
@mock.patch.object instead of
@mock.patch to patch the function and passed
Customer class as the first argument to the decorator. Besides that, it's only a different version of the example in the previous section.
When you want to test code that reads user input, you will need to mock the
stdin. Let's see an example:
a = int(input())b = int(input())print(a + b)
This code gets two numerical user inputs and prints their sum. To test this code with various inputs, we will need to mock the stdin to provide sample user inputs, separating them with a newline character
'\n'. An example test looks like:
import sysfrom unittest import TestCaseimport ioclass Evaluate(TestCase):def test_exercise(self):sys.stdin = io.StringIO('1\n3')import exerciseoutput = sys.stdout.getvalue()self.assertEqual('4\n', output,'Fails for the arguments 1 and 3')
You can find more details in the official mock documentation of Python.