To evaluate student solutions written in Swift, you need to write tests using XCTest.
Let's start with a basic exercise. Your student has to implement a HelloWorld
class with a function helloWorld()
that returns the string "Hello World". For the sake of simplicity, it doesn't print anything.
Here is an example solution:
Exercise.swift
import Foundationclass HelloWorld {func helloWorld() -> String {return "Hello World"}}
And this is an example test of the student's code:
Evaluate.swift
import XCTest@testable import Base​class Evaluate: UMBaseTestCase {var hw = HelloWorld()​func testHelloWorld() {XCTAssertEqual(hw.helloWorld(),"Hello World","You should print 'Hello World'")}}
If you are using Swift 3
, add the following snippet after that:
extension Evaluate {static var allTests : [(String, (Evaluate) -> () throws -> Void)] {return [("testHelloWorld", testHelloWorld)]}}
Let's examine the above test code.
import XCTest@testable import Base
Your code must include those lines to import the XCTest
framework and our module.
class Evaluate: UMBaseTestCase {​}
Your test code must live inside a class named Evaluate
. It is, otherwise, just a plain Swift class.
UMBaseTestCase
inherits from XCTestCase
. It provides some additional helper methods to manipulate Standard Input and Output. The Evaluate
class can inherit from UMBaseTestCase
or directly XCTestCase
.
If you are using Swift 3
, in order to run your test methods, you must add all of them to allTests
. If you are using Swift 5
, you can skip the section below, because we run the tests with --enable-test-discovery
. The test method's names names should comply with a naming convention testFoo()
extension Evaluate {static var allTests : [(String, (Evaluate) -> () throws -> Void)] {return [("testHelloWorld", testHelloWorld)]}}
XCTAssertEqual
is a method provided by XCTest
which asserts that two expressions have the same value. You can use it with 3 arguments:
Actual value
Expected value
Feedback message to be shown if values are not equal
You can see other assertions here.
Our test class has a test case testHelloWorld()
which asserts equality of "Hello World" and the return value of the student's function.
Let's change the previous example a little bit and ask students to print "Hello World" rather than return the string.
Here is an example solution:
Exercise.swift
class HelloWorld {func helloWorld() {print("Hello World")}}
To be able to test the student code, you must be able to read standard output after calling helloWorld()
. For this purpose, we have provided helper methods on UMBaseTestCase
.
For example:
Evaluate.swift
import XCTest@testable import Base​class Evaluate: UMBaseTestCase {var hw = HelloWorld()​func testHelloWorld(){self.prepareStdOut()hw.helloWorld()XCTAssertEqual(self.getOutput(),"Hello World\n","You should print 'Hello World'")}}
If you are using Swift 3
, add the following snippet after that:
extension Evaluate {static var allTests : [(String, (Evaluate) -> () throws -> Void)] {return [("testHelloWorld", testHelloWorld)]}}
There are a few differences between this test and the previous one:
We are using helper methods to gain access to data written to standard output.
Calling self.prepareStdOut()
redirects standard output to a temporary log file to be able to read later.
Comparing the "Hello World" string with the output that we read with self.getOutput()
.
The getOutput()
method reads the output from the log file and restores the standard output to previous state, therefore we can get feedback from test results.
You can see more details about helper methods below in other examples.
How do you test whether students are able to read input from standard input?
Consider that the student's task is to read a customer's first and last name from standard input and print the full name to standard output.
The solution for this problem can be implemented as follows:
Exercise.swift
import Foundationclass Customer {var name:String!​func getName() -> String {​print("Enter your name:")self.name = self.readString()return self.name}​private func readString() -> String {let str = String(data: FileHandle.standardInput.availableData,encoding:String.Encoding.utf8)!.trimmingCharacters(in: CharacterSet.whitespacesAndNewlines)return str}}
UMBaseTestCase
has a method to fill standard input with given values.
func setStdInput(input:)
inputs
: String input to be entered to standard input. A line separator will be appended to each input.
The following test class enters name in the console and checks whether the student's code can return the name.
Evaluate.swift
import XCTest@testable import Base​class Evaluate: UMBaseTestCase {var customer = Customer()​func testGetName(){let str = "Albert Einstein"self.setStdInput(input:str)var name = customer.getName()XCTAssertEqual(name,"Albert Einstein","For inputs 'Albert Einstein' it doesn't work.")}}
If you are using Swift 3
, add the following snippet after that:
extension Evaluate {static var allTests : [(String, (Evaluate) -> () throws -> Void)] {return [("testGetName", testGetName)]}}
You can expect student code to raise exceptions in some cases.
So let's improve the example above and ask students to throw IllegalArgumentException
if the input passed to a function is not valid. setSSN()
won't accept SSNs shorter than 10 characters and setAge()
won't accept negative ages. And if the age is negative, the exception message must be "Age can't be a negative number.".
The solution for this problem in Swift 3
can be implemented as follows:
Exercise.swift
enum CustomerError: Error {case IllegalArgumentError}​class Customer {var SSN:String?var age:Int?​func setSSN(ssn:String) throws {if (ssn.characters.count < 10) {throw CustomerError.IllegalArgumentError}self.SSN = ssn}​func setAge(age:Int) throws {if (age < 0) {// Error with messagethrow CustomerError.IllegalArgumentError}self.age = age}}
The solution for this problem in Swift 5
can be implemented as follows:
Exercise.swift
enum CustomerError: Error {case IllegalArgumentError}​class Customer {var SSN:String?var age:Int?​func setSSN(ssn:String) throws {if (ssn.count < 10) {throw CustomerError.IllegalArgumentError}self.SSN = ssn}​func setAge(age:Int) throws {if (age < 0) {// Error with messagethrow CustomerError.IllegalArgumentError}self.age = age}}
To check whether the student handled exception cases, you can use XCTAssertThrowsError
. The following test class is checking exceptions.
Evaluate.swift
import XCTest@testable import Base​class Evaluate: UMBaseTestCase {var customer = Customer()​func testInvalidSSN() {// Check if it raises exception for invalid Social Security NumberXCTAssertThrowsError(try customer.setSSN(ssn:"123"),"You should throw IllegalArgumentError")}​func testInvalidAge() {// Check if it raises exception for negative ageXCTAssertThrowsError(try customer.setAge(age:-5),"You should throw IllegalArgumentError")}}
If you are using Swift 3
, add the following snippet after that:
extension Evaluate {static var allTests : [(String, (Evaluate) -> () throws -> Void)] {return [("testInvalidSSN", testInvalidSSN),("testInvalidAge", testInvalidAge)]}}
You can also use a try
/ catch
block to check whether an exception is thrown and call XCTFail
if it is not.
For Swift 5
we support splitting the exercises in multiple files. You will still have to make a single evaluation file covering the tests cases you need, but you might have several separate test cases.
For example, if we have Solution with multiple files with classes that perform the simple arithmetic operations like subtraction and addition:
ExerciseAdd.swift
class AppAdd {func add(a:Int, _ b:Int) -> Int {return a + b}}
and
ExerciseSub.swift
class AppSub {func sub(a:Int, _ b:Int) -> Int {return a - b}}
Then you can have a single Evaluation File with multiple Test Cases EvaluateAdd
and EvaluateSub
:
Evaluate.swift
import XCTestimport Foundation@testable import Base​class EvaluateAdd: UMBaseTestCase {var appAdd: AppAdd!​override func setUp() {super.setUp()​appAdd = AppAdd()}​func testAddCheck() {XCTAssertEqual(appAdd.add(a:1, 1), 2)}​}​class EvaluateSub: UMBaseTestCase {var appSub: AppSub!​override func setUp() {super.setUp()​appSub = AppSub()}​func testSubCheck() {XCTAssertEqual(appSub.sub(a:5, 3), 2)}}
Let's see all the details of the UMBaseTestCase
class which is provided by Udemy to help you deal with reading outputs and entering inputs for student solutions.
UMBaseTestCase.swift
import XCTestimport Foundation@testable import Base​class UMBaseTestCase: XCTestCase {​var savedStdOut:Int32!var savedStdIn:Int32!​let stdInLogFile = "/eval/stdIn.log"let stdOutLogFile = "/eval/stdOut.log"​var isStdOutRedirected:Bool!var isStdInRedirected:Bool!​override func setUp() {super.setUp()isStdOutRedirected = falseisStdInRedirected = false}​override func tearDown(){super.tearDown()if(isStdOutRedirected == true){self.restoreStdOutToDefault()}if(isStdInRedirected == true){self.restoreStdInToDefault()}}​// Redirect StandardInput in order to read from log filefunc setStdInput(input:String){do {try input.write(toFile:stdInLogFile, atomically: true, encoding: String.Encoding.utf8)} catch {// failed to write file}isStdInRedirected = truesavedStdIn = dup(STDIN_FILENO)freopen(stdInLogFile, "r", stdin)}​// Redirect StandardOutput to log filefunc prepareStdOut() {isStdOutRedirected = truesavedStdOut = dup(STDOUT_FILENO)freopen(stdOutLogFile, "w", stdout)}​// Read from StandardOutput from log filefunc getOutput()->String {// Restore standartOutput// This is important to restore to get Test resultsself.restoreStdOutToDefault()let content = try! String(contentsOfFile: stdOutLogFile, encoding: String.Encoding.utf8)return content}​private func restoreStdOutToDefault(){isStdOutRedirected = falsefflush(stdout)dup2(savedStdOut, STDOUT_FILENO)close(savedStdOut)}​private func restoreStdInToDefault(){isStdInRedirected = falsedup2(savedStdIn, STDIN_FILENO)close(savedStdIn)}}