lab01 : Writing Test Cases
|lab01||true||Writing Test Cases||Tue 01/14 12:00AM||Tue 01/21 08:59AM|
lab01: Writing Test Cases
In this lab, you’ll practice:
- Creating a file that has some functions in it
- Testing those functions interactively at the Python prompt
- Creating a file of automatic test cases for those functions
- Running those test cases
- Submitting your functions and test cases to Gradescope for grading
Set up your working environment
Before you get started, remember to arrange and resize your windows like you did in the “Quick Detour” in Lab00, so that you can see both the browser (these instructions) and your files side-by-side.
You can view the accompanying slides to lab01 here.
Step 0: Install pytest for your account (or on your machine)
This lab is one that you may find you need to do on the CSIL machines.
It’s important to differentiate between the Python shell
>>> vs the terminal
It is probably the case that
pytest is not installed for your version
of Python3. You can check by typing
python3 at the Terminal prompt
to get to the Python Shell Prompt
>>>, and then typing
If you get an error message like the one shown below, then pytest is not installed. However, if you
do not get an error message, skip to Step 1.
[cgaucho@csil-12 ~]$ python3 Python 3.4.3 (default, Aug 9 2016, 15:36:17) [GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pytest Traceback (most recent call last): File "", line 1, in ImportError: No module named 'pytest' >>> exit() [cgaucho@csil-12 ~]$ pip3 install --user pytest
In order to exit the python shell in terminal press Ctrl+D or type exit() in order to return to the normal terminal.
To install pytest, type this command into the terminal session (the Unix Terminal prompt) to install pytest for your CSIL account:
pip3 install --user pytest
To install pytest on Windows, see this tutorial.
You can also try this command on Mac. It might work, or it might not. If it does—great, then you can do this lab on your own machine. If not, then you’ll need to do it on CSIL.
The output should look something like this:
[cgaucho@csil-12 ~]$ pip3 install --user pytest You are using pip version 7.1.0, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting pytest Downloading pytest-3.2.1-py2.py3-none-any.whl (186kB) 100% 188kB 1.5MB/s Collecting py>=1.4.33 (from pytest) Downloading py-1.4.34-py2.py3-none-any.whl (84kB) 100% 86kB 2.0MB/s Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python3.4/site-packages (from pytest) Installing collected packages: py, pytest Successfully installed py pytest [cgaucho@csil-12 ~]$
To tell whether it worked or not, try the
import pytest command again:
[cgaucho@csil-12 ~]$ python3 Python 3.4.3 (default, Aug 9 2016, 15:36:17) [GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pytest >>>
The lack of an error message (just another
>>> prompt) means “it worked!”.
We are going to use the Python prompt in the next step anyway, so just stay
at the Python prompt. (Or if you need to get out of Python, use CTRL-D to return
to the Unix shell prompt.)
Step 1: Warmup–experiencing floating point inaccuracy
In this step, you will see how to import another Python module (
math) and look at the inaccuracies that we can run into when working with floating point numbers (
floats). You will learn how to plan for this inaccuracy and will be testing these commands on the Python shell before starting the lab assignment in a new file.
If you are not already at the Python prompt, bring up a terminal window, and type
idle3. This should give you the Python Shell Prompt (
>>>) where you can type in some expressions and see the resulting values.
Type in the
import math, followed by
math.sqrt(2). It should look like this:
>>> import math >>> math.sqrt(2) 1.4142135623730951 >>>
Note that the
import statement gives us access to
math.sqrt(). The name that comes after the keyword
import is called a library or a module (here we are importing the
math library). This allows the program to use other code like
math.sqrt() that we didn’t write ourselves (someone else wrote it for us). There are a lot of libraries that come with programming language (such as
math), and some libraries that can be downloaded and imported into your program (see Section 1.2 in Perkovic). In either case, using libraries helps developers focus on solving their problems and manage their code without having to re-implement certain functionality (like making a sqrt function from scratch).
The value we get back is the square root of 2, which is an irrational number—its decimal representation goes on forever. Unfortunately, real world computing devices typically store numbers with a finite number of decimal places†. So, the representation we see for the square root of two is an approximation.
We can see this if we multiply
math.sqrt(2) by itself. Try it:
>>> math.sqrt(2) * math.sqrt(2) 2.0000000000000004 >>>
See that pesky
4 digit in the ten quadrillionths place? My goodness, we are really, really close to 2.0, but if we ask whether the values are equal, Python says no:
>>> math.sqrt(2) * math.sqrt(2) == 2.0 False
In fact, Python is very clear about the difference between
math.sqrt(2) * math.sqrt(2), and can even tell us
how big that difference is. The
4 digit is only the tip of the very, very, very small iceberg:
>>> math.sqrt(2) * math.sqrt(2) - 2.0 4.440892098500626e-16 >>>
We can deal with this in our code by subtracting the expected value from the result, taking its absolute value, and finally checking if it is less than some tolerance. This might come in handy in your future coding experiences, but for now we’ll leave it here.
>>> a = 5 >>> b = 5.00001 >>> a == b False >>> abs(a - b) < .001 True
We should plan to program with the knowledge that some mathematics operations are approximations (especially irrational numbers).
When we test software involving floating point numbers, we must allow for some inaccuracy. This “allowable inaccuracy” is sometimes called the tolerance, and it might be a small value such as
0.001 (1x10-3, or
In Python, we can write
1e-06. (The lowercase
e is the way that Python represents scientific notation.)
You can see that Python, by default, formats numbers in this notation once the fifth decimal place is reached:
>>> 0.01 0.01 >>> 0.0001 0.0001 >>> 0.00001 1e-05 >>> 0.000001 1e-06 >>>
We’ll come back to that idea later in this lab.
As a reminder, CTRL-D gets you out of the Python Shell Prompt (
and returns you to the Unix shell prompt.
Step 2: Make a
In the previous lab, you should already have created the
~/cs8/lab00. You are now going to create a folder
~/cs8/lab01 for the files for this lab.
In general, it is a good idea to keep your work for this class
~/cs8, in folders called
This isn’t exactly required (no-one is going to check), but it is a good habit to develop. Here’s why: all the rest of the instructions will be written based on the assumption that you did things this way. And if you continue to take CS courses at UCSB, you’ll often find that’s the case from one course to the next. Plus, it helps you organize your files from different labs.
So, we strongly encourage you to do it.
As a reminder:
cdreturns you to your home directory (e.g.,
/cs/student/cgaucho/, if your username is
pwdprints your current working directory
cd cs8changes into the cs8 directory if
cs8is in the current directory
cd ~/cs8changes into the cs8 directory under your home directory regardless of the current working directory (because
~is a shortcut for the absolute path of your home directory.)
cd ..will move you up one directory from your current directory.
mkdir lab01will create a lab01 directory under your current working directory
lslists the files in your current directory
With that information, you should be able to determine how to
~/cs8/lab01 directory, and make that directory your current
working directory (the one that comes up when you type
pwd.) Do that now. (Remember that creating and navigating directories happens using the terminal shell prompt, also known as the Unix command line, marked by
Then, you are ready for the next step.
Step 3: Create a file called
convert.py in your
The contents of
convert.py should be as shown below: it should contain three Python function definitions that, at the moment, are incorrect.
Choose “File => New File” in IDLE to bring up an “untitled” window, then typese the functions into that window (i.e., in the new file). Note that the formulas for converting between Feet, Inches, and Cm are incorrect. That’s deliberate, so just go with it for now. We’ll fix those at a later step.
def ftToIn(feet): ''' TODO: add the docstring ''' return (feet * 0) #TODO: Fix this line of code def inToCm(inch): ''' TODO: add the docstring ''' return (inch * 0) #TODO: Fix this line of code def ftToCm(feet): ''' TODO: add the docstring ''' return (feet * 0) #TODO: Fix this line of code
The code above is how we define functions in Python. We will work on defining our own functions throughout the course.
IMPORTANT: Make sure to save your file with this exact filename:
convert.py. Naming it with some other name will confuse Gradescope and result in a score of 0, even if your work is correct.
On a very high level, the
def inToCm(inch), and
def ftToCm(feet) are what we call a function header/signature.
- In the function signature of
def ftToIn(feet), the keyword
deftells python we are defining a function,
ftToInis the name of the function and
feetis the function’s parameter enclosed in parentheses (note, there can be zero or more parameters for a function, but in this case there is only one). You can think of parameters as variables that the function can use (note that calling
ftToIn(0)is telling the computer to call
feet = 0).
returnkeyword is used to pass back a value to whoever called the function.
- In this case, the function
ftToIn(feet)returns the inch value of a feet measurement we passed into the function (using the input argument
- The function
inToCm(inch)returns the centimeter value of an inch measurement we passed into this function (using the input argument
- The function
ftToCm(feet)returns the centimeter value of a feet measurement we passed into this function (using the input argument
- In this case, the function
Step 4: Test your code by hand
To test this code, we’ll first do what many programmers do: test the code by hand.
That is, we’ll run the file in IDLE, and then “call the function” (i.e. enter some function calls in the Python shell) to see what results we get. These functions are supposed to convert between feet, inches, and centimeters. Let’s see if they work properly when you are given 0 feet and 0 inches.
Use the “Run Module” menu command to run the code: doing so will define the functions (i.e., make Python aware that they exist and what they are supposed to do according to the provided function definitions). You can then try entering these function calls at the Python Shell prompt. You should see output similar to what is shown below:
>>> ftToIn(0) 0 >>> inToCm(0) 0 >>> ftToCm(0) 0
As you can see, for those three particular values, the function appears to return the correct answer: 0 feet is 0 inches, 0 inches is 0 centimeters, and 0 feet is indeed 0 centimeters. So clearly, testing with a single value is not enough. Indeed, if we test with another value, such as 5 feet and 60 inches we see that the output is incorrect:
>>> ftToIn(5) 0 >>> inToCm(60) 0 >>> ftToCm(5) 0
One of the problems with testing by hand is that it is tedious, and folks have a tendency to skip it. So, experienced programmers have learned that it is generally a better idea to automate the process of testing. We’ll learn how to do that next. We’ll see that when we set up six tests that we ran above: three of them will pass, and three of them will fail.
Step 5: Setting up automated tests
Add this line at the top of your
Then, add the following code to your
convert.py file after the function definitions for
ftToCm to define six automated tests:
def test_ftToIn_zero(): assert ftToIn(0.0) == pytest.approx(0.0) def test_inToCm_zero(): assert inToCm(0.0) == pytest.approx(0.0) def test_ftToCm_zero(): assert ftToCm(0.0) == pytest.approx(0.0) def test_ftToIn_five(): assert ftToIn(5.0) == pytest.approx(60.0) def test_inToCm_60(): assert inToCm(60.0) == pytest.approx(152.4) def test_ftToCm_five(): assert ftToCm(5) == pytest.approx(152.4)
These are automated tests that use a module(library) known as
pytest. When defining tests using the pytest library, we typically define functions that:
- have names that start with
- include the name of the function they are testing,
- end with exactly one
assertstatement–that is, the keyword
assertfollowed by a boolean expression.
If the expression after
assert is true, the test passes, otherwise it fails, and gives us an error message which explains what went wrong.
Note that all tests for our function
ftToIn will begin with
test_ftToIn_. In order to differentiate between the different test cases for the same function, we will need to include something else in the name of the test function that helps us test different cases. This is why we added
five to the end of the test function’s name, so that we have two test functions that each test a different case:
test_ftToIn_zero: tests the result of calling
test_ftToIn_five: tests the result of calling
Similarly for the other functions and their tests.
We are using
pytest.approx() here because any time you are testing with floating point numbers.
As we discussed earlier, we have to be aware that there may be some inaccuracy. Using
pytest.approx() is a cleaner way of subtracting the expected value from the actual value, taking the absolute value, and checking if it is less than some tolerance which we touched on in Step 1. Using
pytest.approx() allows for a tolerance of
(Recall our discussion of what happens when you multiply
math.sqrt(2.0) by itself. Here, it is probably overkill since we are not using any irrational numbers, but it is still safer to always use some way of approximating equality when dealing with floating point.)
Your entire file should look as follows at this point:
import pytest def ftToIn(feet): ''' TODO: add the docstring ''' return (feet * 0) #TODO: Fix this line of code def inToCm(inch): ''' TODO: add the docstring ''' return (inch * 0) #TODO: Fix this line of code def ftToCm(feet): ''' TODO: add the docstring ''' return (feet * 0) #TODO: Fix this line of code def test_ftToIn_zero(): assert ftToIn(0.0) == pytest.approx(0.0) def test_inToCm_zero(): assert inToCm(0.0) == pytest.approx(0.0) def test_ftToCm_zero(): assert ftToCm(0.0) == pytest.approx(0.0) def test_ftToIn_five(): assert ftToIn(5.0) == pytest.approx(60.0) def test_inToCm_60(): assert inToCm(60.0) == pytest.approx(152.4) def test_ftToCm_five(): assert ftToCm(5) == pytest.approx(152.4)
After entering this, save the file and use “Run Module” to make sure there are no error messages (error messages are displayed as red output in the Python Shell Window). If there are any, make sure to fix them before running the next step.
Step 6: Running the test cases
Running the test cases requires us to go outside of IDLE back to the terminal shell prompt. Make sure to use
quit() or Ctrl-D to exit the python shell (
>>>). Once you are back on the command line prompt (that has a
$) continue on.
Your current directory needs to be the same one that your
convert.py file is stored in. That should be
~/cs8/lab01, but if it is not, then fix things so that the
convert.py file is in that directory, and your current working directory is set to that directory. If you need help, ask for assistance.
You should be able to type the
ls command (or on Windows,
dir) at the terminal shell prompt and see the
convert.py file listed.
your-prompt-here $ ls convert.py your-prompt-here $
When that is the case, type this command after the
$ in your terminal window:
python3 -m pytest convert.py
You should see output like what is shown below. It may be a little overwhelming at first. Once you know what you are looking for, it is very easy to read. After the output, there is a guide to understanding it.
169-231-175-204:lab01 cgaucho$ python3 -m pytest convert.py ================ test session starts ================= platform darwin -- Python 3.6.2, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 rootdir: /Users/cgaucho/cs8/lab01, inifile: collected 6 items convert.py ...FFF [100%] ====================== FAILURES ====================== __________________ test_ftToIn_five __________________ def test_ftToIn_five(): > assert ftToIn(5.0) == pytest.approx(60.0) E assert 0.0 == 60.0 ± 6.0e-05 E + where 0.0 = ftToIn(5.0) E + and 60.0 ± 6.0e-05 = <function approx at 0x106853940>(60.0) E + where <function approx at 0x106853940> = pytest.approx convert.py:23: AssertionError ___________________ test_inToCm_60 ___________________ def test_inToCm_60(): > assert inToCm(60.0) == pytest.approx(152.4) E assert 0.0 == 152.4 ± 1.5e-04 E + where 0.0 = inToCm(60.0) E + and 152.4 ± 1.5e-04 = <function approx at 0x106853940>(152.4) E + where <function approx at 0x106853940> = pytest.approx convert.py:26: AssertionError __________________ test_ftToCm_five __________________ def test_ftToCm_five(): > assert ftToCm(5) == pytest.approx(152.4) E assert 0 == 152.4 ± 1.5e-04 E + where 0 = ftToCm(5) E + and 152.4 ± 1.5e-04 = <function approx at 0x106853940>(152.4) E + where <function approx at 0x106853940> = pytest.approx convert.py:29: AssertionError ============ 3 failed, 3 passed in 0.07s ============= 169-231-175-204:lab01 cgaucho$
Ok, let’s now break down this output.
Step 7: Understanding the output of pytest
Here’s how to understand
Get the big picture first from the
Look for a line near the beginning that has the name of your file, followed by a list of either dots, letter
Fcharacters, like this one:
In this case, the
.characters are tests that passed. If you have four tests, then ideally, you’d want to see four dots, like this
convert.py ...., which means that four tests for convert.py all successfully passed.
Instead, here, we see
...FFF, which means we had three tests that succeeded and three tests that failed. Later in the output, we’ll see more detail about each of those failures.
Understand the overall structure of the output.
The rest of the output will be divided into sections, one for each failure. Here is what it look like with the details taken out so that you can see the big picture more easily:
================ test session starts ================= (blah blah here that you can ignore) convert.py ...FFF ====================== FAILURES ====================== __________________ test_ftToIn_five __________________ ... (details about the first test case failure are here) ... ___________________ test_inToCm_60 ___________________ ... (details about the second test case failure are here) ... __________________ test_ftToCm_five __________________ ... (details about the third test case failure are here) ... ============ 3 failed, 3 passed in 0.07s =============
Note that the last line gives us a nice summary:
3 failed, 3 passed in 0.03 seconds.
We now know that
we need to focus on the three failures.
And the headers tell us which test cases failed, namely
So let’s focus on those next, by first looking in detail at the first one:
Focus in on the first test case failure.
Let’s focus just on the detailed output for the first test case failure,
test_ftToIn_five. What does all of the detailed output mean?
Below is a breakdown of why the assertion turned out to be false, showing every step in the calculation. Let’s break it down one line at a time:
line of output meaning
first line of the failing test case
E assert 0.0 == 60.0 ± 6.0e-05
This is the assertion that turned out not to be true
E + where 0.0 = ftToIn(5.0)
This tells us why
0.0was on the left hand side of the
==. It was the result of computing
E + and 60.0 ± 6.0e-05 = <function approx at 0x106853940>(60.0)
This tells us why
60.0 ± 6.0e-05was on the right hand side of the
==. It shows the expected value (
60.0) and the tolerance (
E + where <function approx at 0x106853940> = pytest.approx
This tells us that
pytest.approxwas used to calculate the tolerance.
This shows which line in the file
convert.pyhad the failed assertion, namely, line 23. That helps us find the error faster if we are dealing with a large file of code.
If we look at this, we can see that amidst all of the clutter is the crucial fact that
0.0when what we were expecting was
60.0, with a tolerance of
Step 8: Fixing the code
So, if you have failing test cases, the thing to do is fix the code so that the test cases pass.
To do that you’ll need to correct the formulas.
Keep in mind that in Python:
*symbol is used for multiplication. In algebra, we can write
x, however, this does not work in Python. In Python you must write
xif you want to multiply the variable
aif you want to multiply the variable
countif you want to multiply the variable
count(which could encode any value (i.e.
-symbols are used for addition and subtraction
/symbol is used for division, e.g ‘9/5’ means nine divided by five.
Also, the order of operations in Python is that multiplication and division are done before addition and subtraction. Some examples:
xis 5, then
x + 2 * 3gives us 11, not 21. The multiplication is performed before the addition.
xis 16, then
x - 6 / 2gives us 13, not 5. The division is performed before the subtraction.
If you want to force the addition or subtraction to be done first, you must use parentheses, e.g.,
(x + 2) * 3or
(x - 6) / 2.
Use the following conversions to correct the formulas in your functions:
1 inch = 2.54 cm 1 foot = 12 inches
When you replace return
feet*0 with the correct formula for converting feet measurement to inches, you should remove the comment that says
# TODO: Fix this line of code and update the docstring accordingly (briefly explain what the input is and what the function returns).
You’ll also want to replace the similar line in the other functions as you fix them.
When you have the test cases passing, try running the pytest command again (shown below)—remembering that:
- it must be done from the Unix terminal shell, NOT the Python shell.
- the current working directory of that terminal session must be
the directory/folder where your
convert.pyfile is located
python3 -m pytest convert.py
When you see all passing tests, your output will look like this. Also, the last line will be a pleasant shade of green instead of an angry looking red.
169-231-175-204:lab01 cgaucho$ python3 -m pytest convert.py ======================== test session starts ========================= platform darwin -- Python 3.6.2, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 rootdir: /Users/cgaucho/github/cs8/lab01, inifile: collected 6 items convert.py .... ====================== 6 passed in 0.04 seconds ====================== 169-231-175-204:lab01 cgaucho$
Step 9: Submit your
convert.py file to Gradescope
Before you submit your code to Gradescope, make sure that your file
- is named
- does not have any
TODOcomments left in the code.
- contains a comment at the top of the file that says
Submitted by <name>, PERM, for CS 8 (W20). (Please, replace
PERMwith your first and last name and PERM number respectively.)
You are ready to submit your work to Gradescope.
Navigate to the Lab assignment “Lab01” and upload your
convert.py, similar to how you submitted
hello.py for Lab00. Gradescope will check if your functions are correct. If your tests do not pass, go back to these functions and double-check your conversion formulas and function syntax.
Final Step: Log Out
Actually, this is the final step of every lab.
In fact, you should do this every time you walk away from a lab computer, either in Phelps 3525 or CSIL.
- Find the System Menu at the top right of the screen.
- Select your name and in the dropdown click on “Log Out”
Acknowledgment: Special thanks to Sara Mandic who helped to update this lab.