Websheets Manual
This manual describes how to write new exercises. It doesn't
cover installation — for that, see the README or the
git repository.
Format Overview
If you look in the
exercises directory,
you will see a bunch of
.py files. Each one of them defines a single Websheet exercise. Here is the
simplest example — it is
TwoPlusTwo.py:
source_code = r"""
public static void main(String[] args) {
\[
System.out.print(2+2);
]\
}
"""
description = r"""
Fill in the program so it <i>prints</i> out the integer obtained by adding 2 and 2.
"""
tests = r"""
testMain();
"""
This defines a Python module, using triple-quotes so that the strings can span multiple
lines, and using the
r""" raw-string prefix so that you don't need to escape backslashes.
However, that is all the Python you need to know.
See this definition in action by visiting the TwoPlusTwo
websheet. Try making some correct and incorrect submissions.
So at its core, a websheet is a list of field names, each taking on a string value.
Reviewing the above, you must define the fields
- source_code, a correct reference Java program with
some areas \[ blanked out ]\ to be filled in by the student
- anything not between \[...]\ delimiters is read-only (try it!)
- description, an HTML fragment that will be shown to the student,
that explains what they should do.
- tests, a snippet of Java code calling the testing API,
which compares the behaviour of the student's solution and the reference solution
There are a few other optional fields beyond these three.
Here are the sections of this manual, which describe how to write each
component in more detail:
Of course, looking at existing examples is the most thorough way to understand
what's going on, once you're familiar with the main concepts.
source_code: The reference solution and blank locations
When writing a websheet, normally you will start by writing
the
source_code as a correct solution to the problem. (Note that
you can either write in
public class ExerciseName like
HelloWorld.py
or let it be filled in implicitly, like
TwoPlusTwo above.)
Once you fill in a working program, you need to decide on what parts
you want to blank out for the student.
For the purposes of websheets, every blank has to
fall in one of the following two categories:
- inline blanks which start and end on the same line; they can expand horizontally, but can never contain newlines
- multi-line blanks which take up the entirety of several lines; they can expand vertically, but never can start or
end in the middle of a line
For example, the
Distance websheet has both kinds of blanks.
Try typing some stuff in the blanks and see how they resize to accomodate unlimited student input.
Both kinds of blanks are indicated by using \[ to denote the start of the blank,
and ]\ to denote the end of the blank. The interior region will be left blank and editable for
the user.
The rules for how these delimiters must appear in source_code are:
- inline blanks are defined by \[ and \] on the same line
- multi-line blanks are defined by \[ on a line by itself, followed some lines later by ]\
on a line by itself.
For example, look at how
Distance.py
defines both kinds of blanks.
(In the multi-line case, additional whitespace on the lines
containing the delimiters is currently ignored, although it
could be meaningful in a later version, say where Python is used and
we want to force indentation of a region.)
Hidden code, fake code, default contents for blanks
The simple
\[...]\ delimiters are sufficient to describe
a large variety of exercises. But to allow the exercise writer
more flexibility, websheets allow a few additional kinds of regions.
You can use \hide[...]\ to enter code that will be hidden
from the student's UI. It will be included in both the reference
solution and the student's combined solution that gets executed upon submission.
For examples, see
- , where it is used in-line to hide a call to .clone()
- , where it is used multi-line to include an inner class
You can use \fake[...]\ to display read-only text in the user interface
that looks like it is part of the program, but that won't actually be included.
For examples, see
- , where it is used in-line to show pseudocode in the middle of a program
- , where it is used similarly, but to contain multiple lines
You can use \show: to have a blank initially populated with some text.
(Think of providing a buggy program fragment for a student to fix.) To do this,
change \[ blanked-out text ]\ to \[ blanked-out text \show: default text here ]\.
For example, see
- , where it is used in multi-line syntax to give a strawman constructor
- (No current example uses the in-line syntax.)
Cheatsheet: What elements appear where?
| in UI? | in reference solution? | in executed/tested combined student solution? |
plain text | yes (read-only) | yes | yes |
---|
blanked-out \[...]\ | yes (editable, initially blank) |
yes | yes (edited student text spliced in) |
\show: of blanked-out | yes (editable) | no | yes (edited student text spliced in) |
\fake[ ... ]\ | yes (read-only) | no | no |
\hide[ ... ]\ | no | yes | yes |
description: The exercise description, in HTML
You can use as few or as many HTML elements as you like in describing the
exercise. The simplest descriptions are just text, using
<p> to separate
paragraphs, and sometimes using
<tt> and
<pre> to use a monospace font.
LaTeX. All websheets have MathJaX processing turned on. This means that you can
include LaTeX using $inline format$ or $$big centered display format$$.
Look at
for examples of both.
Links. Sometimes you'll want to add a link to another websheet inside
of the description. One way to do this nicely is to write a link with a URL like
<a href="?group=SquareSwap">SquareSwap</a>
E.g. see
.
Diagrams.
You can use any software and any formats you like, but the example exercises
using diagrams
(e.g.
,
)
were created using IPE (which uses PDFs as an editable format)
and rasterized to PNG using render.sh.
tests: What the grader should test
The
tests contains a snippet of Java code that makes calls to
our testing API. These API calls cause the automatic grader to dynamically
execute both the reference code and student code in separate namespaces,
capturing their outputs and return values, and comparing the results
for correctness.
Because tests will be spliced in to a Java program, any valid Java
code can be used. For example, instead of writing
testMain("AA");
testMain("BB");
…
testMain("ZZ");
you could write
for (char ch='A'; ch<='Z'; ch++) testMain(""+ch+ch);
You could even define
local or
anonymous
classes if you want (although no current examples do this).
Basics: testing static methods
The testing methods
testMain() for testing
main,
test() for instance methods,
testConstructor() for constructors,
and
testOn() for instance methods
are used to execute
student code and execute reference code and compare the results. It handles
things like comparing return values, capturing and comparing printed output,
catching and comparing exceptions, and looking for changes to arguments
automatically.
- testMain(String... args)
- call main on these arguments.
- E.g. see
- test(String methodName, Object... args)
- call a static method named methodName on this list of arguments
-
E.g. see
- note: if there's exactly one argument, which is an array, Java requires you use
test(methodName, (Object) /* your array here */)
due to inherent ambiguity in varargs.
E.g. see . The ambiguity doesn't
exist if there are multiple arguments of which some are arrays, e.g. see
.
Basics: testing instance methods
For testing instance methods, you'll generally call a constructor defined by
the student, and then call an instance method. The typical approach involves
3 steps, like this sample from
:
saveAs = "attendance"; // (1)
testConstructor(); // (2) i.e. attendance = new Clicker();
testOn("attendance", "curr"); // (3) i.e. attendance.curr();
This means,
- (2) call the zero-argument constructor defined by the student,
- (1) save the result as a variable named attendance,
- (3) call the instance method curr with no arguments on attendance
object.
- saveAs = String (default null)
-
save the value returned by the next test as a named object.
(implementation note: we maintain two parallel namespaces for student
and reference solutions.)
-
Note:
you can save anything, not just constructor calls.
E.g. in ,
we save a method's return value:
saveAs = "scaled";
testOn("x", "times", 10.0);
- testConstructor(Object... args)
- call the constructor with this list of arguments
- testOn(String instanceName, String methodName, Object... args)
-
on the named instance, call the named instance method with this list of arguments
Here is one more tool used in conjunction with
saveAs.
-
var(String)
-
once an object is saved, you may want to pass it an argument to a method,
rather
than just calling its instance methods.
For example in , after constructing
two objects x and y,
we call x.plus(y) like so:
testOn("x", "plus", var("y"));
basically, the var() wrapper is needed to distinguish the saved variable y
from the string literal "y"
Exceptions and standard input
- expectException = boolean (default false)
- indicates that
the student code should throw an exception on the following test.
(it will catch whatever exception the reference code throws, and
expect the student code to throw the same class of exception)
-
E.g., .
-
stdin = String (default null)
-
for the next test, set the contents of
standard input to the following String
-
E.g., .
-
stdinURL = String (default null)
-
same as stdin, but fetch this webpage and use its contents.
Useful when there is a long data set posted online, and you prefer not to
copy a long text into the websheet definition
-
E.g., .
- note: use simple string literals only.
e.g. stdURL = "http://goo.gl/" + "robots.txt";
is bad, it will confuse the prefetcher that grabs the data before starting
the sandbox.
Further methods
- remark(String s)
- Print out a remark to the user explaining what the
tests are doing. Useful when you still want them to see the
automatically-generated description of the test, but you also want to
explain more context to the student.
s should be valid HTML.
c.f. title = which replaces the automatically-generated test description.
-
E.g., .
-
E.g., .
- store(Object o)
- Copy an object to the user and reference namespace. Used
for checking shallow copying in
.
-
construct(String pckg, String clss, String type_prms, Object... args)
-
Use this to create a new instance of some other class. pckg
should be a fully qualified package or null to use student/reference
code; clss should be a simple class name; type_prms
should be "" if you are not constructing a generic, or
a list of type arguments for a generic if you are (this is just shown
to the student in the description of the test since generic erasure happens
at runtime);
and
args are the consturctor arguments.
It is best seen by these two examples:
-
In we call
construct("stdlibpack", "Queue", "<Integer>")
to mimic a call to new Queue<Integer>() (the package
stdlibpack
has the COS 126 "standard" classes, and we want the Queue from
that package)
-
In we call
construct(null, "Person", "", "Tony")
to mimic a call to new Person("Tony") — Person is
in the dependencies of that websheet. Leaving pckg as
null means that the student's code uses the Person from
the student package while the reference uses the one from the reference
package.
- testNamedMain(String methodToCall, String fakeClassName, String... args)
-
Call some hidden internal method, but pretend you
are calling main(args) on some other testing class.
Used in
at
testNamedMain("exampleClientMain", "ExampleClient");
to run the hidden method
exampleClientMain but to tell the student we are running
java ExampleClient.
Note: store(),
construct()
and
testNamedMain() will
reset all options to their defaults,
like the basic
test…() methods.
Further options
- title = String (default null)
-
replace the automatically-generated description of the
next test with
the given HTML. Useful if you are using a lot of fake or hidden code
and the automatically-generated description the student would see
in testing would
make no sense.
c.f. remark() which adds extra
commentary before the next test
- E.g. .
- E.g. .
- quietOnPass = boolean (default false)
-
Don't show the next test unless the student fails it. Especially useful if you
have a lot of tests.
- E.g. .
- E.g. .
- maxOutputBytes = int (default 10000)
- Change the maximum number of bytes the student code can print
per test before it is terminated.
- E.g. .
- E.g. .
- dontRunReference = boolean (default false)
- Don't run the reference solution. Instead, rely on either
template code or hidden code that will determine if the student
passes, and which will throw an exception if they failed, but not
if they passed.
- E.g. .
- E.g. .
- Note that hidden code can make thrown exceptions look a bit nicer by using
websheets.Grader.FailException like FTPLimiter.
- cloneForStudent = boolean (default true)
- Before passing arguments to the student, clone them. So if the
student mutates the arguments, it doesn't affect the grader.
- This is turned off to check shallow copying in .
- cloneForReference = boolean (default true)
- Same as above, but for reference code. No current example.
- realTolerance = double (default 1E-4)
- Relative error accepted for outputs. (Except if reference answer is 0,
this is the absolute error accepted.) No current example.
- ignoreRealFormatting = boolean (default true)
- If false, ignore realTolerance,
also don't accept 5.00 as the same as 5.0. No current example.
- ignoreTrailingSpaces = boolean (default true)
- Ignore trailing spaces at the end of each line. No current example.
Misc
defaultOptions
To set an option permanently (as opposed to just for one test),
use syntax like
defaultOptions.quietOnPass = true;
E.g, see .
randgen
This is a Random instance
just made available for convenience. E.g., see
which calls randgen.nextInt(100) to generate a random number from 0 to 99.
Optional websheet fields
Here are a couple of other fields that are available for use in websheets
- classname
- when you want the visible and executed class name not to be the
same as the name of the overall exercise (the Python module name,
referred to internally as the slug).
-
E.g. .
- show_class_decl
-
set to False if you don't want the user to
see the class declaration.
-
E.g. .
- epilogue
- show a message or explanation after problem is completed.
-
E.g. .
- dependencies
- require student to complete some other websheet before
this one; the student's most recent correct solution to the dependency
will be made available when compiling executing this one.
-
E.g. .
-
Note: if instead you want to give student extra reference code without requiring them
to actually complete the dependency, you can use \hide instead:
-
E.g. .
- imports
- a list of classes to import (will be made visible in UI and added to reference solution)
-
E.g. .
Adding new exercises to the system
- Create a new .py file in the above format in the
exercises directory. (E.g., NewProblem.py)
- Go visit
index.php?group=NewProblem
in your browser. Or, refresh index.php
and select NewProblem from the list of exercises.