Grader config#

Each homework carries a grader_config.json in hw/<homework>/ that tells the grader what to run and what to check. The grader loads this file before grading any submission, and if it is invalid the whole homework is rejected.

1. General structure#

The config is a single object with two entries:

  • homework — the name of the homework this config refers to (text)

  • suites — an object holding all the suites

{
    "homework": "hw0",
    "suites": {
        "Aliquot": {
            "compilation": {},
            "execution": {}
        }
    }
}

2. Suites#

Suites separate the parts of an assignment that differ in nature. For example, an assignment that asks for two binaries for instance, would usually need 2 different suites.

A suite maps category names to category objects:

{
    "Pr1_suite": {},
    "Pr2_suite": {}
}

Note

For suites that grade a binary/executable that must be compiled from the students’ source, it is recommended that the first category is dedicated entirely to the compilation of the program.

3. Categories#

A category is a group of tests that share the same nature (e.g. compiling the program, executing it in one mode, executing it in another). Each category has the following fields:

  • tests — a map of test names to test objects (optional, defaults to an empty map)

  • visibility — optional, one of public, private, grouped (default private)

  • sequential — optional boolean, default false

  • dependancies — optional, a list of category names that must have passed first

  • display_name — optional, a human-readable name

3.1 visibility#

Controls how much of the results the student sees:

  • public — the student can see the failure details (the report points them at the runner logs)

  • private (default) — the student only sees pass/fail, with no knowledge of what is actually the reason for failure.

  • grouped — the category is reported as a single row in the summary with no per-test breakdown; useful when a test needs several commands to run and the intermediate ones are of no use to the student

{
    "category1": {
        "visibility": "grouped"
    }
}

3.2 sequential#

A quality-of-life option: each test in the category automatically depends on the successful execution of the previous one, instead of needing an explicit dependancies entry per test.

{
    "category1": {
        "sequential": true
    }
}

3.3 dependancies#

Limits the tests performed based on the completion status of other categories. It is a list of category names; if any of them has not passed, the whole category is skipped: none of its tests run, no per-test results are produced, and the category is reported as skipped with the failing dependency listed (a config error — a missing or later dependency — is reported as “Config Failure - Contact Instructor”).

Dependencies only work for categories written earlier in the config. If the dependency is a later category or does not exist, the current category is skipped; the grader runs each category exactly once, in config order, and will not go back. If the entry is omitted, the category always runs. Dependencies are resolved within the suite only — references to categories in other suites always skip the current category.

{
    "category1": {},
    "category2": {},
    "category3": {
        "dependancies": ["category1", "category2"]
    }
}

Here is an example of incorrect use, where category1 depends on a later category and will therefore never run:

{
    "category1": {
        "dependancies": ["category2"]
    },
    "category2": {}
}

This mistake does not make the config invalid — the grader runs fine, it just always skips category1.

Test-level dependencies work differently and are described in §4.7.

3.4 tests#

All the tests that need to run in the category, as a map of test names to test objects:

{
    "category1": {
        "tests": {
            "test1": {},
            "test2": {},
            "test3": {}
        }
    }
}

3.5 Name rules#

All suite, category and test names must be globally unique across the whole config and match [\w-]+ (word characters and hyphens). Duplicate or invalid names make the config invalid. Artifact names given to preserve (see §4.5) must also match [\w-]+.

4. Tests#

Tests are the individual probes that check a single feature of the student’s work. Each test mainly consists of running one command and comparing its output against expectations.

Each test has the following fields:

  • cmd — required, the command to run (array of the binary and its arguments); an empty array fails the test

  • exit — optional integer, the expected exit code (default 0)

  • cwd — optional, the working directory for the command (default /)

  • suppressible — optional boolean, default true (see §4.6)

  • display_name — optional, a human-readable name

  • test_number — assigned automatically by the grader, no need to set it

  • dependancies — optional, a list of test names that must have passed first (see §4.7)

  • input — optional, an array of input specifications

  • output — optional, an array of output checks

  • constraints — optional, memory and time limits (see §4.3)

4.1 cmd#

The command to run. The first element is the binary, the rest are its arguments. The command runs inside the sandbox (no network); /repo, /test, /artifact are read-only mounts, /scratch is the only writable location. /artifact is on PATH, so a preserved artifact can be run by bare name.

{
    "cmd": ["binary", "5", "100", "output.txt"]
}

4.2 exit#

The exit code the command is expected to return. If the actual exit code differs, the test is considered failed. If the process is killed by a signal instead of exiting normally, the test fails regardless of the expected value.

{
    "test1": {
        "cmd": ["/artifact/aliquot_bin"],
        "exit": 1
    }
}

4.3 constraints#

Optional limits on the execution of the program. Both fields are optional and must be positive integers:

  • mem — upper limit of memory in MegaBytes

  • timeout — upper limit in seconds

{
    "constraints": {
        "mem": 400,
        "timeout": 2000
    }
}

If a limit is exceeded, the execution is terminated and the test fails.

4.4 input#

The input of the program, as an array of objects. Each object describes one input location and its contents:

{
    "input": [
        { "fd": 0, "raw": "12\n0\nf\n" }
    ]
}

Each input object has:

  • fd — required, the file descriptor to feed (a number, or the specifiers stdin, stdout, stderr which map to 0, 1 and 2)

  • raw or file_path — exactly one must be provided: inline text, or a path to a file

  • enforce_pipe — optional boolean, default false

{
    "input": [
        { "fd": "stdin", "raw": "5\n" }
    ]
}

{
    "input": [
        { "fd": 42, "file_path": "test://input.txt" }
    ]
}

raw content is always streamed to the child through a pipe. For file_path inputs, enforce_pipe selects the delivery mechanism:

  • false (default) — the file is opened and its descriptor is duplicated directly onto the child’s fd: the program sees a real file (it can lseek, fstat, etc.).

  • true — the file’s contents are read and streamed through a pipe: the program sees a pipe. Use this when the program must not be able to detect a real file or when it expects streaming input.

file_path resolution follows the rules in §5.

4.5 output#

The expected output of the program, as an array of objects. Each object describes one output location and the check to run against it:

{
    "output": [
        { "fd": 1 }
    ]
}

Each output object has:

  • fd or file_path — exactly one must be provided: the file descriptor to check (a number, or stdin/stdout/stderr), or the path of a file the program produces. file_path here is resolved against /scratch — this is the only place where the §5 resolution rules do not apply.

  • compare — optional, what to compare the output against (see below)

  • preserve — optional, an artifact name matching [\w-]+; if set and the check passes, the produced output is kept and becomes available to later tests as /artifact/<name> or @<name>. For fd checks the captured bytes are saved; for file checks the produced file is copied.

  • sensitivity — optional, strict or loose (default loose)

If compare is omitted:

  • an fd check always passes — the bytes are collected but not verified (mostly useful combined with preserve);

  • a file check becomes an existence test: it fails if the program did not produce the file.

The compare object must provide exactly one of:

  • raw — the expected output as inline text

  • file_path — a path to a file containing the expected output

  • regex — a regex pattern the output must match (max 500 characters, must compile)

compare.file_path follows the §5 rules (test://, @name, or relative to /repo).

{
    "output": [
        {
            "fd": 1,
            "compare": {
                "raw": "Length of aliquot sequence: 7\n"
            }
        }
    ]
}

{
    "output": [
        {
            "fd": 67,
            "compare": {
                "file_path": "test://expected_out_file.txt"
            },
            "sensitivity": "strict"
        }
    ]
}

{
    "output": [
        {
            "fd": 1,
            "compare": {
                "regex": "^[a-z]+$"
            }
        }
    ]
}

Sensitivity controls how the comparison is done. In every mode (including regex) the comparison is a search: the expected text may appear anywhere in the captured output, and extra content before or after it is allowed:

  • strict — case-sensitive search with exact whitespace: the expected bytes must appear verbatim in the output

  • loose (default) — case-insensitive search where runs of whitespace in the expected text match any amount of whitespace in the output (including none)

regex is matched with a plain (unanchored) re.search, case-sensitively. Use anchors (^...$) when the whole output must match.

4.6 suppressible#

A test marked suppressible (default true) is excluded from the dependency pre-runs the grader performs when a single suite, category or test is graded with --specific: in that mode only non-suppressible dependency tests are executed, the suppressible ones are skipped. In a normal full run it has no effect; it is a knob for keeping --specific pre-runs short and free of noise.

4.7 dependancies (test level)#

A test’s dependancies is a list of test names that must have passed first. Unlike category dependencies, they are resolved against tests of the same category, in config order: when the test is reached, each dependency must already have run and passed. A dependency on a test that appears later in the category, or that does not exist, is a config error — the test is skipped with a dependency error and will never run, because the grader iterates the tests once, in order. Inside a sequential category (§3.2) tests implicitly depend on the previous one.

5. Paths#

The sandbox exposes four fixed locations:

  • /repo — the student’s submission, mounted read-only

  • /test — files inside hw/<homework>/tests/, mounted read-only

  • /artifact — preserved outputs from earlier tests, mounted read-only

  • /scratch — a writable working directory the program may write to; it is the only writable location in the sandbox

file_path fields are resolved as follows:

  • in input and in compare: @<name> → /artifact/<name>, test://<path> → /test/<path>, anything else → /repo/<path>

  • in output (the program-produced file): relative to /scratch; the prefixes above are not interpreted there

Inside cmd arguments no substitution happens — paths are used literally inside the sandbox. /artifact is on PATH, so a preserved binary can be run as aliquot_bin or with its full path /artifact/aliquot_bin.

A full example combining compilation and execution:

Full example 
{
    "homework": "hw0",
    "suites": {
        "Aliquot": {
            "compilation": {
                "display_name": "Compilation",
                "visibility": "private",
                "tests": {
                    "compiles": {
                        "cwd": "/repo/aliquot/src/",
                        "cmd": ["gcc", "aliquot.c", "-o", "/scratch/aliquot_out"],
                        "output": [],
                        "exit": 0,
                        "constraints": { "timeout": 10000 }
                    },
                    "no_warnings": {
                        "cwd": "/repo/aliquot/src/",
                        "suppressible": false,
                        "cmd": ["gcc", "-Wall", "-Wextra", "-Werror", "-o", "/scratch/aliquot", "aliquot.c"],
                        "output": [{ "file_path": "aliquot", "preserve": "aliquot_bin" }],
                        "exit": 0,
                        "constraints": { "timeout": 10000 }
                    }
                }
            },
            "execution": {
                "visibility": "private",
                "dependancies": ["compilation"],
                "tests": {
                    "basic_sequence": {
                        "cmd": ["/artifact/aliquot_bin"],
                        "input": [{ "fd": 0, "raw": "12\n0\nf\n" }],
                        "output": [
                            {
                                "fd": 1,
                                "compare": {
                                    "raw": "12\n16\n15\n9\n4\n3\n1\n0\n"
                                },
                                "sensitivity": "loose"
                            }
                        ],
                        "exit": 0,
                        "constraints": { "timeout": 2000 }
                    }
                }
            }
        }
    }
}