Sunday, January 17, 2016

Automating R exercises and exams using the exams package

It's a pain to design statistics exercises each semester, and because students from previous share old exercises with the new incoming students, it's hard to design simple exercises that students haven't already seen the answers to. On top of that, some students try to cheat during the exam by looking over the shoulder of their neighbors. Homework exercises almost always involve collaboration even if you prohibit it.

It turns out that you can automate the generation of fixed-format exercises (with different numerical answers being required each time). You can also randomly select questions from a question bank you create yourself. And you can even create a unique question paper for each student in an exam, to make cheating between neighbors essentially impossible (even if they copy the correct answer to question 2 from a neighbor, they end up answering the wrong question on their own paper).

All this magic is made possible by the exams package in R. The documentation is of course comprehensive and there is a journal article explaining everything:
Achim Zeileis, Nikolaus Umlauf, Friedrich Leisch (2014). Flexible Generation of E-Learning Exams in R: Moodle Quizzes, OLAT Assessments, and Beyond. Journal of Statistical Software 58(1), 1-36. URL
I also use this package to deliver auto-graded exercises to students over See here for the course I teach, and here for the datacamp exercises.

Here is a quick example to get people started on designing their own customized, automated exams. In my example below, there are several files you need.

1. The template files for your exam (what your exam or homework sheet will look like), and the solutions file. I provide two example files: test.tex and solutiontest.tex

2. The exercises or exam questions themselves: I provide two as examples. The first file is called pnorm1.Rnw. It's an Sweave file, and it contains the code for generating a random problem and for generating its solution. The code should be self-explanatory. The second file is called sesamplesize1multiplechoice.Rnw and has a multiple choice question.

3.  The exam generating R code file: The code is commented and self-explanatory. It will generate the exercises, randomize the order of presentation (if there are two or more exercises), and generate a solutions file. The output will be a single or multiple exam papers (depending on how many versions you wanted generated), and the solutions file(s).  Notice the cool thing that even in my example, with only one question, the two versions of the exams have different numbers, so two people cannot collaborate and consult each other and just write down one answer.  Each student could in principle be given a unique set of exercises, although it would be a lot of work to grade it if you do it manually.

Here is the exam generating code:

Save from the gists given above (a) the test.tex and solutiontest.tex files, (b) the Rnw files containing the exercise (pnorm1.Rnw, and sesamplesize1multiplechoice.Rnw), and (c) the exam generating code (ExampleExamCode.R).  Put all of these into your working directory, say ExampleExam. Then run the R code, and be amazed.

If something is broken in my example, please let me know.
Shuffling questions: If you want to reorder the questions in each run of the R code, just change myexamlist to sample(myexamlist) in the call below that appears in the file ExampleExamCode.R:

sol <- exams(sample(myexamlist), n = num.versions, 
             dir = odir, template = c("test", "solutiontest"),
             header = list(ID = getID, Date = Sys.Date()))


bczernecki said...

The package is great! The only thing I miss is an additional options for random ordering of questions. Are there any plan to include such a possibility in the nearest future?

Shravan Vasishth said...

That is not correct; you can randomize the questions, no problem. I will add example code.

bczernecki said...

I had to accidentally omit this option while getting familiar with the documentation to the package. Thanks for quick reply and once again I must admit that the "exams" package is just a brilliant job!

Achim Zeileis said...

First, let me thank you for promoting our "exams" package. We're glad if this is useful for you :-)

Second, as for the shuffling. You can always do any kind of shuffling "by hand" in the R code chunks of the exercise template. If you just need a simple permutation, you can also set \exshuffle{TRUE} in the metainformation of the exercise.

This is one of the many new features of the package that are unfortunately not yet accompanied by some user-friendly and non-technical documentation. The best you can do at the moment is do say exams_skeleton() and browse through the different exercise types and output formats to look for something that is similar to what you need.

The new features added over the last year also include support of .Rmd instead of .Rnw exercises. Also more output formats have been added, especially for the very nice live voting software ARSnova - and for a PDF output format for single- and multiple-choice exercises that can be automatically scanned and evaluated in R.

If you experience problems or have questions about the package, please feel free to join the discussion in the forum on R-Forge:

jenine said...


Great stuff! I am wondering if you or anyone out there has some examples of Rnw files to do more simple multiple choice and short answer questions. I'm new to all of this programming and I can get the tests to generate with your questions but I cannot seem to build my own that will work.

Thank you so much!

Shravan Vasishth said...

Jenine, I guess you are best off posting on the exams mailing list about this. Someone will surely send you examples.

Achim Zeileis said...

Jenine, thanks for your interest. As Shravan said, it would be best to post at

and then we can help you there. Maybe you can post a simple multiple-choice exercise there and then I can show you how to put it into a format for the "exams" package.

Unknown said...

Great example, terrific for learning the package!
I put all these files in a directory, and tried to run them, but I got this error

Error in texi2dvi(out_tex[j], pdf = TRUE, clean = TRUE, quiet = quiet) :
Running 'texi2dvi' on 'test1.tex' failed.

I am running OSX 10.11.6 and R version 3.4.2

Any guidance on why the code isn't working would be very much appreciated! Looking forward to taking advantage of this new package, and thanks again!

Shravan Vasishth said...


pdflatex test1.pdf

Unknown said...

Thanks for the suggestion. I figured out I needed to tell r where the binaries were. All set, and thanks!