A customised African release of CAST (Computer Assisted Statistics Textbooks) has provided much-needed access to a series of modern statistics textbooks for many students in Africa. The recent addition of a collection of interactive CAST exercises has proved valuable in strengthening the learning of statistical concepts by students. A new CAST testing system has been developed to present the exercises as formal test questions. Students can practice beforehand with similar randomised exercises and can get immediate feedback as soon as the test is finished. This paper describes a pilot study in Kenya to evaluate the added benefit of using the CAST tests and is its first evaluation. Use of these tests is linked to an on-line course called “Statistics Made Simple”. The effectiveness of the tests is discussed.