Analysis of the use of automatic judges in computer programming classes in vocational education

Abstract

The goal of this thesis is to analyse the use of automatic judges in the teaching of Computer Programming in Vocational Training programs, in a sample of secondary schools in Catalonia and Spain. We review the academic literature on automatic judges and describe their history and the different services that are currently available. Through surveys to teachers and students from Vocational Training centres, we have collected information about the tools in use. We classify twelve of them according to features that we found to be relevant to the teachers we surveyed. Using the collected data, we study the reasons in favour and against automatic judges, as well as the level of familiarity of active teachers with those tools. We identify some discrepancies between teacher's expectations and actual experiences of teachers who have adopted them. We use students' responses to derive statistics about their satisfaction, and to detect differences in the attitude of those who use judges in class and those who don't with respect to the subject of programming. In order to detect any effects on learning, we use three Bebras challenges to evaluate the competence of algorithmic and computational thinking. No difference in the ability of students to complete these challenges has been observed, but a difference in how willing they are to work on them has been detected. The effect is contrary to the one originally hypothesised, which leads to some interesting questions for further study.