Probabilistic models for competence assessment in education
Visualitza/Obre
156717.pdf (812,8Kb) (Accés restringit)
Estadístiques de LA Referencia / Recolecta
Inclou dades d'ús des de 2022
Cita com:
hdl:2117/347824
Tipus de documentProjecte Final de Màster Oficial
Data2021-04-19
Condicions d'accésAccés restringit per decisió de l'autor
Tots els drets reservats. Aquesta obra està protegida pels drets de propietat intel·lectual i
industrial corresponents. Sense perjudici de les exempcions legals existents, queda prohibida la seva
reproducció, distribució, comunicació pública o transformació sense l'autorització del titular dels drets
Abstract
Probabilistic models of competence assessment join the benefits of automation with human judgement. In this work, two probabilistic models of peer assessment (PG1- bias and PAAS) are replicated and compared. We also present PG-bivariate, a model combining the approaches from the first two. After introducing some fundamental concepts of probability and Bayesian inference, the Bayesian models are imple- mented in Stan, whereas PAAS is converted into Python. The posterior predictive analysis of the Bayesian models showed that the distributions of new samples con- ditioned on the observed data were very similar to the distributions of samples, indicating that the data generating processes were being captured. We checked that PAAS software was correctly implemented in Python (pyPAAS) by replicating the synthetic experiment reported in the original paper. When comparing PG1 -bias with PG-bivariate, the first one produces predictions with lower RMSE. However, both models show similar behaviours when assessing how to choose the next assignment to grade, with an RMSE decreasing policy reporting better results than the random policy. To make fair comparisons among all models, we adapted the Bayesian infer- ence models to compare with pyPAAS. Under these conditions, PG1 -bias reported the lowest error in situations of scarce ground truths. Nevertheless, once nearly the 10% of the teacher's grades were observed, pyPAAS came to equal and sometimes exceeded the quality of PG1-bias' predictions by following an entropy minimization heuristic. Altogether, these results suggest that 1) PG1 -bias and PAAS have been successfully replicated, though further tuning of PG1-bias shall yield even better predictions and 2) our proposal to reconcile PAAS' trust-based approach with PG1- bias' theoretical background has resulted in similar values of the percentage error to those by the other two models. However, the longer computation times of PG- bivariate compared to PG 1 -bias may indicate that the phase space of our model has a more problematic surface to the sampler than that one of PG1 -bias'. Future work includes the application of the models to real experimental data, as well as exploring new heuristics to determine which teacher's grade should be observed next.
MatèriesBayesian statistical decision theory, Probabilities -- Mathematical models, Education -- Evaluation, Estadistica bayesiana, Probabilitats -- Models matemàtics, Educació -- Avaluació
TitulacióMÀSTER UNIVERSITARI EN INTEL·LIGÈNCIA ARTIFICIAL (Pla 2017)
Col·leccions
Fitxers | Descripció | Mida | Format | Visualitza |
---|---|---|---|---|
156717.pdf | 812,8Kb | Accés restringit |