Performance indicators for corporate MOOCs – Regularity
In our previous post, we introduced the need to create new performance indicators for corporate MOOCs. In this post, we explore the value of regularity as an indicator.
How regular is a user’s learning in a MOOC? How important is regularity for course success? How can regularity be measured? What does a learner’s regularity pattern tell recruiters? How should course design support regularity?
The activity level of employed learners in MOOCs is typically equally distributed from Monday to Thursday with increasing engagement in the morning and a peak in the evening hours, while on Friday activity levels are very low. Peak time shifts to the afternoon on weekends. (cf. Shirvani Boroujeni, Kidzinski & Dillenbourg , 2016 (2))
MOOC learners report that a time management problem is the number 1 reason to drop out of a course (cf. Nawrot &Doucet, 2014 (4)). Apart from self-reported data, behavioral data on regularity have also been shown to predict completion rates in MOOCs: Learners with higher regularity values have higher chances of finishing a course. Among different measures that were suggested, the best predictive value was obtained when measuring peak of day (i.e. learning at similar hours of the day) as well as similarity of weekly patterns (i.e. learning with a similar distribution of study time over weekdays). For a deep dive into measuring regularity, refer to to the article by EPFL outlining several measures with specific and complementary information value. (cf. Shirvani Boroujeni, Sharma, Kidzinski, Lucignano & Dillenbourg , 2016 (1))
Time management during MOOCs depends on external factors, in particular professional constraints, and internal factors, that is self-regulation (cf. Kizilcec & Halawa, 2015 (3)). Interestingly. learners who are full-time employed professionals are learning with significantly higher regularity compared to full-time student learners. (cf. Shirvani Boroujeni, Sharma, Kidzinski, Lucignano & Dillenbourg , 2016 (1)) Employees seem to compensate supposedly higher time constraints by better time management skills and these skills pay off in their lifelong learning journey given that employees also show higher engagement and better achievement in MOOCs (cf. Shirvani Boroujeni, Kidzinski & Dillenbourg , 2016 (2)). Thus, measuring the regularity of a MOOC learner has great potential in informing HR about a learner’s time-related self-regulation competence.
In order to increase regularity, online courses can act on communication and delivery modes in particular.
Push notifications and course-related communication should be delivered at the right moment for each learner. Therefore, temporal information needs to be included in the user model and in predictive analyses.
The peak in evening hours indicates that learning is accomplished during commute times. Thus, courses need to be delivered in a mobile-friendly mode that supports short sessions and that allows learners to interrupt and continue the course easily.
One could even think of support tools for example for planning, prioritizing or monitoring, these would be integrated in a course with the goal of providing learners with time management guidance. This approach is especially useful if the development of metacognitive and self-regulation skill is a training goal by itself.
Further indicators are discussed in these posts:
(1) Mina Shirvani Boroujeni, Kshitij Sharma, Lukasz Kidzinski, Lorenzo Lucignano, and Pierre Dillenbourg. (2016) How to quantify student’s regularity? 11th European Conference on Technology Enhanced Learning, Lyon, France, September 13-16, 2016
(2) Mina Shirvani Boroujeni, Lukasz Kidzinski, and Pierre Dillenbourg. (2016) How employment constrains participation in moocs? In Proceedings of the 9th International Conference on Educational Data Mining, pages 376–377, 2016.
(3) René Kizilcec and Sherif Halawa. (2015) Attrition and achievement gaps in online learning. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale, pages 57–66. ACM, 2015.
(4) Ilona Nawrot and Antoine Doucet. (2014) Building engagement for mooc students: introducing support for time management on online learning platforms. In Proceedings of the companion publication of the 23rd international conference on World wide web companion, pages 1077–1082. International World Wide Web Conferences Steering Committee, 2014.