Author: Jessica Dehler Zufferey

By Jessica Dehler, Head of R&D at Coorpacademy

Data, measurement, and analysis have always been important for Learning & Development (L&D). The scope, approaches, and use cases have, however, evolved a lot with the arrival of Big Data. And learning sciences and educational research are no exception to the Big Data rule, where more and more methods from data science are applied to study learning and teaching.

This article explores the evolution from evaluation-focused corporate learning to an analytics-driven one.

Since Human Resources and L&D in particular have adopted the role of business partners rather than mere internal service providers in their companies, they have always measured the impact of their actions. This was mainly done using a logic of evaluation and it was very often based on a model similar to the one suggested by Donald Kirkpatrick which measures the impact of training on 5 levels (see Fig 1.)

The lower levels were generally privileged because of how easy they were to measure: a combination of satisfaction surveys, completion rates and learning assessments was often considered sufficient. Analyzing only those lower levels of the model would affect the decisions and have negative implications on the usefulness of the conclusions (for a critical discussion on the Kirkpatrick model, see Bates (2004).)

This kind of evaluation, overemphasizing the “effect” in search of a proof of training effectiveness, creates a loss of interest in an age of lean processes, agile methods and continuous improvements. Today, we are looking for data and analyses that have a more descriptive nature, thereby contributing to understand learning and not simply justifying the (budget spent on) training initiatives.

L&D departments in modern data-driven organisations are getting more and more interested in analytics-driven approaches. These help to understand the “how” and the “why” of the learning that takes place during a training, to identify the needs for improvement and to develop ideas for interventions. This is not just a trend. It is deemed essential, as a significant part of the training budget is allocated to analytics (up to 5%).

This phenomenon goes along with an evolution towards new types of indicators that are measured. While completion rates were the reference metric even in the early days of Massive Open Online Courses (MOOCs), additional indicators inform more broadly about the learners and their learning processes. The creation of these new indicators is accelerated by new ways of tracking data, such as the Experience API or xAPI which stores individual learning actions, not just data about achieved results.

Our complete revision of analytics dashboards was intended to support our clients in this new approach. We looked for inspiration from learning analytics research and games. We discovered recruiting games designed to identify a set of performance indicators, identified as crucial for job performance based on analyses of previous employees’ performance beforehand. The game score determines the candidate’s match with the job profile. Our reasoning was that behaviour in a job-related online training would be an even more valuable source of information about a person’s qualities. We then developed a set of behavioural indicators – curiosity, perseverance, performance, regularity, and social learning – which help to understand the learners, to identify personal behavioural preferences, and to drive decisions related to communicating about and dispensing the training initiative. We benefited from the flexibility and self-directed learning on our platform, as only when learners have a certain degree of choice can their behaviour be interpreted in this way. If, for instance, the learning path was completely scripted, measuring curiosity would be impossible.

Our main challenge will now be to measure the usage and usefulness of these new insights and improve them iteratively, as can be expected in a consistent data-driven analytics approach.

References:

Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347.

 

By Jessica Dehler, Head of R&D at Coorpacademy

In the educational field, gamification has had an incredible success story in the last ten years. While in the beginning of this century, there was a clear distinction between “standard” learning and game-based learning, since then the “use of game-design elements within non-game contexts” (Deterding, Khaled, Nacke & Dixon, 2011, p. 1; also see Fig. 1), which is defined as gamification, has been applied in many learning settings, and in particular in corporate training. This spread of gamification was not unique to education but also applies to areas like health, work, data collection, software and tools, social networks, and environmental behaviour. The success of gamification may also be related to the fact that accessing (digital) learning opportunities has become so much easier and more democratic. With learning being available anytime anywhere, the crucial point of interest has shifted towards engagement and gamification was identified as a good candidate to help out.

Academic research is trying to catch up with the movement. An analysis on Google Scholar revealed that the number of scientific publications on the keywords “gamification” and “learning” has increased by a factor of 78 in the last ten years. The focus of research was primarily to validate that gamification positively affects performance. In their review, Hamarai et al. showed that 15 out of 24 studies found positive effects on engagement as well as quantity and quality of performance (Hamari, Koivisto, & Sarsa, 2014). As most studies focused on reward-based gamification concepts, the alarm bells rang: Would an over-emphasis on extrinsic motivation undermine intrinsic motivation and finally diminish engagement and performance?

The research community called for deeper approaches in the studies about gamified learning. Landers and Landers proposed to investigate the intermediate variables that mediate the effect of gamification on performance and identified time spent on tasks as such (Landers & Landers, 2014.) Many researchers requested that empirical work was needed to disentangle the different factors influencing learners’ motivation. Enter self-determination theory (SDT), one of the most remarkable and widely applied motivation theories in education (Deci & Ryan, 2002). It posits that well-being and personal growth can be achieved through the satisfaction of three basic psychological needs: autonomy, competence, and relatedness. Autonomy means that you are free to decide based on your own interests and values. Competence refers to how effective we perceive ourselves when reacting to the demands of the environment. This need also makes us look for optimal challenges for our own capacities. Relatedness refers to the connections and interactive support with others.

SDT was applied in order to a) inform the design and b) study the underlying mechanisms behind the positive effect of gamification on motivation.

In terms of design of learning environments, Groh (2012) proposed to foster the three psychological needs of SDT by the following interventions, which can be considered in training and learning initiatives:

Autonomy

Voluntary activity: Deciding when and what to play is an autonomous choice, and therefore is intrinsic.

Personal goals: Although users in a meaningful community share some interests, each user has different goals, and it is important to recognize that each user could follow their own path inside the gamification application.

Competence

Interesting Challenges: There should be a good connection between difficulty and the required skill to perform. Something really easy or really hard could decrease engagement, however increasing difficulty with time will increase engagement.

Meaningful feedback: Whether you win or lose, feedback is the key for effective learning. This also implies that errors are allowed.

Relatedness

Meaningful Community: intrinsic motivation is increased if the learner is part of a community that shares the same interests. This also fosters learners’ confidence.

The SDT was also used in studies that tried to disentangle the effect of diverse game design elements instead of dealing with gamification as a unified concept. Recent work by Sailer et al. was able to demonstrate in a digital warehouse that game design elements had specific effects on psychological needs (Sailer, Hense, Mayr, & Mandl, 2017). They put logistics apprentices in front of a learning game, and divided the poll into 3 groups. Learners in experimental condition 1 who were provided with leaderboards, badges and performance graphs self-reported significantly higher levels of satisfaction in the need for competence compared to a control group (the ones who would just do the learning game):

Learners in experimental condition 2 who could choose an avatar and could see other teammates reported significantly higher levels of satisfaction in relatedness than learners in the control group.

Interestingly, the perceived autonomy with regard to freedom of decision was not influenced by any of the experimental conditions. It seems that freedom of choice is a basic instructional design principle independent of game elements.

As a wrap-up, we see that gamification works and there is much more about it than simple rewards such as points, leaderboards and badges. It actually does work because specific game design elements effectively address specific psychological needs that are determining motivation and thus engagement.

The review of this empirical research comforts us in our design choices: challenging peer learners in a battle demonstrates mastering a skill and reinforces relatedness in a common goal of progression, peer coaching creates meaningful community, direct feedback in learning modules and story-linked feedback in branching modules work towards competence, and flexibility has always been a building block fostering autonomy. When we’re analysing our battle data, we’re comparing engagement and performance indicators of learners who play battles and learners who never play battles. We found that battle players are up to 4 times more engaged with the learning material and are successfully completing two times more modules (see Fig. 2).

a) Engagement with learning material

b) Performance Indicator

Fig 2. Difference between battle players and non-battle-players regarding a) number of distinct learning videos watched and b) percentage of successfully completed modules from started modules (i.e. success rate).

In order to leave you with an inspiring concept, let us insist on the importance of activities that are enjoyable – though not only that – but that are also valuable for the user, for their work and life, as proposed by the notion of meaningful gamification (Nicholson, 2015). In times where lifelong learning has much value for our employability and wellbeing at work, the most basic and efficient recommendation of all has become our credo at Coorpacademy: Enjoy learning!

References:

Deci, E. L., & Ryan, R. M. (2002). Handbook of self-determination research. University Rochester Press.

Deterding, S., Khaled, R., Nacke, L., & Dixon, D. (2011). Gamification: Toward a Definition. Paper presented at the CHI 2011, Vancouver.

Groh, F. (2012). Gamification: State of the art definition and utilization. Institute of Media Informatics Ulm University, 39, 31.

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does Gamification Work? — A Literature Review of Empirical Studies on Gamification. In System Sciences (HICSS), 2014 47th Hawaii International Conference on (pp. 3025–3034).

Landers, R. N., & Landers, A. K. (2014). An Empirical Test of the Theory of Gamified Learning: The Effect of Leaderboards on Time-on-Task and Academic Performance. Simul. Gaming, 45(6), 769–785.

Nicholson, S. (2015). A recipe for meaningful gamification. In Gamification in education and business (pp. 1–20). Springer.

Sailer, M., Hense, J. U., Mayr, S. K., & Mandl, H. (2017). How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in Human Behavior, 69, 371–380.