ľ�ϸ���Ӱ��

DACE is a dashboard that provides a comprehensive overview of course evaluations, student activities in a course, course development history, and notes from course developers. By visualizing this information in one place, it will be easier to update and improve courses.

Background and hypotheses

In 2021, the decision was made to start using Caracal as a university-wide evaluation system. This required a technical redevelopment of the system. Prior to this, an inventory was made within the faculties of the needs with respect to course evaluations. Discussions with various stakeholders revealed that in addition to Caracal outcomes, there was also a need for process and policy information, information about (student) experiences, opportunities for improvement and educational quality. Also mentioned were panel discussions, key figures, test analyses, correspondence with the program committee, teacher reflection and unsolicited feedback. Looking at multiple sources in context gives a much richer and more complete picture of teaching. The proposal in the advisory report was therefore to purchase or develop a platform that serves as a dashboard and brings together various sources of information. More and more sources can be linked to this platform step by step, so that the complete quality assurance file of an educational unit (such as a course or program) is kept together. In this way the information about that particular education can be viewed in context.
In this project, a first pilot will be started with such a platform: the Dashboard for Course Evaluation (DACE) with the following core functionalities: 1) visualizations of Caracal evaluations, 2) visualizations of student activity in the learning management system (LMS), 3) historical progression, and 4) a link to notes and other documents.

The previously written advisory report identifies several long-term goals. In this pilot, we are making a start on realizing those goals by trying to achieve the following: 1) the course coordinator has a richer and more complete picture of the course than without DACE; 2) the course coordinator gets additional information; information needs are met (usefulness) and this information is easy to use (usability); 3) the course coordinator gets concrete insights (“action perspectives”) on how the course can be modified; 4) by creating an evaluation plan, and communicating this to students, we hope to have a higher response rate to Caracal (we are aiming for 75%).  

Realization of DACE

The dashboard was developed in powerBI and combines data from Caracal, Blackboard and documents with course notes stored in OneDrive. It also displays data from surveys conducted with Qualtrics, for example concerning student attendance.

Evaluation

2024 - 2025

DACE was piloted in block 1 and block 3 of academic year 2024-2025 in a graduate course in the Faculty of Social and Behavioral Sciences (Course 1), and in an undergraduate course in the Faculty of Science (Course 2). At the end of their courses, they were given access to DACE with the actual student data from their courses. The coordinators were interviewed about their experiences with Caracal and DACE.

DACE complemented insights obtained from Caracal by providing a behavioral layer to students' subjective feedback, helping coordinators better understand how students engage with materials and learning activities. Caracal offers course coordinators general feedback, such as satisfaction with course structure or workload. In turn, DACE leads to insights on which materials were utilized or underutilized, whether students in different grade show different study patterns, and how attendance differs across course weeks. Based on DACE, coordinators planned targeted changes concerning instructional design, for example encouraging students to engage earlier with key materials or modifying tutorial formats. Coordinators also expressed interest in sharing insights from DACE with students to promote better study strategies.

In both courses, the course coordinators were aware that DACE only captures LMS data and to a limited extent what happens offline through surveys. Obtaining sufficient response on the surveys was a challenge. On the other hand, coordinators appreciated the tailored design and requested further enhancements, such as comparisons of the course across cohorts, exam analytics integration, and the ability to upload course manuals. These suggestions align with the original vision of DACE as a central repository of course-related data.

DACE has shown promise as a tool that augments existing evaluation methods, offering course coordinators rich and actionable insights into student behavior and course effectiveness. It can help in advancing UU’s Education Model by supporting educational innovation and enhancing the quality of education through analysis of student activities and feedback. Overall, DACE represents a valuable step toward realization of the updated UU Education Model in which research into teaching practice is prioritized.

For broader adoption, technical reliability, integration with diverse data sources, and clear guidelines for interpretation and use must be addressed. In that regard, the university’s current transition to a new LMS (Brightspace) offers opportunities. For instance, Brightspace includes built-in attendance tracking, which could reduce the need for manual survey administration and enhance the accuracy of attendance data. Additionally, Brightspace’s more flexible data architecture may allow for more comprehensive behavioral analytics and easier integration with DACE.

Timeline

2025 - 2026

DACE will not be continued for now. However, feel free to contact us if you are interested in using DACE. We will evaluate what the possibilities are. 

Contact information

Team LA via learninganalytics@uu.nl