Caliper Metric Profiles: Common Explanations
Introduction
This document is intended to define—in simple terms—the types of data represented and validated using 1EdTech Caliper Analytics®. Caliper is a standard intended to be used within the source code of an application so that learning analytics data may be passed to a data aggregator, such as a custom created application or dashboard that can receive Caliper data from the instructional technology application. Instructional technology applications using the Caliper standard can range from an LMS, such as Blackboard or Canvas, to a standalone application such as Piazza.
Caliper is often used in combination with other technologies, such as Learning Record Stores (LRS)—sometimes called a Learning Record Warehouse (LRW). One example of an LRS is Learning Locker. While applying Caliper to a single instructional technology might provide some additional data about how that technology is used, the power of Caliper comes when you apply a specific Caliper profile across several instructional technology applications, and aggregate that data into an LRS. By applying the same Caliper profile across technologies from which you are interested in capturing data, you are ensuring that the data you aggregate is all in the same format, allowing for much easier comparisons and analysis.
Once Caliper data resides in a central repository, you have many options to leverage the data. You can create software, such as a web-based dashboard that illustrates how often users are logging into all of your learning technology applications over time. You can provide access to the data for institutional researchers that make strategic use of the data to drive institutional decision making. You can create subsets of the data for faculty researchers, that have novel methods to explore the data in combination with other data sources they have access to. You could also provide student-facing dashboards allowing students to explore, holistically, their own interaction data in comparison to aggregate or anonymous groups of other students.
What it Means to Be 1EdTech Caliper Certified
When examining products or vendors that are using Caliper, you might see references to them being "Caliper compliant" or 1EdTech Caliper certified. This is an important distinction, as a product that is just Caliper compliant means that the developers followed the Caliper specification when creating the product. When a product is 1EdTech certified for Caliper Analytics, it means the developers followed the Caliper specification when developing the product and the product has gone through the Caliper certification process with 1EdTech Consortium (see 1EdTech Glossary of Terms). If a product or platform is officially certified by 1EdTech, you are assured that the product has implemented the Caliper specification correctly, allowing your tools to work together seamlessly and securely exchange learning data. You can find a list of Caliper certified products at imscert.org.
Caliper Analytics Profiles and Descriptions
Caliper defines a number of metric profiles, each of which models a learning activity or a supporting activity that helps facilitate learning.
The Caliper Analytics 1.1 profiles include:
- Annotation Profile
- Assessment Profile
- Assignable Profile
- Forum Profile
- Grading Profile
- Media Profile
- Reading Profile
- Session Profile
- Tool Use Profile
Annotation Profile
Link to Annotation Profile technical specification.
Common Description: Tracks annotation actions made by a reader in a digital text, such as an ebook. The annotation action recorded includes creating a bookmark, highlighting selected text and tagging. Additionally, the Annotation Profile records whether a student has viewed an annotation created by a fellow reader.
Related Profile(s): Reading Profile
Researcher Use Case (Value Proposition): Example research problems may include:
- Which students are generating annotations?
- What content is most often annotated?
- What types of annotations are being created?
- Which segments of text are being highlighted?
- What tags are being used?
- What notes are being added?
Faculty Use Case (Value Proposition): Instructors can use the clusters of annotation marks to determine whether or non-students are modeling effective learning behaviors, such as note-taking in the course material.
Instructors may also use the contextualized data to determine whether learners have the right idea about which material should be highlighted. In addition, if there are learners who are asking questions or making annotations indicating confusion about a particular piece of content, this can inform the instructor about the suitability or quality of the material which they have chosen to use.
Assessment Profile
Link to Assessment Profile technical specification.
Common Description: Collects data on how students work through assessments. It records the number of attempts, as well as five pattern types that record how students work through the assessment.
*Pattern types are common question formats, such as multiple choice, matching, true/false, calculated, and fill in the blank.
Related Profile(s): Assignable Profile
Researcher Use Case (Value Proposition): Tracking patterns using the assessment profile will allow researchers to identify correlations between assessment completion patterns, assessment scores, class grades, and other measures of success.
- Does time spent on assessment correlate with final score?
- Do multiple attempts result in significantly improved scores?
- If test-taking times are flexible, does taking the test earlier or later correlate with final score?
Faculty Use Case (Value Proposition): Tracking patterns using the assessment profile will allow instructors to understand more about how students are interacting with their assessments.
- How much time is required to complete and submit an assessment?
- How much time is spent answering individual questions?
- Which questions are completed?
- Which questions are skipped?
- If test-taking times are flexible, when do learners start their assessments?
Assignable Profile
Link to Assignable Profile technical specification.
Common Description: The assignable profile identifies who assigned material, length of time assignments take to complete, and how many attempts were taken on an assignment. Tracks “lifecycle” of an assignment, which includes assignment dates, due date, and attempts, but does not include criteria for completion.
Related Profile(s): Media Profile, Grading Profile (may have more information relevant to the information included in this profile)
Researcher Use Case (Value Proposition): Researchers can use data captured in this profile to determine what types of material students find more challenging based on the number of attempts, the level, and the disciplines associated with those assignments
- Who is assigned what material?
- How long does it take to complete something that has been assigned?
- What piece of assigned material presents the biggest challenge (i.e. needs most retake attempts)
Faculty Use Case (Value Proposition): Faculty can use this profile to monitor the number of attempts a student takes an assessment or works on an assignment to identify items that may be more challenging than others. Like the researcher, this data can be used to gain insight into:
- How long a student works on an assessment?
- What items present the biggest challenges?
- What assessment items are consistently misunderstood?
Forum Profile
Link to Forum Profile technical specification.
Common Description: The forum profile allows tracking of student engagement with discussion forum content, including how they navigate through the content, whether they’ve subscribed to any message threads, which posts they’ve written or viewed, and when each activity occurred.
Related Profile(s): n/a
Researcher Use Case (Value Proposition): Possible research questions might be:
- Who is posting most often?
- Which posts create the most replies? Compare graded vs. non-graded discussions?
Faculty Use Case (Value Proposition): Could be used to improve discussion prompts and monitor discussion activity.
Grading Profile
Link to Grading Profile technical specification.
Common Description: This profile allows including the assignment and assessment attempt scoring to the event stream. Comparison of scoring events to learner data can shed light on study and/or test prep behaviors.
Related Profile(s): Assessment Profile
Researcher Use Case (Value Proposition): Using this profile can provide researchers with information on student study practices and insight into academic success, including:
- Which courses/programs have a high rate of grade changes?
Starting with that question could lead a researcher down many paths.
- How are assignments in those courses designed?
- Are there trends among certain programs?
Faculty Use Case (Value Proposition): Data provided by this metric can help instructors and instructional designers analyze the effectiveness of assessments. For example:
- Do any of my assignments requirement more frequent grade changes than others?
- Why did those assignments require grade changes?
- What can modification need to be made to the assessment item reduce grade changes and increase student success?
Media Profile
Link to Media Profile technical specification.
Common Description: This profile allows tracking of activities related to multimedia objects, including viewers’ identities, date/time stamps when viewing started or stopped, use of pause, rewind, fast-forward, and other navigation functions, and activation of features such as mute, closed-captioning, and full screen viewing.
Note: Use of this profile requires that the content is hosted on a Caliper-compliant or (ideally) Certified platform. Interactions with content hosted on a non-compliant platform (e.g. YouTube) cannot be fully tracked, even if that content is embedded in the LMS.
Related Profile(s): Assignable Profile, Session Profile
Researcher Use Case (Value Proposition): How students access and view media is among the general inquiries often posed by researchers. This profile enables researchers to analyze and answer questions relating to:
- Who is using the tool?
- Which videos are being played the most?
- How long are they spending on each video?
- Where do they pause the video?
Faculty Use Case (Value Proposition): Digital media is increasingly becoming widely used in instruction. From an instructor’s perspective, how this medium is accessed and used by students can provide insights into:
- Are students reviewing the content?
- Are there particular sections of content that get replayed more than others?
- Do viewing patterns correspond with material covered in class, homework assignments, or test preparation?
Reading Profile
Link to Reading Profile technical specification.
Common Description: This profile allows tracking of activities related to reading digital texts, including readers’ identities, date/time stamps of reading sessions, and number and identification (document, chapter, and page number) of pages viewed.
Note: Use of this profile requires that the content is hosted on a Caliper-compliant or (ideally) Certified platform. Interactions with content hosted on a non-compliant platform cannot be fully tracked, even if that content is embedded in the LMS.
Related Profile(s): Annotation Profile, Assessment Profile
Researcher Use Case (Value Proposition): When used in conjunction with the Assessment Profile, viewing patterns can be correlated with performance measures.
- Does quantity of annotations correlate with assessment scores?
- Does overall time spent reading correlate with assessment scores?
- Are particular viewing patterns/habits predictive of student success?
- What are the average viewing patterns of students? Do they differ between courses, course sections, instructors, or student demographics?
Faculty Use Case (Value Proposition): When used in conjunction with the annotation profile, instructors can see readers’ notes and highlights in conjunction with their reading sessions.
- Who is consuming the content and what content are they annotating?
- Do students’ notes indicate an understanding of the material or thoughtful reflection?
- What materials are being accessed and when?
- How often is the content viewed?
- What paths are taken to reach the content?
Session Profile
Link to Session Profile technical specification.
Common Description: The session profile captures data about a user entering a software application, thus creating a user session within that application. For example, when a student logs in to a learning management system.
Related Profile(s): Tool Use Profile, which can capture more granular data about what a user is doing within a session with a specific software tool.
Researcher Use Case (Value Proposition): Researchers can use data captured in the session profile to track and compare sessions across various software applications over time within the same ecosystem/university.
- What are average viewing patterns or trends? Do they differ between courses, course sections, instructors, student demographics, or applications?
- Are particular access patterns/habits predictive of student success?
Faculty Use Case (Value Proposition): Faculty can use data captured in the session profile to identify students who are or aren’t logging in to particular platforms or applications.
- Which students are not logging into assigned platforms regularly or at appropriate times?
- What does a particular student’s access history look like?
- How do the access patterns of students in the instructor’s course compare to institutional or department averages?
Tool Use Profile
Link to Tool Use Profile technical specification.
Common Description: This profile can be used to instrument a variety of learning-centric software, and captures details about how a person interacts with a specific piece of software. This profile can capture event data, such as when a student utilizes specific features of a software application or clicks specific links.
Related Profile(s): Session Profile, which compliments tool usage with more generic session data.
Researcher Use Case (Value Proposition): Researchers can use this profile to explore the relationship between specific tool features and student performance.
- Explore the use of a specific tool across time (i.e. semesters), across disciplines, across programs, etc.
- Compare trends across different tool utilization (which are growing, shrinking, etc.)
Faculty Use Case (Value Proposition): Faculty can use this profile to enhance understanding of which tools are being used across the tool ecosystem in a course, course sequence, or degree program.
- Identify students not participating in the digital ecosystem.