• RSS Feed

Perusall
  

Note: Currently there are some issues with linking Perusall through Safari. Please use a different browser to access Perusall (Chrome and Firefox are available on the UWP). 

Context of the Perusall project

Perusall is a brand new and innovative online social platform for collaboration. The motto is every student prepared for every class.
 
The University of Groningen conducts a Perusall project in this academic year with the financial support of the executive board. The size of the project and the enthusiasm of the staff reflects its importance and possibilities. It is part of the educational strategy of the UG to use more active learning in the curricula.
 
Feedback given by our users is very important to us. What went well and what could be better?

Active learning template

 
This template describes integrating the use of Perusall into an existing course. First, the “Why” is covered, after that, the process of constructive alignment is followed to make sure that course activities align with learning outcomes and assessment.
 

Why it works

Perusall helps every student prepare for every class. Students carry out assignments to annotate texts before class for a part of the course credit. Thus, they are invited to read and think about your texts before the lecture. With prepared students, the time during the lecture can be used for active learning: discussions, tasks and exercises. Perusall is a social system. Students annotate together, so, studying becomes a social rather than solitary experience. Some students may want to be the first to ask questions based on a text; others may like answering questions or responding to fellow students more. The lecturer sees the annotations beforehand. This makes it possible to pick out good annotations from students who may normally be too shy to speak up in class. By rewarding these annotations it is possible to encourage more students than usual to participate in class discussions. Scoring and grading are done automatically. As a lecturer you can review the scores, and check marginal students scores by hand, saving you a lot of time. At the same time, Perusall helps to make sense of the many annotations by providing a confusion report and overviews of most upvoted comments and questions. This helps you as a teacher to focus on the most important issues that students come up with.
 
 

Constructive alignment

With any course, the design is about the alignment of teaching and learning activities, learning outcomes, and assessment (Biggs & Tang, 2011). This principle is called constructive alignment. The idea is that all course activities are directly aligned with the learning outcomes of a course. Further, the learning outcomes and the activities are aligned with the course assessment. This helps students to focus their attention on the important parts of the course and course materials. Students are often quite goal-directed: they spend their time on course tasks that are rewarding for them, which means it helps them pass their exams. For Perusall, this means that you should make sure that tasks are rewarded (e.g. via course credit, bonus exam question, or participation score), and that the relation of the task to the learning outcomes is clear.

Learning outcomes

In many courses, studying literature is an essential part of the course. Most of the time, recall of information is not enough. Defining learning outcomes using the learning taxonomy of Bloom and the associated action verbs are a way to plan this. What do you want your students to focus on with their annotations? You may want to ask students to compare the literature to other known approaches, evaluate the author's viewpoints, ephrase sources, or reflect based on an original text.

Students need to know what is expected of their annotations. On what level should they be? Via course evaluations of Perusall, we have seen that this is a learning process during the course. From the start, you as an instructor can give guiding principles (a reading guide and explanation about your learning outcomes). By giving feedback on the annotations that were done by the students, you can tweak their responses to make sure they get on the level where you want them to be. Consider picking out the best student examples, and capitalizing on them. (For example, you might call out in class particularly good questions, and use them as the basis for class discussion.)

Course activities

Perusall is based on weekly assignments that students complete before a lecture starts. Lecture time is used to discuss issues around the course content. The assignments are made for points (course credits) or rewarded in some other way (for example, as a bonus exam question). Without some form of reward, our experience is that students do not carry out the task. The weekly assignments consist of reading one or more the texts and making annotations based on this text. Annotations consist of questions based on the text or responses to the other student questions. Perusall grades the student work automatically, and provides an overview of all annotations in a confusion report, providing advice to the lecturer on what points are interesting for discussion. The lecturer reviews the annotations, checks and releases student grades, and organises discussions. Make sure to plan enough time for this; typically, we advise starting no later than an afternoon before the lecture starts, and setting aside two hours of review time for two hours of lecture time. With more experience, about one hour of review time is needed for two hours of lecture time. To plan the lecture, first, take some time to think back on why this text is included in your course, then use the confusion report, browse through the most active discussion threads, and review the most highly upvoted questions and discussions.

The workflow typically looks like this:
 

Based on your particular course requirements, variations on this template are possible. Many styles of courses have been supported. Some examples: Courses have been set up following a two-day schedule, students have been tasked with organizing in-class discussions themselves, a large group of students (over 200) have annotated on a poem, and in one particular course students self-produced texts that were annotated. With this setup, we can ensure that Perusall works to help students prepare for class.

 

Giving feedback to students

This article gives guidelines for steps to take to give feedback, after they have finished Perusall assignments. 

 

 

Think of what you want to see from your students

As a lecturer, first make a list of the most important topics that you think should be addressed. What do you need or want to see from students in order to reach the learning objectives? On what pages would you expect many annotations? Which steps would you like them to follow?

Generate confusion report

A confusion report is a summary of the questions of students. When the deadline for your assignment has passed, and student annotations are gathered, click on the assignment, then click on “Confusion report” to generate the report. Perusall needs around 20 unanswered questions to generate the confusion report. 

Check whether or not you agree with the topics as mentioned in the confusion report. In the confusion report, you can click on the little icon next to the student comments to copy a comment over to your (Powerpoint) presentation. 

To view the confusion report, first, click the assignment. Now the button for the confusion report will become available and the confusion report will appear once you click on the button. To copy an annotation from the confusion report, click the copy button next to the annotation. At the bottom of the confusion report, you can find the buttons for printing or emailing the confusion report. For a demonstration, see the image below. Some of the analytics reports are computationally-intensive to generate, so they are updated once a day. The updated-at time is shown at the bottom of the report.

Check discussions in Perusall

Go to the screen on Perusall where you see the text and the number of Perusall comments (see figure), check whether you have missed big discussion topics (the number behind the conversation indicates the amount of annotations):

 

 

The topics that you want to discuss, the topics, from the confusion report and the topics you have identified from the annotations together make a list of topics to address during your face-to-face lecture. Clicking on the link icon under the students comments will copy the link to the discussion to the clipboard. You can copy this into your Powerpoint presentation. When you are logged into Perusall clicking the link will jump straight to the discussion in Perusall. 

 

 

Decide on in-lecture activities to address topics

Per topic: decide what will be the best way to address this. Think about how to make your students active. Let them find the answers. Organize a discussion, pass the question on to peer-students. Reward students for good annotations, but praising them (“that was a good comments, but…”). Maybe the topic goes beyond the content of your course, you could point students on to a follow-up course where this is addressed, or give them additional literature (it can also be a cliffhanger). Also, you can still give short presentations (Powerpoint) to explain some content to your students. 

Students do not always wish to speak up in a group. A good in-between activity is to organise small group discussions first, then discuss the results with the whole group. You can ask for a spokesperson per group to share discussion results. Youtube has great timer videos to keep students at task. 

Many more ideas to activate your lectures: https://teaching.wikit.itu.dk/Activities+during+the+lecture 

Help for organising discussions in class: https://cms.bsu.edu/-/media/www/departmentalcontent/educational%20excellence/pdf/beatty-2006deq.pdf?la=en

Follow-up in class

In class, show Perusall on the projector, and jump within discussions. Decide on whether or not you want students to join in Perusall during the lecture. Laptop screens may be a distraction. On the other hand, it can be good for students to check things back, and keep adding to the discussion during the lecture. 

Testing and assessment

To use Perusall in your course, it is recommended to reward its use by giving points for it, counting towards the total course mark. Students spend a significant amount of time working in Perusall, so it is fair they are rewarded. It needs some thought to make this work for your particular course.

Perusall grading within a course

First, check your course description in Ocasys to see what you have promised the students. It is important that you have written in Ocasys that students are graded on “assignments”, or better, that the Perusall assignments are described, since Ocasys is legally binding.

Typically, a very small fraction of the total points for a course is enough to invite students to do their work in Perusall. The system is set up to reward studying, so, when students make an effort to do their annotations, it is easy to reach the full points for it. The Perusall part of the grade is typically arranged to reward students for inputs -- their effort and engagement -- rather than differentiate between students based on their learning outcomes, which will be handled by your exams, papers, etc. Ideally every student would get full credit for the Perusall part of their course grade!

For your course manual you can make a table similar to this:
 

Component

Weight

Written exam (Digital Exam)

50 %

Literature review

25 %

Literature review outline and reading list

5 %

Perusall assignments

10 %

Participation

10 %

Total

100 %

 

In this case, Perusall assignments count for 10 % of the course grade. With 5 assignments, they would count for 2 % each.

We suggest that you provide students with two documents relating to Perusall. First, you can download and modify a one-page rubric that explains what Perusall is, how it works, and how students are graded. (You may need to edit this document to reflect the scoring settings you have set for your course.) You can also provide students with a set of example annotations with associated quality scores and an explanation for each score, to help them get a feel for what sorts of comments and questions they should be posting.

Feedback to students in class focuses on picking out discussion points, and picking out good examples of student annotations. You can explain why you think this example is particularly good. Thus, you are giving feedback on level of student annotations (see the section about learning outcomes).

Automatic calculation of student scores

Based on the criteria of Perusall, students are graded on quantity, timeliness, distribution and quality. Based on these criteria, individual scores for students are calculated.

In practice, when students make an effort, they will receive full credit for the Perusall assignments. We advise only reviewing student scores that are marginal (say 4 to 6 out of 10). Scores below this number will be generally very poor; scores above this number will be fine. You can manually change the overall student score, or change the points given for a particular annotation.


  

In the example this particular student has received 4 out of 10 points. Of the required 7 annotations, 4 of them were submitted on time, the average quality of the annotations was 1 (out of a scale of 0, 1, 2), and this particular student has lost points on distribution, by not spreading the annotations over the text evenly. (Perusall makes this latter deduction to encourage students to engage with the full text, not just the first few pages.) This leads to an overall score of 4 out of 10 points.

In the gradebook for the overall course, you can see the overview of the student grades per assignment. Typically, though feedback on their annotations, students learn what kind of annotations are expected, and thus their scores gradually increase over time.

 

 

How does automatic scoring work?

The quality of annotations is determined by a machine learning algorithm. Annotations are scored by a set of defining features. Via a process of repeatedly feeding the system data, and comparing it to data that is scored the weight (importance) of the defining features is determined.

To give an idea of how this works we can describe how Perusall was trained to deal with texts in Dutch. Three lecturers were asked to score a total of 1000 annotations on the same text. After scoring the first 100 annotations, they met up to discuss the differences among their grading, to reach a better inter-rater reliability. The algorithm was changed to reflect the agreement between the lecturers, after which the lecturers were asked to do another set of 100 annotations. This was done in 10 sets, hereby training the algorithm. So, basically, we told the system what we think are good annotations, and the system learned what are defining features of our “good annotations”.

References

Biggs, J. and Tang, C. (2011). Teaching for Quality Learning at University. Maidenhead: McGraw-Hill.


Miller, K., Lukoff, B. King, G. & Mazur, E. (2018). Use of a Social Annotation Platform for Pre-Class Reading Assignments in a Flipped Introductory Physics Class. Frontiers in Higher Education. https://doi.org/10.3389/feduc.2018.00008

Getting your course material into Perusall

Perusall can work with several types of materials. Most lectures work with selected text from journals. Standard, uploads are done by pdf, but web pages and open content materials can be added as well. For books, Perusall has made agreements with many publishers. Most large publishers are in (such as Sage, Pearson, Wiley, Wolters-Kluwer) and many others are there as well. When you decide to use a book from Perusall’s catalog ( https://app.perusall.com/catalog ) for use in your course, students will be prompted to purchase the book online within Perusall the first time they go to access a reading assignment from the book.

Copyright

To make sure we do not breach copyrights, we have a procedure to check course literature with the library. Once the copyrights for your course literature have been cleared, you can use the materials in your course. As a rule of thumb, most literature that can be found in the UoG library systems can be used in Perusall as well. 

 

Demo: How does it work?

A demo can be found at: https://app.perusall.com/demo

 

 

Assignment options

In the images and examples below, we will shortly explain how to use Perusall. Navigating to an assignment in Perusall should always be done through Nestor. This is necessary for getting your credentials right and making sure Perusall can communicate back to Nestor. You should never have to log in to Perusall, as this is handled by Nestor.

 

  1. Select the document you would like to assign and click the assign button.
  2. Select the range of the document for the assignment.

    • The first option is “Assign from the table of content”. This option allows you to select one or more content sections of the document. The second option is: “Assign a page range.” This option allows you to assign a page range for this assignment (ex. page 1-15). The last option is: ”Assign entire document”. This option allows you to assign the entire document.

  3. Set the deadline for the assignment.

  4. Fill in the name of the assignment (this should be the same as the name of the Nestor assignment).

  5. Set the number of annotations to grade.

 
 

 

Buttons explained

 

  1. Perusall button brings you to Perusall home
  2. This button represents the Course Code for your course. When clicked, it redirects you to your course. 
  3. Title of the document
  4. Back to last read location
  5. Change the readsize of your document
  6. Select the comments that have to be displayed
  7. Make text annotations
  8. Make image annotations
  9. Page selector
  10. Your account settings

 

Show comments for a specific student

  1. Click the all comments button (button no. 6)
  2. In the menu select a specific student

  3. You can now select a specific student

  4. Now only the comments for this specific student show up

 

Automatic vs manual groups

Perusall automatically groups students in the course into smaller "discussion groups," by default with about 20 students. Students will only see the comments of other students in their group, plus any threads that an instructor has started. Each document has a different set of groups formed, so students have an opportunity to interact with different people over the course of the semester. Groups for a particular document are fixed once the document is uploaded, to ensure students have a consistent view of each document.

If you prefer to manually place students into groups, you can do so under the Grouping Options tab of the course settings. Here you can manually assign students to groups by entering a group number next to each student in your roster. For example, to place students A, B, C, and D into two groups, you could assign students A and B to group 1 and students C and D to group 2. Students that enroll in your course after you set the groups will be placed into one of the groups at random, but you can change this in the Grouping Options at any time.