From Educators to Artificial Intelligence: Three Perspectives on Writing Support
Many students whose last years of high school were disrupted by the pandemic are struggling academically in foundational college courses.
Artificial Intelligence, or AI, has developed to the point where it can provide meaningful support to writing instruction.
Learn how AI writing tools are being used at the University of Rochester, Florida International University, and Arkansas State University, and the successes each institution is seeing as a result.
The Process:
Each use case implements AI technology differently, and each institution has seen a drastic improvement in student writing proficiency.
The insights delivered by each artificial engine helps to support each educator in effectively improving student writing. IntelliMetric® benchmark assessments provide an empirical way to measure growth, while IntelliWriter™ provides students with immediate feedback and data on their writing. The AI technologies equip faculty and tutors with the knowledge needed to guide students quickly and effectively on a path towards writing proficiency.
Case Study 1: Rochester University
Interview with Dr. Rebecca Red Wolf, Coordinator of the Writing Center:
“We are looking at writing across the curriculum. What we want to do is have the AI assessment at the beginning of the program and assesses them at the end of the program to see if writing development is continuing.”
Play Video
After years of identifying a deficiency in writing proficiency for Master’s Nursing students, The University of Rochester sought out a remedy. Dr. Rebecca Red Wolf, the Coordinator of the Writing Center at the University of Rochester, designed a writing course to incorporate unique aspects such as clarified expectations, formative feedback, scaffolding techniques, opportunities for reflection, self-directed learning and faculty development while incorporating an IntelliMetric® benchmark writing assessment to measure the impact of the course in students’ writing growth towards proficiency. IntelliMetric® is an AI engine designed to accurately and consistently score student writing. All students were delivered a pre- and post-assessment at the beginning of the course that scored on a 6pt scale. The results of her study were used in her dissertation, and you can watch her presentation on the study and her dissertation above.
Figure 1:Demographic Data Fall 2021 Demographics for Students Enrolled in The Writing Workshop
Improvement in Self-Efficacy: Self-Efficacy Survey Results
Sample t-tests were used to compare the pre- and post- self-efficacy and task value scores. Cohen’s (1988) d was calculated as a measure of the effect size, with the interpretation following Cohen’s convention: .20 (small effect), .50 (medium effect) and .80 (large effect). Statistically significant differences between pre- and post-scores were found with respect to writing essentials (t (49) = 6.8, p < 0.001; d = .96), relational reflective writing (t (48) = 7.7, p < 0.001; d = 1.1), and creative identity (t (47) = 7.8 p < 0.001; d = 1.12). All three sub-scales for self-efficacy revealed significantly higher scores on the post-test compared to the pre-test, with large effect sizes.
Case Study 2: Florida International University
Interview with Dr. Vicenta Shepard, Director of the Center for Academic Success:
“We are continually expanding the use of IntelliWriter with different programs interested in using the tool at the curricular level. We have approval to renew our licenses for the next academic year.”
Play Video
Florida International University (FIU) has used IntelliWriter™ since January of 2021 as a way to better support those students that visit the Center for Academic Success and the tutors that are employed by the center. FIU is a Hispanic serving institute (HSI) supporting more than 60,000 students annually. Since implementing IntelliWriter™ at FIU, they are seeing an average improvement in student writing by up to 30%.
FIU’s implementation of IntelliWriter™ began in the Center for Academic Success, led by its’ Director, Vicenta Shepard. Tutors utilize the program to support students that visit the center and attend their workshops throughout the year. The tutors also utilize the Writer’s Guides provided in IntelliWriter™ as resources for their workshops.
Students and tutors have the ability to collaborate virtually within the platform. Tutors receive student work accompanied by AI data analytics prior to tutoring sessions (virtually or in-person) to better coach students on how to improve their writing.
As students write in the platform, they receive immediate feedback and guidance on how to improve their writing. The feedback delivered can be converted into nine (9) different languages making IntelliWriter™ ideal for ELL students.
Students are provided with color-coded indicators aligned with the domains of writing to clearly see which areas need improvement.
FIU Center for Student Success
The Center for Academic Success also partnered with a variety of Departments, including the Biology Department, the Psychology Department, and the Department of Criminology and Criminal Justice to launch IntelliWriter™ at the curricular level.
The Department of Criminology and Criminal Justice expanded access to the platform via an LTI integration with Canvas and have seen staggering improvements in student writing ability as a result.
Tutors, faculty, and administrators have access to the companion analytics technology, WriteVue™, to measure writing growth. With WriteVue™, tutors can filter to a single student to view the average holistic and per domain scores and most common errors.
With WriteVue™, tutors and administrators can see the total number of users and the average holistic scores and domain scores including Content Development, Organization, Language Use & Style, and Grammar & Mechanics.
WriteVue™ also provides the holistic and per domain scores for every document written by a student within the IntelliWriter™ platform. WriteVue™ also provides statistics on average holistic scores by month alongside usage data. The table below displays FIU data for all users of the IntelliWriter™ platform for all of 2021. The purple bar indicates the activity, while the green bar illustrates the average holistic score for that month. Note that when students are most active in the system (i.e. January) we also see a correlation to high average holistic scores (i.e. 3.78 in January). In the sample, student writing improved from a 2.55 in August to a 3.27 in December (a 28% increase).
Department of Criminology and Criminal Justice LTI Pilot
In Spring 2022, the Department of Criminology and Criminal Justice opted to use IntelliWriter™ in conjunction with their capstone course, which requires a research paper at the end of the course. In order to provide students in the course with access to IntelliWriter™, the department piloted an LTI integration with IntelliWriter™ into Canvas. As a result, FIU was able to get writing data aligned with a single course and sections without relying on students to self-identify. The data is aligned by student and course via the integration.
The data captured for this cohort was different than the data for students that visited the Center for Academic Success. Their average holistic score was 4.64 and this cohort struggled the most with Grammar & Mechanics.
A measurement of writing improvement was also found via the LTI, with students starting with an average holistic score of a 3.02 in January and concluding the semester at a 4.55 in March (a 50% increase).
The Florida International University program illustrates a successful case study of the use of IntelliWriter™. FIU anticipates continued use of IntelliWriter™ in the Center for Academic Success and expanding their use of the platform at the curricular level via the LTI integration for the 2022/2023 academic year.
Case Study 3: Arkansas State University
Interview with Arkansas State University:
“In 2018, our students performed poorly in style and mechanics. Other forms of intervention have not proven successful. We piloted the IntelliWriter AI tool and produced great results. Our Department of Psychology has since implemented it and the University leadership team is considering a full scale rollout.”
The business school at Arkansas State University piloted Vantage Learning’s virtual writing support tool, IntelliWriter™ (formerly CorrectEnglish®), in their business program for two terms in the 2020-2021 academic year to support and improve their students’ writing proficiency. IntelliWriter™ offers immediate writing support powered by Vantage’s patented automated essay scoring engine, IntelliMetric®. IntelliWriter™ features in-text revision tips, feedback aligned with the domains of writing, multi-lingual and cross-disciplinary writing instruction, and the ability to virtually collaborate with peers, faculty and tutors. Arkansas State provided access to IntelliWriter™ to all students in multiple courses each term and utilized a benchmark assessment scored by IntelliMetric® to measure writing improvement with IntelliWriter™ as the “treatment.” The following will provide an overview of their experience using the IntelliWriter™ platform and an analysis of the results of the benchmark assessments which illustrates evidence of continued writing improvement each term.
The Field Study:
- The study took place over the Fall and Spring semesters of the 2020-21 academic year at Arkansas State University.
- The students involved were taking a Survey of Management Information Systems (MIS) class in the College of Business.
- The classes were taught online during both semesters due to COVID-19 restrictions.
- All sections of the classes were taught in an identical fashion for both fall and spring semesters; all assignments, grading weights, time frames, lectures, and class meetings remained the same for both semesters.
- Assignments requiring a major writing component constituted 80% of each student’s final grade – two essay exams (25% each), term paper (25%), and two developmental essays (2.5% each).
- Students were required to handle MIS concepts in the essay exams and term paper, but not in the developmental exams. The developmental essay scores focused inherently with writing composition and provide results that were not comingled with a student’s understanding of MIS.
- The two developmental essays were scored by IntelliMetric®, submitted as a Word file from a different cloud site; there was no way of knowing if a student had used IntelliWriter™ or not.
Fall '20 Conclusions
- During the fall semester, the groups of who began using IntelliWriter™ (“Users”) and those who did not (“Non-Users”) began with equivalent writing skills, based on the DE1 results.
- The Users scored higher on DE2 than the Non-Users.
- Additionally, the Users improved their writing performance by an average of 8.8% from DE1 to DE2, while the Non-Users’ performance remained unchanged.
- There is no evidence that this improvement in performance can be attributed to differences in scholastic achievement, as there was no difference found in overall class average between the two groups.
Spring '21 Conclusions
- Non-Users have a higher baseline set of writing skills than the students that became Users . This is evidenced by DE1 Non-Users 90.12 average versus 85.78 for Users.
- By the end of the semester, the User students appear to have caught-up with the Non-User students, as there was no statistically significant difference in DE2 performance between the two groups.
- During the semester, Users’ performance improved by an average of 5.7%, while the Non-Users’ performance did not change.
- This improvement for Users cannot be attributed to higher scholastic achievement by the Users.
- There was no difference found between the groups in terms of overall class average.
