Practical Pathways to Using Open-Ended Student Responses

My name is Ben Naismith, and I am a PhD candidate in the Department of Linguistics, entering the fifth year of the program. Prior to undertaking a PhD, I had spent 15 years in the field of Teaching English as a Foreign Language, and it is this professional experience which has shaped my research interests in vocabulary pedagogy and development. Currently, I am writing my dissertation, which focuses on language learners’ use of vocabulary in their writing, specifically different aspects of collocational proficiency (words that go together like conduct research or make an assumption). I am most interested in comparing how the features that expert human raters notice compare to quantitative automated metrics.

The project I have been a part of this summer for the Humanities Engage Administrative Micro-Internship is the Pedagogy-focused Text-Analysis Software Project. This project is hosted by the University Center for Teaching and Learning. In brief, this project is an analysis of students’ comments to open-ended questions as part of an end-of-course survey. The course is an online one-credit course entitled Anti-Black Racism that the University of Pittsburgh required undergraduates to take during the 2020-2021 academic year. Courses such as these have large enrollment, making it less practical for instructors to read through thousands of individual responses. To make the best use of these invaluable data, our project investigated potential computational tools for extracting salient patterns through sentiment analysis, topic extraction, text clustering, and lexical analyses. The long-term goal of the project is to find practical pathways for teachers and other stakeholders to make use of responses from open-ended data. The short-term outcomes of this initial stage of the project were a) familiarization with potential computational tools for analyzing meaning, and b) a better understanding of the challenges involved in this undertaking and directions for future potential research. The project’s team consisted of myself, a former dean at Pitt, two faculty members of the University Center for Teaching and Learning, and an undergraduate intern.

As a Humanist seeking a doctorate, my main takeaway from this project was the hands-on experience of working on an ambitious endeavor with appreciable real-world impact. The micro-internship afforded me the opportunity to work with teaching experts with whom I might not have otherwise come into contact, and it was interesting to observe the different perspectives and skillsets that everyone brought with them. In terms of practical skills development, as the team member with the most coding experience, I was tasked with processing the dataset and exploring analysis options using Python. These coding skills are a direct result of my academic preparation that I have received at Pitt, from courses such as Data science for Linguists, and from my work as graduate student researcher with the English Language Institute Data Mining Group.

In applying this previous training to the micro-internship project, I developed further technical skills relating to machine learning and analyzing meaning in texts. These skills will be of critical importance for my future professional goals and postdoctoral career exploration, regardless of whether I pursue employment in academia or industry. For example, if I ultimately dedicate my time to research, these skills can be leveraged for any number of other linguistic projects, especially those relating to student texts. Similarly, in the fields of educational technology and natural language processing, my newfound knowledge will allow me to contribute in a range of work environments that work with linguistic data.

Overall, the experience of this eight-week micro-internship was a positive one, leading to personal and professional development that I had not anticipated prior to the outset. Of course, there were challenges to overcome along the way as the team learned more about the inherent difficulties of this type of data analysis. However, through collaboration and flexibility on the part of team members, the group was able to pivot and to continue to make progress on achieving the project’s overarching goals. Based on this experience, I would highly recommend to other students in similar situations to seek out potential micro-internships of their own. Regardless of the outcome of the project, the impact on one’s own professional development will invariably be positive, and it may well lead to discovering new career and academic interests which had previously not been considered. If possible, having departments more formally incorporate these projects into their programs would serve to increase the visibility and relevance of these opportunities.

Ben Naismith
Linguistics
September 2021
 
Learn about all the Administrative Micro-Internships.