In choosing an LMS, the committee prioritized several key pathways of gathering feedback. User testing was at the heart of this process, providing students, faculty, and staff multiple opportunities to evaluate the LMS platforms. User data-collection took place over two main stages:
- Interviews and Surveys: Determined faculty, staff, and students’ satisfaction with the current LMS and desired features for a future product through in-depth interviews and a widely distributed survey.
- Evaluating LMSs: Faculty, staff, and students evaluated potential LMSs and their features. Users tried the LMS test environments hands-on and watched vendor-created videos of the most used features.
Teaching and Learning Centers promoted these opportunities to engage in data collection through customized communication plans. For example, they used mass emails, sent targeted invitations, promoted at division events, presented to departments, coordinated messages from division leadership, hosted live demos and office hours, orchestrated test cases, and more.
The process also included a technical evaluation of the systems. IT@JH and Student Disability Services completed the foundational evaluations to ensure that the vendors could comply with the University’s policies and basic needs. The teaching and learning center staff from each division evaluated the possible LMS platforms. Using their technical knowledge and experience, they judged division-specific requirements as well as rated each vendor on the Request for Proposal requirements. They attended vendor demonstrations, completed in-depth, hands-on testing in the LMSs, and facilitated user testing.
Our current LMS, self-hosted Blackboard Learn, will no longer be supported by the vendor. This system retirement allowed us to explore modern, cloud-based LMSs that could better meet the University’s needs for an excellent learning platform. After a robust evaluation process, faculty, staff, students, and the Teaching and Learning Centers rated Instructure Canvas as the best system.
Evaluation Process Details
|LMS Requirements Development||Fall 2018 – Sumer 2019||As the end of our current Blackboard Learn contract neared, the Instructional Technology Advisory Council (ITAC) summarized and prioritized fifty requirements provided by the divisions for an LMS. With the rise of the COVID-19 pandemic, the project was put on hold.|
|Blackboard Product Retirement Announced||July 2020||Blackboard announced the end of support for self-hosted Blackboard Learn. They would require migration to their cloud-based services, Blackboard Software as a Service (SaaS), and ultimately a move to Blackboard Ultra, their new product.|
|Release of Request for Proposal||December 2020||After evaluation of business needs, JHU released an RFP.|
|Vendors’ Proposals Submitted||January 2021||Five vendors responded with proposals.|
|Round 1 Vendor Review||January 2021||Vendor proposals were evaluated by teaching and learning center staff and IT@JH to see if they met the RFP requirements. Three of the five vendors were selected for testing and further evaluation.|
|LMS Steering Committee Formed||February 2021||Sponsored by Stephen Gange, the steering committee includes faculty, students, IT@JH, representatives from each division’s teaching and learning center, and other leadership.|
|Vendor Presentations||February – April 2021||The LMS vendors offered a series of presentations demoing their product and particular features of interest to divisions. They also hosted office hours for teaching and learning staff questions.|
|Stage 1 of User Evaluation||February – March 2021||Thousands of faculty, staff, and students completed surveys and in-depth interviews, providing data about their satisfaction with the current LMS and what they would want out of a future one.|
|IT@JH and Student Disability Services Testing||March – May 2021||IT@JH completed testing of the LMSs to see if the product would meet the University’s needs. The evaluation included security, data, system administration, and more. Student Disability Services also reviewed the LMSs for accessibility considering students with disabilities.|
|Stage 2 User and TLC (Teaching and Learning Center) Evaluation||April – May 2021||Faculty, staff, and students provided feedback after completing hands-on testing of the potential LMSs and watching demos. Teaching and learning centers completed in-depth testing of the LMSs, rating them on the RFP requirements and divisional needs.|
|LMS Committee Recommendation||June 2021||The committee reviewed the aggregated data from faculty, staff, students, teaching and learning centers, IT@JH, SDS. The evaluation overwhelmingly favored adopting Instructure Canvas.|