Software systems should never stand still. They should constantly be evolving, striving to provide a better, more modern experience for their users. The developers and product managers who support and maintain them should always be asking the questions:
“Is our system becoming outdated? Is it serving our customers’ needs? Can we do this better?”
For a system like Sherpa, established in 2002 as a ‘one stop shop’ to help academics and research staff discover and understand the policies associated with open access publishing, this is no easy task.
One system – three distinct datasets
Photo by Scott Graham via Unsplash
In the first 20 years of life, we have built up a huge database within the system containing tens of thousands of records. These records are collected into three distinct datasets, presented as separate but interconnected platforms which have been used by research institutions worldwide to make informed decisions about academic publishing.
This volume and complexity of data alone makes the idea of updating the service a daunting proposition, but coupled with the number of tools and actively used APIs that also comprise the service, the software development team tasked with updating the system and modernising the platform faced a real challenge.
Migrating and modernising
Photo by Kaleidico via Unsplash
In 2022, the redevelopment team was given the remit to not only migrate the service off its EPrints based architecture, but also to modernise the user interface. Bringing the multiple systems together into a single point of access would allow users to discover more of the available dataset and enable enrichment of the data provided. The open access publishing landscape is constantly changing and evolving, so flexibility and scalability was a key consideration with this new architecture, as was long term maintainability within Jisc.
So how did the team tackle this task of migrating the service to its new home while making sure that it could still function for its users and minimise disruption? By exploring many modern approaches to system architecture, we quickly realised that we should not strive for a complete migration of the service as a first deliverable, but instead look at what we could achieve in increments.
This would not only enable us to gradually move the system to where it needed to be, but also take advantage of being able to build this new system in the open, allowing our users to experience new features as they were implemented and give us valuable feedback to inform future development. To help us achieve this, we turned to the systems architecture methodology commonly known as ‘jamstack’.
Design and building a modern user interface
Photo by Kelly Sikkema via Unsplash
Jamstack methodology describes software that incorporates loosely coupled components which rely on robust APIs to transfer data to different parts of the system which ultimately have little to do with one another. This ‘building block’ style approach allows the core data to operate independently from any user interface elements, providing opportunities for software to rapidly scale and develop as the system requires it, rather than being built with high degrees of complexity from the start. Embracing this methodology would be the key to the success of the project and allow for user visibility from its early stages.
The key part of the new service was its ‘unification’, bringing together the three core systems of Romeo, Juliet and OpenDOAR, each of which represented a different dataset within the existing system, together in one single user interface. By making use of the service’s existing APIs, we were able to design and build a modern user interface using the NextJS framework to tap into the existing data to present this unified concept.
This provided users with a single point of entry, allowing them to perform a search which surfaced data from across the three different datasets, rather than just a single record type. This change brought visibility to some parts of the service that users may not normally have explored. After just under a year in design and development, the first pass of this new user interface was released into public beta in March 2023 to allow users to experience it and offer feedback.
The next challenge
Photo by Alicia Christin Gerald via Unsplash
We then turned to our next challenge – migrating the extensive and, in some places, complex dataset, to a new data management system. Again, embracing jamstack concepts to allow us to be informed but move quickly, we took advantage of existing EPrints APIs and looked at how we could move forward, migrating each dataset while minimising disruption to the service and its users (internal and external).
Using the recently released Open Access for Books feature as a case study to build and refine our initial infrastructure, as well as experiment with the systems we have invested in, has given us valuable insight into how we can fully realise this task, which will be the team’s focus of development during 2024.
Lessons learned
Photo by Timur Shakerzianov via Unsplash
Throughout all of this, though, one key thing has remained relevant and important – the ability to listen and respond to our users to help inform how we move forward with the service.
By embracing jamstack and by working iteratively to release scalable systems, perform multiple instances of user testing and feedback, we have built a service that is not only modern and supportable, but is also able to adapt easily to the requirements of our users and the sector at large.
Get involved
We want your feedback! If you have any comments on the current version of the redeveloped service, please email help@jisc.ac.uk.
