More than 12 months of collaboration between developers from the Atlas of Living Australia (ALA) and the GBIF Secretariat has produced a major upgrade to the ALA’s systems that makes them more reliable, more robust and better equipped to manage increasing volumes of biodiversity data.
The new infrastructure, using the shared GBIF/ALA codebase, increases data indexing speeds for the ALA, reduces its operational costs and reduces long-term technical debt accumulated by developing independent systems. These efficiencies are critical for the ALA’s infrastructure to keep pace with data growing at a rate of more than one million species occurrence records every year.
Users have already reported that ALA’s systems are more responsive, particularly when searching species data, navigating the maps and using the Spatial Portal analytical tool, reports Javier Molina, project lead for the Core Infrastructure Upgrade Project. Publishers may also see improvements, given that the infrastructure smoothly processed an update to a dataset of more than 20 million records from eBird Australia.
“A dataset that large would have created lots of headaches for us in the old system,” said Molina. “Prior to the upgrade, we were experiencing about some unscheduled outages every month. Since this portion of the project went live in June, we haven’t received any reports of unscheduled outages, and the system has coped well during peak utilization.”
The improvements have also generated an immediate return on the teams’ investment in the collaboration. During its first day in production, the cost of operating the ALA infrastructure was reduced by 43 per cent, a rate that will net annual savings of nearly €50,000 (AU$ 79,000).
“As a result of working together to develop this major software upgrade, we have more affordable infrastructure that performs better,” said Andre Zerger, director of the ALA. “We expect to see additional savings from running more stable systems and co-maintaining a codebase that is more up-to-date.”
The recent upgrade also improves the consistency in how the ALA, GBIF and other biodiversity infrastructures process data. Given that many users of the ALA also frequently access data through GBIF, uniform handling of data flags and interpretations eliminates potential sources of confusion and enables them to use both platforms with proficiency.
“The ALA and GBIF development teams have worked so well together—there’s a strong sense of trust and confidence in the teams’ respective skills that leads to individual contributions that contribute to the larger whole,” said Joe Miller, executive secretary of GBIF and former research group leader at the Centre for Australian National Biodiversity Research, which (like the ALA) is hosted by CSIRO. “Our teams know that accessible, high-quality biodiversity data is improving research and environmental outcomes across the globe and that their efforts are integral to making this happen.”
With this first stage of the project complete, the ALA will set its sights on simplifying data ingestion processes and further collaboration with GBIF on shared goals such as establishing common vocabularies and managing DNA-derived records more consistently.
Last, and most definitely not least, comes engagement of the wider Living Atlases community. This group comprises more than two dozen international teams in the GBIF network who implement the open-source platform originally developed by the ALA in in other national, thematic, regional and institutional settings.
“Our collaborative work with GBIF has built social capital in the global community,” said Molina, “and it is an excellent entry point for the International Living Atlases community to jump onboard.”