Digital scholarship blog

This blog post is by Harry Lloyd, Research Software Engineer in the Digital Research team, British Library. You can sometimes find him at the Rose and Crown in Kentish Town.

Last week Dr Adi Keinan-Schoonbaert delved into the invaluable work that she and others have done on the Convert-a-Card project since 2015. In this post, I’m going to pick up where she left off, and describe how we’ve been automating parts of the workflow. When I joined the British Library in February, Victoria Morris and former colleague Giorgia Tolfo had prototyped programmatically extracting entities from transcribed catalogue cards and searching by title and author in the OCLC WorldCat database for any close matches. I have been building on this work, and addressing the last yellow rectangle below: “Curator disambiguation and resolution”. Namely how curators choose between OCLC results and develop a MARC record fit for ingest into British Library systems.

The Convert-a-Card workflow at the start of 2023

Entity Extraction

We’re currently working with the digitised images from two drawers of cards, one Urdu and one Chinese. Adi and Giorgia used a layout model on Transkribus to successfully tag different entities on the Urdu cards. The transcribed XML output then had ‘title’, ‘shelfmark’ and ‘author’ tags for the relevant text, making them easy to extract.

Card with layout model and resulting XML for an Urdu card, showing the `structure {type:title;}` parameter on line one

The same method didn’t work for the Chinese cards, possibly because the cards are less consistently structured. There is, however, consistency in the vertical order of entities on the card: shelfmark comes above title comes above author. This meant I could reuse some code we developed for Rossitza Atanassova’s Incunabula project, which reliably retrieved title and author (and occasionally an ISBN).

Chinese cards. Although the layouts are variable, shelfmark is reliably the first line, with title and author following.

Querying OCLC WorldCat

With the title and author for each card, we were set-up to query WorldCat, but how to do this when there are over two thousand cards in these two drawers alone? Victoria and Giorgia made impressive progress combining Python wrappers for the Z39.50 protocol (PyZ3950) and MARC format (Pymarc). With their prototype, a lot of googling of ASN.1, BER and Z39.50, and a couple of quiet weeks drifting through the web of references between the two packages, I built something that could turn a table of titles and authors for the Chinese cards into a list of MARC records. I had also brushed up on enough UTF-8 to fix why none of the Chinese characters were encoded correctly.

For all that I enjoyed trawling through it, Z39.50 is, in the words of a 1999 tutorial, “rather hard to penetrate” and nearly 35 years old. PyZ39.50, the Python wrapper, hasn’t been maintained for two years, and making any changes to the code is a painstaking process. While Z39.50 remains widely used for transferring information between libraries, that doesn’t mean there aren’t better ways of doing things, and in the name of modernity OCLC offer a suite of APIs for their services. Crucially there are endpoints on their Metadata API that allow search and retrieval of records in MARCXML format. As the British Library maintains a cataloguing subscription to OCLC, we have access to the APIs, so all that’s needed is a call to the OCLC OAuth Server, a search on the Metadata API using title and author, then retrieval of the MARCXML for any results. This is very straightforward in Python, and with the Requests package and about ten lines of code we can have our MARCXML matches.

Selecting Matches

At all stages of the project we’ve needed someone to select the best match for a card from WorldCat search results. This responsibility currently lies with curators and cataloguers from the relevant collection area. With that audience in mind, I needed a way to present MARC data from WorldCat so curators could compare the MARC fields for different matches. The solution needed to let a cataloguer choose a card, show the card and a table with the MARC fields for each WorldCat result, and ideally provide filters so curators could use domain knowledge to filter out bad results. I put out a call on the cross-government data science network, and a colleague in the 10DS data science team suggested Streamlit.

Streamlit is a Python package that allows fast development of web apps without needing to be a web app developer (which is handy as I’m not one). Adding Streamlit commands to the script that processes WorldCat MARC records into a dataframe quickly turned it into a functioning web app. The app reads in a dataframe of the cards in one drawer and their potential worldcat matches, and presents it as a table of cards to choose from. You then see the image of the card you’re working on and a MARC field table for the relevant WorldCat matches. This side-by-side view makes it easy to scan across a particular MARC field, and exclude matches that have, for example, the wrong physical dimensions. There’s a filter for cataloguing language, sort options for things like number of subject access fields and total number of fields, and the ability to remove bad matches from view. Once the cataloguer has chosen a match they can save a match to the original dataframe, or note that there were no good matches, or only a partial match.

Screenshot from the Streamlit Convert-a-Card web app, showing the card and the MARC table curators use to choose between matches. As the cataloguers are familiar with MARC, providing the raw fields is the easiest way to choose between matches.

After some very positive initial feedback, we sat down with the Chinese curators and had them test the app out. That led to a fun, interactive, user experience focussed feedback session, and a whole host of GitHub issues on the repository for bugs and design suggestions. Behind the scenes discussion on where to host the app and data are ongoing and not straightforward, but this has been a deeply easy product to prototype, and I’m optimistic it will provide a light weight, gentle learning curve complement to full deriving software like Aleph (the Library’s main cataloguing system).

Next Steps

The project currently uses a range of technologies in  Transkribus, the OCLC APIs, and Streamlit, and tying these together has in itself been a success. Going forwards, we have the possibility of extracting non-English text from the cards to look forward to, and the richer list of entities this would make available. Working with the OCLC APIs has been a learning curve, and they’re not working perfectly yet, but they represent a relatively accessible option compared to Z39.50. And my hope for the Streamlit app is that it will be a useful tool beyond the project for wherever someone wants to use Worldcat to help derive records from minimal information. We still have challenges in terms of design, data storage, and hosting to overcome, but these discussions should have their own benefits in making future development easier. The goal for automation part of the project is a smooth flow of data from Transkribus, through OCLC and on to the curators, and while it’s not perfect, we’re definitely getting there.

Contents

Introduction

Share Family and National Bibliographies

       What is a National bibliography?

       BNB in the Share Family

Benefits

Future developments

Beta service

Further information

Introduction

The British National Bibliography (BNB), first published in January 1950, is a weekly listing of new books and journals published or distributed in the United Kingdom and the Republic of Ireland.  Over the last seventy-three years, the BNB has adapted to changing customer needs by embracing new technologies, from cards in the 1950s to mark-up languages for data exchange in the 1970s and CD-ROM in the 1980s. The BNB now provides online access to details of over 5 million publications and forthcoming titles, ranging in scope from computer science to history, from novels to textbooks.

1. Examples of British National Bibliography records, April 19th 2023. Please click the image to see it in full size & detail.

In 2011, the Library launched the Linked Open Data BNB.  At that time, linked data was an emerging technology using Web protocols to link data sets, as envisaged in Sir Tim Berners-Lee’s concept of a Semantic Web^[1].  Our initial foray into linked data was successful from a technical perspective. We were able to convert BNB data held in Machine Readable Cataloging (MARC) format into linked data structures and make it available in a variety of schemas under an open licence.  Nevertheless, we lacked the capacity to re-model our data in order to realise the potential of linked data.  As the technology matured, we began to look around for partners with whom we could collaborate to take BNB forward.

As described in my September 2020 blogpost, British Library Joins Share-VDE Linked Data Community, the British Library joined the Share Community (now the Share Family) to develop our linked data service. The Share Linked Data Environment is “a global family built on collaboration that brings libraries, archives and museums together with a common goal and joins their knowledge in an ever-widening network of inter-connected bibliographic data.” (Share Family, 2022).

Share Family and National Bibliographies

“The Share Family is a suite of innovative tools and services, developed and driven by libraries, for libraries, in an international collaborative, consortial effort. Share-VDE enables the discovery of knowledge to increase user engagement with library and cultural heritage collections.”^[2]

2. Share family components^[3]. Please click the image to see it in full size & detail.

The Share Family has supported us through the transition from our traditional MARC data to linked open data.  We provided a full copy of the British National Bibliography to the Share team for identification and clustering of entities, e.g. works, publications, persons. Working with colleagues from other institutions on Share-VDE working groups we contribute to the development of the underlying data structures and the presentation of data.  This collaborative approach has enabled delivery of the British National Bibliography as the first institutional tenant of the Share Family National Bibliographies Portal. 

What is a National bibliography?

“National bibliographies are a permanent record of the cultural and intellectual output of a nation or country, which is witnessed by its publishing output. They gather the bibliographic information of current publications to preserve and provide ongoing access to this record.”

IFLA Bibliography Section

The IFLA (International Federation of Library Associations and Institutions) Register of national bibliographies contains 52 entries, ranging from Andorra to Vietnam.  National bibliographies vary in scope, but each provides insights into the intellectual and cultural history of society, literature and publishing.  The Share Family National Bibliographies Portal offers the potential for clustering and searching multiple national bibliographies on a single platform.

BNB in the Share Family

3. Screenshot BNB home screen. Please click the image to see it in full size & detail.

The British Library is proud that the British National Bibliography is the first tenant selected for the Share Family National Bibliographies Portal.

BNB is now available to explore in Beta: https://bl.natbib-lod.org. You can search for publications, original works and people, as illustrated by these examples:

You can use the national bibliography to search for a specific publication, such as a large print edition of the novel Small island by Andrea Levy.

4. Screenshot: Bibliographic description of large print edition of Small Island by Andrea Levy. Please click the image to see it in full size & detail.

You can also find original works inspired by earlier works:

5. Screenshot: Results set for publication of the work, Small island by Helen Edmundso. Please click the image to see it in full size & detail.

Alternatively, you can search for works by a specific author… 

6. Screenshot: Original works by Douglas Adams. Please click the image to see it in full size & detail.

…or about a specific person

7. Screenshot: Original works about Douglas Adams. Please click the image to see it in full size & detail.

…or by organization

8. Screenshot: Results set for BBC. Please click the image to see it in full size & detail.

Benefits

What benefit do we expect to gain from this collaboration?

• We profit from practical experience our collaborators have gained through other linked data initiatives
• We gain access to a state of the art, extensible infrastructure designed for library data
• We gain a new channel for dissemination of the BNB, in aggregation with other national bibliographies

We are able to re-tool our metadata for the 21^st Century:

• Our data will be remodelled and clustered making it more compatible with current data models, including the IFLA Library Reference Model, RDA: Resource Description and Access, and Bibframe
• Our data will be enriched with URIs that will make it more effective in linked data environments
• The entity-centred view of the British National Bibliography offers new perspectives for researchers

Future developments

Conversion of the BNB and publication in the National Bibliographies Portal is only the beginning. 

• The BNB data from the Cluster Knowledge base will also be published in the triple store
• Original records will be available to the British Library as Bibframe 2.0, for dissemination or reuse as linked data
• Users will be provided with access to the data via data dumps and a SPARQL endpoint
• Our MARC records will be enriched with original Share URIs and URIs from external sources
• Other national bibliographies will join BNB in the national bibliographies portal

The British National Bibliography represents only a fraction of the Library’s data.   You can explore the British Library’s collection through our catalogue, which we plan to contribute to Share-VDE in future.

Beta service

The British National Bibliography in the Share Family is being made available in Beta. The service is still being tested. The interface and the functionality are subject to change and may not work for everyone.  You can tell us what you think about the service or report problems by contacting [email protected].

Further information:

British National Bibliography https://bnb.bl.uk  

Share VDE http://www.share-family.org/

Share Family wiki https://wiki.share-vde.org/wiki/Main_Page

Share VDE Virtual Discovery Environment in linked open data https://svde.org/

National Bibliographies in Linked Open Data https://natbib-lod.org

British National Bibliography Linked Open Data Portal https://bl.natbib-lod.org

Footnotes

[1]  Berners-Lee, Tim; James Hendler; Ora Lassila (May 17, 2001). "The Semantic Web". Appeared in: Scientific American. (284(5):34-43 (May 2001). 

[2] Share-VDE: supporting the creation, management and discovery of linked open data for libraries: executive summary. Share-VDE Executive Committee. December 7^th, 2022. Share-VDE Website (viewed 19^th June 2023)

[3] Share Family – Linked data ecosystem. How does it work?  http://www.share-family.org/  (viewed on 23^rd June 2023)

This is a guest post by Natalie Lucy, a PhD student at University College London, who recently undertook a British Library placement to work on a project Mapping Caribbean Diasporic Networks through the correspondence of Andrew Salkey.

Project Objectives

The project, supervised by curators Eleanor Casson and Stella Wisdom, focussed on the extensive correspondence contained within Andrew Salkey’s archive. One of the initial objectives was to digitally depict the movement of key Caribbean writers and artists, as it is evidenced within the correspondence, many of whom travelled between Britain and the Caribbean as well as the United States, Central and South America and Africa. Although Salkey corresponded with a diverse range of people, we therefore focused on the letters in his archive which were from Caribbean writers and academics and which illustrated  patterns of movement of the Caribbean diaspora. Much of the correspondence stems from 1960s and 1970s, a time when Andrew Salkey was particularly active both in the Caribbean Artists Movement and, as a writer and broadcaster, at the BBC.

Photograph of Andrew Salkey

Andrew Salkey was unusual not only for the panoply of writers, artists and politicians with whom he was connected, but that he sustained those relationships, carefully preserving the correspondence which resulted from those networks. My personal interest in this project stemmed from the fact that my PhD seeks to consider the ways that the Caribbean trickster character, Anancy, has historically been reinvented to say something about heritage and identity. Significant to that question was the way that the Caribbean Artists Movement, a dynamic group of artists and writers formed in London in the mid-1960s, and of which Andrew Salkey was a founder, appropriated Anancy, reasserting him and the folktales to convey something of a literary ‘voice’ for the Caribbean. For this reason, I was also interested in the writing networks which were evidenced within the correspondence, together with their impact.

What is Gephi?

Prior to starting the project, Eleanor, who had catalogued the Andrew Salkey archive and Digital Curator, Stella, had identified Gephi as a possible software application through which to visualise this data. Gephi has been used in a variety of projects, including several at Harvard University, examples of the breadth and diversity of those initiatives can be found here. Several of these projects have social networks or historical trading routes as their focus, with obvious parallels to this project. Others notably use correspondence as their main data.

Gathering the Data

Andrew Salkey was known as something of a chronicler. He was interested in letters and travel and was also a serious collector of stamps. As such, he had not only retained the majority of the letters he received but categorised them. Eleanor had originally identified potential correspondents who might be useful to the project, selecting writers who travelled widely, whose correspondence had been separately stored by Salkey, partly because of its volume, and who might be of wider interest to the public. These included the acclaimed Caribbean writers, Samuel Selvon, George Lamming, Jan Carew and Edward Kamau Brathwaite and publishers and political activists, Jessica and Eric Huntley.

Our initial intention was to limit the data to simple facts which could easily be gleaned from the letters. Gephi required that we did so on a spreadsheet ,which had to conform to a particular format. In the first stages of the project, the data was confined to the dates and location of the correspondence, information which could suggest the patterns of movement within the diaspora. However, the letters were so rich in detail, that we ultimately recorded other information. This included any additional travel taken by any of the correspondents,  and which was clearly evidenced in the letters, together with any passages from the correspondence which demonstrated either something of the nature and quality of the friendships or, alternatively, the mutual benefit of those relationships to the careers of so many of the writers.

Creating a visual network

Dr Duncan Hay was invited to collaborate with me on this project, as he has considerable expertise in this field, his research interests include web mapping for culture and heritage and data visualisation for literary criticism.  After the initial data was collated, we discussed with Duncan what visualisations could be created. It became apparent early on that creating a visualisation of the social networks, as opposed to the patterns of movement, might be relatively straightforward via Gephi, an application which was particularly useful for this type of graph. I had prepared a spreadsheet but, Gephi requires the data to be presented in a strictly consistent way which meant that any anomalies had to be eradicated and the data effectively ‘cleaned up’ using Open Refine. Gephi also requires that information is presented by way of a system of ‘nodes’; ‘edges’  and ‘attributes’ with corresponding spreadsheet columns. In our project, the ‘nodes’ referred to Andrew Salkey and each of the correspondents and other individuals of interest who were specifically referred to within the correspondence. The edges referred to the way that those people were connected which, in this case, was through correspondence. However, what added to the potential of the project was that these nodes and edges could be further described by reference to ‘attributes.’ The possibility of assigning a range of ‘attributes’ to each of the correspondents allowed a wealth of additional information to be provided about the networks. As a consequence, and in order to make any visualisation as informative as possible, I also added brief biographical information for each of the writers and artists to be inputted as ‘attributes’ together with some explanation of the nature of the networks that were being illustrated.

The visual illustration below shows not only the quantity of letters from the sample of correspondents to Andrew Salkey (the pink lines),  but also shows which other correspondents formed part of those networks and were referenced as friends or contacts within specific items of correspondence. For example, George Lamming references academic, Rex Nettleford and writer and activist, Claudia Jones, the founder of the Notting Hill Carnival, in his correspondence, connections which are depicted in grey. 

Gephi: Andrew Salkey correspondence network

The aim was, however, for the visualisation to also be interactive. This required considerable further manipulation of the format and tools. In this illustration you can see the information that is revealed about the prominent Barbadian writer, George Lamming which, in an interactive format, can be accessed via the ‘i’ symbols beside many of the nodes coloured in green.  

Whilst Gephi was a useful tool with which to illustrate the networks, it was less helpful as a way to demonstrate the patterns of movement, one of the primary objectives of the project. A challenge was, therefore, to create a map which could be both interactive and illustrative of the specific locations of the correspondents as well as their movement over time. With Duncan’s input and expertise, we opted for a hybrid approach, utilising two principal ways to illustrate the data: we used Gephi to create a visualisation of the ‘networks’ (above) and another software tool, Kepler.gl, to show the diasporic movement.

A static version of what ultimately will be a ‘moving’ map (illustrating correspondence with reference to person, date and location) is shown below. As well as demonstrating patterns of movement, it should also be possible to access information about specific letters as well as their shelf numbers through this map, hopefully making the archive more accessible.

Patterns of diasporic movement from Andrew Salkey's correspondence, illustrated in Kepler.gl

Whilst we are still exploring the potential of this project and how it might intersect with other areas of research and archives, it has already revealed something of the benefits of this type of data visualisation. For example, a project of this type could be used as an educational tool, providing something of a simple, but dynamic, introduction to the Caribbean Artists Movement. Being able to visualise the project has also allowed us to input information which confirms where specific letters of interest might be found within the archive. Ultimately, it is hoped that the project will offer ways to make a rich, yet arguably undervalued, archive more accessible to a wider audience with the potential to replicate something of an introductory model, or ‘pilot’ for further archives in the future. 

One half of the 2023 British Library Labs Symposium will be dedicated to digital storytelling. This has been a significant part of BL Labs work over the years; we have collaborated with experimental artists from David Normal’s creative reuse of British Library Flickr images for his giant lightbox collage Crossroads of Curiosity installation at the 2014 Burning Man festival, to working with first runner up in the BL Labs 2016 competition Michael Takeo Magruder on his 2019 exhibition Imaginary Cities.

Crossroads of Curiosity by David Normal

In the last few years, due to the COVID-19 pandemic disruption, digital stories and engagement have become mainstream across the Galleries, Libraries, Archives and Museums (GLAM) sector. New types of digital storytelling mixing social media, online exhibitions embedding narratives and digital objects, and interactive online events reaching entirely new audiences, delighted us all. However, we also discovered that there can be a saturation point with online engagement, and that many digital developments have some way to go to reach their full potential.

As we are hopefully entering healthier times, new opportunities to mix virtual and physical worlds are starting to open up. With this in mind, we felt that this is the right moment to explore a new age of digital storytelling at the 2023 BL Labs Symposium.

The idea is to explore what is changing in the world of technological possibilities and how they are continuing to develop. We have envisaged a journey that will take us from the big picture of the arising digital possibilities to more specific examples from the British Library’s work. In true BL Labs spirit we will also celebrate initiatives that creatively reuse the Library’s digital collections.

To help us look into the big trends, we are delighted to be joined by Zillah Watson, whose extraordinary breath of experience working with BBC, Meta, BFI and Royal Shakespeare Company amongst many others, will help us to get a deeper sense of the opportunities of virtual reality (VR). Zillah will look into what it means, not just to be dazzled with technological possibilities, but also to enter the magic of storytelling.

Talking of magic, we are lucky to welcome award winning Director, Anrick Bregman, and award winning Producer, Grace Baird. Anrick and Grace will take us deeper into the potential of using VR to uncover hidden stories. Anrick’s film A Convict Story is an interactive VR project built on British Library data that brings to life a story discovered by the linking of data from centuries ago, using data research powered by machine learning.

Even closer to home, our own Stella Wisdom and Ian Cooke, will talk about their current work on curating the British Library’s forthcoming Digital Storytelling exhibition (2 June – 15 October 2023), which will explore the ways technology provides opportunities to transform and enhance the way writers write and readers engage. Drawing on the Library’s collection of contemporary digital publications and emerging formats to highlight the work of innovative and experimental writers. It will feature interactive works that invite and respond to user input, reading experiences influenced by data feeds, and immersive story worlds created using multiple platforms and audience participation. This is an exciting development, as we can see how earlier British Library creative digital experiments, collaborations and research projects are building into an exhibition in its own right.

We hope you can join us for discussion at the BL Labs Symposium on Thursday 30 March 2023. For the full programme, and further information on all our speakers, please read our earlier blog post.

 You can book your place here. 

British Library digitised image from BL Flickr Collection - When Life is Young: a collection of verse for boys and girls by Mary Elizabeth Dodge

The BL Labs Symposium 2023 is taking place on Thursday 30^th March as an online webinar.

This year we will be exploring two themes – digital storytelling and innovative uses of data and AI. As always, we are aiming to hear from some guest speakers, as well as showcase the recent work using the British Library digital collections. The programme also include an update of BL Labs, including our new website and services.

We hope this will spark many further ideas and collaborations.

The full programme for the BL Labs Symposium is as follows:

14.00 – Welcome

Part 1: Digital Storytelling

14.05 – How to bring the magic of VR to audiences – Zillah Watson

14.15 – There Exists – A VR experience about hidden narratives – Anrick Bregman and Grace Baird

14.25 – Curating a Digital Storytelling exhibition – Stella Wisdom and Ian Cooke

14.35 – Panel Q&A

15.00 – In Memoriam Maurice Nicholson

15.05 – Break

15.15 – BL Labs Update – Silvija Aurylaite

Part 2 – Data and AI

15.35 - Ithaca: Restoring and attributing ancient texts using deep neural networks - Yannis Assael

15.45 – Living with Machines: Using digitised newspaper collections from the British Library in a data science project – Kalle Westerling

15.55 – Locating a National Collections through audience research. How cultural heritage organisations can engage the public using geospatial data – Gethin Rees

16.05 – Panel Q&A

16.30 – END

You can register for the BL Labs Symposium here.

We are currently planning an evening networking session at the British Library, starting at 18.30 for those who can join us in London. We are aware of the train strike planned for this day, so will confirm details nearer the time.

Below are a few details about our speakers:

Zillah Watson	Zillah Watson Zillah Watson led the BBC's award winning VR studio, winning a host of awards at festivals around the world, including an Emmy nomination. She led pioneering work taking VR to audiences in libraries around the UK. She now consults on the metaverse, and content and audience growth strategies for organisations including Meta, London & Partners, the BFI, International News Media Association, Arts Council England, and the Royal Shakespeare Company. She's had a long and varied media career, including 20 years at the BBC, where she was a TV and radio current affairs journalist, head of editorial standards for BBC Radio and led R&D research on future content. She is a lecturer at UCL and the new London Interdisciplinary School. She recently co-founded Phase Space, a tech for good start-up to use VR to support mental health for students and young people.
Anrick Bregman	Anrick Bregman Anrick is director and founder of an R&D studio that explores the future of spatial immersive storytelling by creating experiences built with virtual and augmented reality, computer vision and machine learning. His mission is to find new and interesting ways to merge technology with meaningful narratives which explore the human experience.
Grace Baird	Grace Baird Grace is a Producer with twelve years' experience working on audience-centred projects in the Arts, TV, and Immersive industries. She is experienced in immersive and digital production and distribution, particularly entertainment content. Grace has produced a variety of innovative projects including site-specific installations, an interactive feature-film, and social-VR experiences.
Stella Wisdom	Stella Wisdom Stella is Digital Curator for Contemporary British Collections at the British Library. Promoting creative and innovative reuse of digital collections, encouraging game making and digital storytelling in libraries, including collaborating widely with The National Videogame Museum, AdventureX, International Games Month in Libraries, the New Media Writing Prize and on research projects with University College London’s Institute of Education and Lancaster University. Stella research interests also explore the archiving of complex born digital material, examining methods for the collection, preservation and curation of narrative apps, digital comics and interactive fiction.
Ian Cooke	Ian Cooke Ian is Head of Contemporary British Publications at the British Library. He has worked in academic and research libraries with a focus on 20th and 21st-century history and social sciences. His interests are in the role of publishing in contemporary communications, and the everyday experience and expression of politics.
Silvija Aurylaite	Silvija Aurylaite Silvija Aurylaite is BL Labs Manager. She previously worked on the British Library Heritage Made Digital Programme. Her interests and domain of expertise include copyright, curation of digital collections of museums, archives and libraries, data science, design, creativity and social entrepreneurship. Previously, she was an initiator of a new publishing project Public Domain City that aimed to bring a new life into curious & obscure historical books on science, technology and nature. She also organized a retrospective dance film festival Dance in Film, Choreography, Body and Image, and media dance educational activities at the National Gallery of Art in Vilnius.
Yannis Assael	Yannis Assael Dr. Yannis Assael is a Staff Research Scientist at Google DeepMind working on Artificial Intelligence, and he is featured in Forbes' "30 Under 30" distinguished scientists of Europe. In 2013, he graduated from the Department of Applied Informatics, University of Macedonia, and with full scholarships, he did an MSc at the University of Oxford, finishing first in his year, and an MRes at Imperial College London. In 2016, he returned to Oxford for a DPhil degree with a Google DeepMind scholarship, and after a series of research breakthroughs and entrepreneurial activities, he started as a researcher at Google DeepMind. His contributions range from audio-visual speech recognition to multi-agent communication and AI for culture and the study of damaged ancient texts. Throughout this time, his research has attracted the media's attention several times, has been featured on the cover of the scientific journal Nature, and focuses on contributing to and expanding the greater good.
Kalle Westerling	Kalle Westerling Dr Kalle Westerling is a Digital Humanities Research Software Engineer with Living with Machines, a collaboration between the British Library, the Alan Turing Institute, and researchers from a range of UK universities. Kalle holds a Ph.D. in Theatre and Performance Studies from The Graduate Center, City University of New York (CUNY), where he visualised and analysed networks of itinerant nightlife performers around New York City in the 1930s. Prior to joining the British Library, Kalle managed the Scholars program at HASTAC and the Digital Humanities Research Institute at CUNY, both efforts across higher education institutions in the United States, aiming to build nation-wide infrastructures and communities for digital humanities skill-building.
Gethin Rees	Gethin Rees Gethin’s role at the British Library includes helping to manage the non-print legal deposit of digital maps and coordinating the Georeferencer crowd-sourcing project. He is interested in helping research projects to get the most out of geospatial data and tools and was principal investigator of the AHRC-funded Locating a National Collection project. Before taking up his current position in 2018 he worked on two collaborative history projects funded by the ERC and as a software developer. His PhD in archaeology from University of Cambridge made use of Geographical Information Systems for spatial analysis and data management.

Digital Curator Dr Mia Ridge shares news from a collaboration between the British Library and Zooniverse that means you can more easily create crowdsourcing projects with cultural heritage collections. There's a related blog post on Zooniverse, Fun with IIIF.

IIIF manifests - text files that tell software how to display images, sound or video files alongside metadata and other information about them - might not sound exciting, but by linking to them, you can view and annotate collections from around the world. The IIIF (International Image Interoperability Framework) standard makes images (or audio, video or 3D files) more re-usable - they can be displayed on another site alongside the original metadata and information provided by the source institution. If an institution updates a manifest - perhaps adding information from updated cataloguing or crowdsourcing - any sites that display that image automatically gets the updated metadata.

Playbill showing the title after other large text

We've posted before about how we used IIIF manifests as the basis for our In the Spotlight crowdsourced tasks on LibCrowds.com. Playbills are great candidates for crowdsourcing because they are hard to transcribe automatically, and the layout and information present varies a lot. Using IIIF meant that we could access images of playbills directly from the British Library servers without needing server space and extra processing to make local copies. You didn't need technical knowledge to copy a manifest address and add a new volume of playbills to In the Spotlight. This worked well for a couple of years, but over time we'd found it difficult to maintain bespoke software for LibCrowds.

When we started looking for alternatives, the Zooniverse platform was an obvious option. Zooniverse hosts dozens of historical or cultural heritage projects, and hundreds of citizen science projects. It has millions of volunteers, and a 'project builder' that means anyone can create a crowdsourcing project - for free! We'd already started using Zooniverse for other Library crowdsourcing projects such as Living with Machines, which showed us how powerful the platform can be for reaching potential volunteers. 

But that experience also showed us how complicated the process of getting images and metadata onto Zooniverse could be. Using Zooniverse for volumes of playbills for In the Spotlight would require some specialist knowledge. We'd need to download images from our servers, resize them, generate a 'manifest' list of images and metadata, then upload it all to Zooniverse; and repeat that for each of the dozens of volumes of digitised playbills.

Fast forward to summer 2021, when we had the opportunity to put a small amount of funding into some development work by Zooniverse. I'd already collaborated with Sam Blickhan at Zooniverse on the Collective Wisdom project, so it was easy to drop her a line and ask if they had any plans or interest in supporting IIIF. It turns out they had, but hadn't had the resources or an interested organisation necessary before.

We came up with a brief outline of what the work needed to do, taking the ability to recreate some of the functionality of In the Spotlight on Zooniverse as a goal. Therefore, 'the ability to add subject sets via IIIF manifest links' was key. ('Subject set' is Zooniverse-speak for 'set of images or other media' that are the basis of crowdsourcing tasks.) And of course we wanted the ability to set up some crowdsourcing tasks with those items… The Zooniverse developer, Jim O'Donnell, shared his work in progress on GitHub, and I was very easily able to set up a test project and ask people to help create sample data for further testing. 

If you have a Zooniverse project and a IIIF address to hand, you can try out the import for yourself: add 'subject-sets/iiif?env=production' to your project builder URL. e.g. if your project is number #xxx then the URL to access the IIIF manifest import would be https://www.zooniverse.org/lab/xxx/subject-sets/iiif?env=production

Paste a manifest URL into the box. The platform parses the file to present a list of metadata fields, which you can flag as hidden or visible in the subject viewer (public task interface). When you're happy, you can click a button to upload the manifest as a new subject set (like a folder of items), and your images are imported. (Don't worry if it says '0 subjects).

Screenshot of manifest import screen

You can try out our live task and help create real data for testing ingest processes at ​​https://frontend.preview.zooniverse.org/projects/bldigital/in-the-spotlight/classify

This is a very brief introduction, with more to come on managing data exports and IIIF annotations once you've set up, tested and launched a crowdsourced workflow (task). We'd love to hear from you - how might this be useful? What issues do you foresee? How might you want to expand or build on this functionality? Email [email protected] or tweet @mia_out @LibCrowds. You can also comment on GitHub https://github.com/zooniverse/Panoptes-Front-End/pull/6095 or https://github.com/zooniverse/iiif-annotations. 

Digital work in libraries is always collaborative, so I'd like to thank British Library colleagues in Finance, Procurement, Technology, Collection Metadata Services and various Collections departments; the Zooniverse volunteers who helped test our first task and of course the Zooniverse team, especially Sam, Jim and Chris for their work on this.

Daniel van Strien, Digital Curator, Living with Machines, writes:

The British Library’s digital scholarship department has made many digitised materials available to researchers. This includes a collection of digitised books created by the British Library in partnership with Microsoft. This is a collection of books that have been digitised and processed using Optical Character Recognition (OCR) software to make the text machine-readable. There is also a collection of books digitised in partnership with Google. 

Since being digitised, this collection of digitised books has been used for many different projects. This includes recent work to try and augment this dataset with genre metadata and a project using machine learning to tag images extracted from the books. The books have also served as training data for a historic language model.

This blog post will focus on two challenges of working with this dataset: size and documentation, and discuss how we’ve experimented with one potential approach to addressing these challenges. 

One of the challenges of working with this collection is its size. The OCR output is over 20GB. This poses some challenges for researchers and other interested users wanting to work with these collections. Projects like Living with Machines are one avenue in which the British Library seeks to develop new methods for working at scale. For an individual researcher, one of the possible barriers to working with a collection like this is the computational resources required to process it. 

Recently we have been experimenting with a Python library, datasets, to see if this can help make this collection easier to work with. The datasets library is part of the Hugging Face ecosystem. If you have been following developments in machine learning, you have probably heard of Hugging Face already. If not, Hugging Face is a delightfully named company focusing on developing open-source tools aimed at democratising machine learning. 

The datasets library is a tool aiming to make it easier for researchers to share and process large datasets for machine learning efficiently. Whilst this was the library’s original focus, there may also be other uses cases for which the datasets library may help make datasets held by the British Library more accessible. 

Some features of the datasets library:

• Tools for efficiently processing large datasets 
• Support for easily sharing datasets via a ‘dataset hub’ 
• Support for documenting datasets hosted on the hub (more on this later). 

As a result of these and other features, we have recently worked on adding the British Library books dataset library to the Hugging Face hub. Making the dataset available via the datasets library has now made the dataset more accessible in a few different ways.

Firstly, it is now possible to download the dataset in two lines of Python code: 

We can also use the Hugging Face library to process large datasets. For example, we only want to include data with a high OCR confidence score (this partially helps filter out text with many OCR errors): 

One of the particularly nice features here is that the library uses memory mapping to store the dataset under the hood. This means that you can process data that is larger than the RAM you have available on your machine. This can make the process of working with large datasets more accessible. We could also use this as a first step in processing data before getting back to more familiar tools like pandas. 

In a follow on blog post, we’ll dig into the technical details of datasets in some more detail. Whilst making the technical processing of datasets more accessible is one part of the puzzle, there are also non-technical challenges to making a dataset more usable. 

Documenting datasets 

One of the challenges of sharing large datasets is documenting the data effectively. Traditionally libraries have mainly focused on describing material at the ‘item level,’ i.e. documenting one dataset at a time. However, there is a difference between documenting one book and 100,000 books. There are no easy answers to this, but libraries could explore one possible avenue by using Datasheets. Timnit Gebru et al. proposed the idea of Datasheets in ‘Datasheets for Datasets’. A datasheet aims to provide a structured format for describing a dataset. This includes questions like how and why it was constructed, what the data consists of, and how it could potentially be used. Crucially, datasheets also encourage a discussion of the bias and limitations of a dataset. Whilst you can identify some of these limitations by working with the data, there is also a crucial amount of information known by curators of the data that might not be obvious to end-users of the data. Datasheets offer one possible way for libraries to begin more systematically commuting this information. 

The dataset hub adopts the practice of writing datasheets and encourages users of the hub to write a datasheet for their dataset. For the British library books, we have attempted to write one of these datacards. Whilst it is certainly not perfect, it hopefully begins to outline some of the challenges of this dataset and gives end-users a better sense of how they should approach a dataset. 

Every year the British Library host a range of PhD placement scheme projects. If you are interested in applying for one of these, the 2022 opportunities are advertised here. There are currently 15 projects available across Library departments, all starting from June 2022 onwards and ending before March 2023. If you would like to work with born digital collections, you may want to read last week’s Digital Scholarship blog post about two projects on enhanced curation, hybrid archives and emerging formats. However, if you are interested in Caribbean diasporic networks and want to experiment creating network analysis visualisations, then read on to find out more about the “Mapping Caribbean Diasporic Networks through correspondence (2022-ACQ-CDN)” project.

This is an exciting opportunity to be involved with the preliminary stages of a project to map the Caribbean Diasporic Network evident in the ‘Special Correspondence’ files of the Andrew Salkey Archive. This placement will be based in the Contemporary Literary and Creative Archives team at the British Library with support from Digital Scholarship colleagues. The successful candidate will be given access to a selection of correspondence files to create an item level dataset and explore the content of letters from the likes of Edward Kamau Brathwaite, C.L.R. James, and Samuel Selvon.

Photograph of Andrew Salkey, from the Andrew Salkey Archive, Deposit 10310. With kind permission of Jason Salkey.

The main outcome envisaged for this placement is to develop a dataset, using a sample of ten files, linking the data and mapping the correspondent’s names, location they were writing from, and dates of the correspondence in a spreadsheet. The placement student will also learn how to use the Gephi Open Graph Visualisation Platform to create a visual representation of this network, associating individuals with each other and mapping their movement across the world between the 1950s and 1990s.

Gephi Quick Start Tutorial

Gephi is open-source software  for visualising and analysing networks, they provide a step-by-step guide to getting started, with the first step to upload a spreadsheet detailing your ‘nodes’ and ‘edges’. To show an example of how Gephi can be used, We've included an example below, which was created by previous British Library research placement student Sarah FitzGerald from the University of Sussex, using data from the Endangered Archives Programme (EAP) to create a Gephi visualisation of all EAP applications received between 2004 and 2017.

Network visualisation of EAP Applications created by Sarah FitzGerald

In this visualisation the size of each country relates to the number of applications it features in, as country of archive, country of applicant, or both.  The colours show related groups. Each line shows the direction and frequency of application. The line always travels in a clockwise direction from country of applicant to country of archive, the thicker the line the more applications. Where the country of applicant and country of archive are the same the line becomes a loop. If you want to read more about the other visualisations that Sarah created during her project, please check out these two blog posts:

• Visualising the Endangered Archives Programme project data on Africa, Part 1. The project
• Visualising the Endangered Archives Programme project data on Africa, Part 2. Data visualisation tools

We hope this new PhD placement will offer the successful candidate the opportunity to develop their specialist knowledge through access to the extensive correspondence series in the Andrew Salkey archive, and to undertake practical research in a curatorial context by improving the accessibility of linked metadata for this collection material. This project is a vital building block in improving the Library’s engagement with this material and exploring the ways it can be accessed by a wider audience.

If you want to apply, details are available on the British Library website at https://www.bl.uk/research-collaboration/doctoral-research/british-library-phd-placement-scheme. Applications for all 2022/23 PhD Placements close on Friday 25 February 2022, 5pm GMT. The application form and guidelines are available online here. Please address any queries to [email protected]. 

This post is by Digital Curator Stella Wisdom (@miss_wisdom) and Eleanor Casson (@EleCasson), Curator in Contemporary Archives and Manuscripts.