Digital scholarship blog

Enabling innovative research with British Library digital collections

Introduction

Tracking exciting developments at the intersection of libraries, scholarship and technology. Read more

30 October 2020

Mind Your Paws and Claws

I’m not a summer creature, autumn is my favourite time of the year and I especially love Halloween. It is a perfect excuse for reading ghost stories, watching folk horror films and playing spooky videogames. If this sounds like fun to you too, then I recommend taking a look at the games created for Gothic Novel Jam.

Screen capture of the Gothic Novel Jam itch.io website with thumbnails of the games made as part of this jam

One of my favourite entries is The Lady's Book of Decency, A Practical Treatise on Manners, Feeding, and Etiquette, by Sean S. LeBlanc. I don't want to give away any spoilers, but I will say that it is a real howl! - also remember that this year there is a full moon on 31st October.

Game makers taking part in Gothic Novel Jam were encouraged to use images from the Ghosts & Ghoulish Scenes album in the British Library's Flickr site, which are all freely available for artistic and commercial reuse.

It is always a pleasure to see how creatives use the Flickr images to make new works, such as animations, like The Phantom Monk shown below, made by my talented colleague Carlos Rarugal from the UK Web Archive. He has animated a few spooky creatures for Halloween, which will shared be shared from the WildlifeWeb Archive and Digital Scholarship Twitter accounts. My colleague Cheryl Tipp has been Going batty for Halloween, making a Flappy Bat online game using Scratch, and the UK Web Archive have been celebrating their crawlers with this blog post.

Video created by Carlos Rarugal, using a British Library digitised image from page 377 of "The Lancashire Witches. A novel". Audio is Thunder, Eric & May Nobles, Wales, 1989 (W Thunder r3 C1) and Grey Wolf, Tom Cosburn, Canada, 1995 (W1CDR0000681 BD9)

If you enjoy making games and works of interactive fiction, then you may want to sign up to participate in AdventureX Game Jam, which is taking place online, during 14-28 November 2020. The jam's theme will be announced when AdvXJam opens on the 14th November. You are invited to interpret the theme in any way you choose, and AdventureX are very open-minded about what constitutes a narrative game. All genres, styles and game engines are welcome, as they are very keen to encourage participants to get involved regardless of background or experience level. 

Sadly the AdventureX Narrative Games Convention event is cancelled this year due to Covid-19, but we are hoping that the online AdventureX Game Jam will bring some cheer, creativity and community spirit during this year's International Games Week in Libraries in November. So keep your eyeballs peeled for blog posts about this jam next month.

This post is by Digital Curator Stella Wisdom (@miss_wisdom)

29 October 2020

Happy Eighth Birthday Wikidata!

Sadly 2020 is not being a year for in-person parties! However, I hope you'll raise a socially distanced glass safely at home to celebrate the eighth birthday of Wikidata, which first went live on 29th October 2012.

You can follow the festivities on social media with posts tagged #WikidataBirthday and read a message from the development team here. The WikiCite 2020 virtual conference kicked the celebrations off a few days early, with sessions about open citations and linked bibliographic data (videos online here) and depending what time you read this post, you may still be able to join a 24-hours long online meetup, where people can drop in to chat to others about Wikidata.

If you are reading this post and wondering what Wikidata is, then you might want to read this introduction. Essentially it "is a document-oriented database, focused on items, which represent topics, concepts, or objects. Each item is allocated a unique, persistent identifier, a positive integer prefixed with the upper-case letter Q, known as a "QID". This enables the basic information required to identify the topic that the item covers to be translated without favouring any language."[1]

Wikidata 8th birthday logo

Many libraries around the world have been actively adding data about their collections to Wikidata, and a number of groups to support and encourage this work have been established.

The IFLA Wikidata Working Group was formed in late 2019 to explore and advocate for the use of and contribution to Wikidata by library and information professionals. To support the integration of Wikidata and Wikibase with library systems, and alignment of the Wikidata ontology with library metadata formats such as BIBFRAME, RDA, and MARC.

This group was originally due to host a satellite event for the World Library and Information Congress 2020 in Dublin, which was sadly cancelled due to Covid-19. However this event was quickly converted into the Wikicite + Libraries series of six online discussions; about open citations, language revitalisation, knowledge equity, access to scholarly publications, linking and visualising bibliographic data. The recordings of which have all been made available online, via a Youtube playlist.

They have also set up a mailing list (wikidatawg@iflalists.org) and held an online launch party on the 8th October (slides). If you would like to attend their next meeting, it will be on the 24th November, the booking form is here.

illustration of a hand taking a book out of an image of a bookshelf on a computer monitor

Another online community for librarians working with Wikidata, is the LD4 Wikidata Affinity Group, which explores how libraries can contribute to and leverage Wikidata as a platform for publishing, linking, and enriching library linked data. They meet biweekly via Zoom. At each meeting, either the co-facilitators or an invited guest will give a presentation, or a demonstration, then there is a wider discussion of any issues, which members have encountered, and an opportunity for sharing helpful resources.

If you work in libraries and are curious about Wikidata, I highly recommend attending these groups. If you are looking for a introductory guide, then Practical Wikidata for Librarians is an excellent starting point. There is also Library Carpentry Wikidata currently in development, which is shaping up to be a very useful resource.

It can't be all work and no play though, so I'm celebrating Wikidata's birthday with a seasonal slice of Frankencolin the Caterpillar cake!

This post is by Digital Curator Stella Wisdom (@miss_wisdom)

1. https://en.wikipedia.org/wiki/Wikidata  ↩︎

23 October 2020

BL Labs Public Award Runner Up (Research) 2019 - Automated Labelling of People in Video Archives

Example people identified in TV news related programme clips
People 'automatically' identified in digital TV news related programme clips.

Guest blog post by Andrew Brown (PhD researcher),  Ernesto Coto (Research Software Engineer) and Andrew Zisserman (Professor) of the Visual Geometry Group, Department of Engineering Science, University of Oxford, and BL Labs Public Award Runner-up for Research, 2019. Posted on their behalf by Mahendra Mahey, Manager of BL Labs.

In this work, we automatically identify and label (tag) people in large video archives without the need for any manual annotation or supervision. The project was carried out with the British Library on a sample of 106 videos from their “Television and radio news” archive; a large collection of news programs from the last 10 years. This archive serves as an important and fascinating resource for researchers and the general public alike. However, the sheer scale of the data, coupled with a lack of relevant metadata, makes indexing, analysing and navigating this content an increasingly difficult task. Relying on human annotation is no longer feasible, and without an effective way to navigate these videos, this bank of knowledge is largely inaccessible.

As users, we are typically interested in human-centric queries such as:

  • “When did Jeremy Corbyn first appear in a Newsnight episode?” or
  • “Show me all of the times when Hugh Grant and Shirley Williams appeared together.

Currently this is nigh on impossible without trawling through hundreds of hours of content. 

We posed the following research question:

Is it possible to enable automatic person-search capabilities such as this in the archive, without the need for any manual supervision or labelling?

The answer is “yes”, and the method is described next.

Video Pre-Processing

The basic unit which enables person labelling in videos is the face-track; a group of consecutive face detections within a shot that correspond to the same identity. Face-tracks are extracted from all of the videos in the archive. The task of labelling the people in the videos is then to assign a label to each one of these extracted face-tracks. The video below gives an example of two face-tracks found in a scene.


Two face-tracks found in British Library digital news footage by Visual Geometry Group - University of Oxford.

Techniques at Our Disposal

The base technology used for this work is a state-of-the-art convolutional neural network (CNN), trained for facial recognition [1]. The CNN extracts feature-vectors (a list of numbers) from face images, which indicate the identity of the depicted person. To label a face-track, the distance between the feature-vector for the face-track, and the feature-vector for a face-image with known identity is computed. The face-track is labelled as depicting that identity if the distance is smaller than a certain threshold (i.e. they match). We also use a speaker recognition CNN [2] that works in the same way, except it labels speech segments from unknown identities using speech segments from known identities within the video.

Labelling the Face-Tracks

Our method for automatically labelling the people in the video archive is divided into three main stages:

(1) Our first labelling method uses what we term a “celebrity feature-vector bank”, which consists of names of people that are likely to appear in the videos, and their corresponding feature-vectors. The names are automatically sourced from IMDB cast lists for the programmes (the titles of the programmes are freely available in the meta-data). Face-images for each of the names are automatically downloaded from image-search engines. Incorrect face-images and people with no images of themselves on search engines are automatically removed at this stage. We compute the feature-vectors for each identity and add them to the bank alongside the names. The face-tracks from the video archives are then simply labelled by finding matches in the feature-vector bank.

Face-tracks from the video archives are labelled by finding matches in the feature-vector bank.
Face-tracks from the video archives are labelled by finding matches in the feature-vector bank. 

(2) Our second labelling method uses the idea that if a name is spoken, or found displayed in a scene, then that person is likely to be found within that scene. The task is then to automatically determine whether there is a correspondence or not. Text is automatically read from the news videos using Optical Character Recognition (OCR), and speech is automatically transcribed using Automatic Speech Recognition (ASR). Names are identified and they are searched for on image search engines. The top ranked images are downloaded and the feature-vectors are computed from the faces. If any are close enough to the feature-vectors from the face-tracks present in the scene, then that face-track is labelled with that name. The video below details this process for a written name.


Using text or spoken word and face recognition to identify a person in a news clip.

(3) For our third labelling method, we use speaker recognition to identify any non-labelled speaking people. We use the labels from the previous two stages to automatically acquire labelled speech segments from the corresponding labelled face-tracks. For each remaining non-labelled speaking person, we extract the speech feature-vector and compute the distance of it to the feature-vectors of the labelled speech segments. If one is close enough, then the non-labelled speech segment and corresponding face-track is assigned that name. This process manages to label speaking face-tracks with visually challenging faces, e.g. deep in shadow or at an extremely non-frontal pose.

Indexing and Searching Identities

The results of our work can be browsed via a web search engine of our own design. A search bar allows for users to specify the person or group of people that they would like to search for. People’s names are efficiently indexed so that the complete list of names can be filtered as the user types in the search bar. The search results are returned instantly with their associated metadata (programme name, data and time) and can be displayed in multiple ways. The video associated with each search result can be played, visualising the location and the name of all identified people in the video. See the video below for more details. This allows for the archive videos to be easily navigated using person-search, thus opening them up for use by the general public.


Archive videos easily navigated using person-search.

For examples of more of our Computer Vision research and open-source software, visit the Visual Geometry Group website.

This work was supported by the EPSRC Programme Grant Seebibyte EP/M013774/1

[1] Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In Proc. International Conference on Automatic Face & Gesture Recognition, 2018.

[2] Joon Son Chung, Arsha Nagrani and Andrew Zisserman. VoxCeleb2: Deep Speaker Recognition. INTERSPEECH, 2018

BL Labs Public Awards 2020

Inspired by this work that uses the British Library's digital archived news footage? Have you done something innovative using the British Library's digital collections and data? Why not consider entering your work for a BL Labs Public Award 2020 and win fame, glory and even a bit of money?

This year's public and staff awards 2020 are open for submission, the deadline for entry for both is Monday 30 November 2020.

Whilst we welcome projects on any use of our digital collections and data (especially in research, artistic, educational and community categories), we are particularly interested in entries in our public awards that have focused on anti-racist work, about the pandemic or that are using computational methods such as the use of Jupyter Notebooks.