THE BRITISH LIBRARY

Digital scholarship blog

50 posts categorized "Tools"

09 March 2017

Archaeologies of reading: guest post from Matthew Symonds, Centre for Editing Lives and Letters

Add comment

Digital Curator Mia Ridge: today we have a guest post by Matthew Symonds from the Centre for Editing Lives and Letters on the Archaeologies of reading project, based on a talk he did for our internal '21st century curatorship' seminar series. Over to Matt...

Some people get really itchy about the idea of making notes in books, and dare not defile the pristine printed page. Others leave their books a riot of exclamation marks, sarcastic incredulity and highlighter pen.

Historians – even historians disciplined by spending years in the BL’s Rare Books and Manuscripts rooms – would much prefer it if people did mark books, preferably in sentences like “I, Famous Historical Personage, have read this book and think the following having read it…”. It makes it that much easier to investigate how people engaged with the ideas and information they read.

Brilliantly for us historians, rare books collections are filled with this sort of material. The problem is it’s also difficult to catalogue and make discoverable (nota bene – it’s hard because no institutions could afford to employ and train sufficient cataloguers, not because librarians don’t realise this is an issue).

The Archaeology of Reading in Early Modern Europe (AOR) takes digital images of books owned and annotated by two renaissance readers, the professional reader Gabriel Harvey and the extraordinary polymath John Dee, transcribes and translates all the comments in the margin, and marks up all traces of a reader’s intervention with the printed book and puts the whole thing on the Internet in a way designed to be useful and accessible to researchers and the general public alike.

image from https://s3.amazonaws.com/feather-client-files-aviary-prod-us-east-1/2017-03-09/76bacc2c-befe-4e7c-b729-c49cf47adf0b.png
Screenshot, The Archaeology of Reading in Early Modern Europe

AOR is a digital humanities collaboration between the Centre for Editing Lives and Letters (CELL) at University College London, Johns Hopkins University and Princeton University, and generously funded by the Andrew W. Mellon Foundation.

More importantly, it’s also a collaboration between academic researchers, librarians and software engineers. An absolutely vital consideration of how we planned AOR, how we work on it, how we’re planning to expand it, was to identify a project that could offer a common ground to be shared between these three interests, where each party would have something to gain from it.

As one of the researchers, it was really important to me to avoid forming some sort of “client-provider” relationship with the librarians who curate and know so much about my sources, and the software engineers who build the digital infrastructure.

But we do use an academic problem as a means of giving our project a focus. In 1990, Antony Grafton and the late Lisa Jardine published their seminal article ‘“Studied for Action: how Gabriel Harvey read his Livy’ in the journal Past & Present.

One major insight of the article is that people read books in conjunction with one another, often for specific, pragmatic purposes. People didn’t pick up a book from their shelves, open at page one and proceed through to the finis, marking up as they went. They put other books next to them, books that explained, clarified, argued with one another.

By studying the marginalia, it’s possible to reconstruct these pathways across a library, recreating the strategies people used to manage the vast quantities of information they had at their disposal.

In order to produce this archaeology of reading, we’ve built a “digital bookwheel”, an attempt to recreate the revolving reading desk of the renaissance period which allowed the lucky owner to manoeuvre back and forth their books. From here, the user can call up the books we’ve digitised, read the transcriptions, and search for particular words and concepts.

image from http://s3.amazonaws.com/feather-files-aviary-prod-us-east-1/98739f1160a9458db215cec49fb033ee/2017-03-09/ac83353a40f24bea921e478b1450993e.png
Screenshot, The Archaeology of Reading in Early Modern Europe


It’s built out of open source materials, leveraging the International Image Interoperability Framework (IIIF) and the IIIF-compliant Mirador 2 Viewer. Interested parties can download the XML files of our transcriptions, as well as the data produced in the process.

The exciting thing for us is that all the work on creating this digital infrastructure – which is very much a work in progress -- has provided us with the raw materials for asking new research questions, questions that can only be asked by getting away from our computer and returning back to the rare books room.

14 November 2016

British Library Labs Symposium 2016 - Competition and Award runners up

Add comment

The 4th annual British Library Labs Symposium was held on 7th November 2016, and the event was a great success in celebrating and showcasing Digital Scholarship and highlighting the work of BL Labs and their collaborators. The exciting day included the announcement of the winners of the BL Labs Competition and BL Labs Awards, as well as of the runners up who are presented in this blog post. Posts written by all of the winners and runners up about their work are also scheduled for the next few weeks - watch this space!

BL Labs Competition finalist for 2016
Roly Keating, Chief Executive of the British Library announced that the runner up of the two finalists of the BL Labs Competition for 2016 was...

Black Abolitionist Performances and their Presence in Britain
By Hannah-Rose Murray (PhD student at the University of Nottingham)

Bl_labs_symposium_2016_027
Roly Keating, Chief Executive of the British Library, welcoming Hannah-Rose Murray on to the stage.

The project focuses on African American lives, experiences and lectures in Britain between 1830–1895. By assessing black abolitionist speeches in the British Library’s nineteenth-century newspaper collection and using the British Library’s Flickr Commons 1 million collection. to illustrate, the project has illuminated their performances and how their lectures reached nearly every corner of Britain. For the first time, the location of these meetings has been mapped and the number and scale of the lectures given by black abolitionists in Britain has been evaluated, allowing their hidden voices to be heard and building a more complete picture of Victorian London for us. Hannah-Rose has recently posted an update about her work and the project findings can also be found on her website: www.frederickdouglassinbritain.com.

RoseHannah-Rose Murray is a second year PhD student with the Department of American and Canadian Studies, University of Nottingham. Her AHRC/M3C-funded PhD focuses on the legacy of formerly enslaved African Americans on British society and the different ways they fought British racism. Hannah-Rose received a first class Masters degree in Public History from Royal Holloway University and has a BA History degree from University College London (UCL). In Nottingham, Hannah-Rose works closely with the Centre for Research in Race and Rights and is one of the postgraduate directors of the Rights and Justice Research Priority Area, which includes the largest number of scholars (700) in the world working on rights and justice.

BL Labs Awards runners up for 2016

Research Award runner up
Allan Sudlow, Head of Research Development at the British Library announced that the runner up of the Research Award was...

Nineteenth-century Newspaper Analytics
By Paul Fyfe (Associate Professor of English, North Carolina State University) and Qian Ge (PhD Candidate in Electrical and Computer Engineering, North Carolina State University)

News
Nineteenth-Century Newspaper Analytics

The project represents an innovative partnership between researchers in English literature, Electrical & Computer Engineering, and data analytics in pursuit of a seemingly simple research question: How can computer vision and image processing techniques be adapted for large-scale interpretation of historical illustrations? The project is developing methods in image analytics to study a corpus of illustrated nineteenth-century British newspapers from the British Library’s collection, including The Graphic, The Illustrated Police News, and the Penny Illustrated Paper. 

Paul_fyfe_qian_ge
Paul Fyfe and Qian Ge gave a recorded acceptance speech at the Symposium as they were unable to attend in person.

It aims to suggest ways of adapting image processing techniques to other historical media while also pursuing scholarship on nineteenth-century visual culture and the illustrated press. The project also exposes the formidable technical challenges presented by historical illustrations and suggests ways to refine computer vision algorithms and analytics workflows for such difficult data. The website includes sample workflows as well as speculations about how large-scale image analytics might yield insights into the cultural past, plus much more: http://ncna.dh.chass.ncsu.edu/imageanalytics 

Commercial Award runner up
Isabel Oswell, Head of Business Audiences at the British Library announced that the runner up of the Commercial Award was...

Poetic Places
By Sarah Cole (TIME/IMAGE organisation and Creative Entrepreneur-in-Residence at the British Library)

Bl_labs_symposium_2016_172Sarah Cole, presenting Poetic Places PoeticPoetic Places

Poetic Places is a free app for iOS and Android devices which was launched in March 2016. It brings poetic depictions of places into the everyday world, helping users to encounter poems in the locations described by the literature, accompanied by contextualising historical narratives and relevant audiovisual materials. These materials are primarily drawn from open archive collections, including the British Library Flickr collection. Utilising geolocation services and push notifications, Poetic Places can (whilst running in the background on the device) let users know when they stumble across a place depicted in verse and art, encouraging serendipitous discovery. Alternatively, they can browse the poems and places via map and list interfaces as a source of inspiration without travelling. Poetic Places aspires to give a renewed sense of place, to bring together writings and paintings and sounds to mean more than they do alone, and to bring literature into people’s everyday life in unexpected moments.

Artistic Award runner up
Jamie Andrews, Head of Culture and Learning at the British Library announced that the runner up of the Artistic Award was... 

Bl_labs_symposium_2016_190Kristina Hofmann and Claudia Rosa Lukas

Fashion Utopia
By Kris Hofmann (Animation Director) and Claudia Rosa Lukas (Curator)

 
Fashion Utopia

The project involved the creation of an 80 second animation and five vines which accompanied the Austrian contribution to the International Fashion Showcase London, organised annually by the British Council and the British Fashion Council. Fashion Utopia garnered creative inspiration from the treasure trove of images from the British Library Flickr Commons collection and more than 500 images were used to create a moving collage that was, in a second step, juxtaposed with stop-frame animated items of fashion and accessories.

Teaching / Learning Award runner up
Ria Bartlett, Lead Producer: Onsite Learning at the British Library announced that the runner up of the Teaching / Learning Award was...

The PhD Abstracts Collections in FLAX: Academic English with the Open Access Electronic Theses Online Service (EThOS) at the British Library

By Shaoqun Wu (FLAX Research & Development and Lecturer in Computer Science), Alannah Fitzgerald (FLAX Open Education Research and PhD Candidate), Ian H. Witten (FLAX Project Lead and Professor of Computer Science) and Chris Mansfield (English Language and Academic Writing Tutor)

Flax
The PhD Abstracts Collections in FLAX

The project presents an educational research study into the development and evaluation of domain-specific language corpora derived from PhD abstracts with the Electronic Theses Online Service (EThOS) at the British Library. The collections, which are openly available from this study, were built using the interactive FLAX (Flexible Language Acquisition flax.nzdl.org) open-source software for uptake in English for Specific Academic Purposes programmes (ESAP) at Queen Mary University of London. The project involved the harvesting of metadata, including the abstracts of 400,000 doctoral theses from UK universities, from the EThOS Toolkit at the British Library. These digital PhD abstract text collections were then automatically analysed, enriched, and transformed into a resource that second-language and novice research writers can browse and query in order to extend their ability to understand the language used in specific domains, and to help them develop their abstract writing. It is anticipated that the practical contribution of the FLAX tools and the EThOS PhD Abstract collections will benefit second-language and novice research writers in understanding the language used to achieve the persuasive and promotional aspects of the written research abstract genre. It is also anticipated that users of the collections will be able to develop their arguments more fluently and precisely through the practice of research abstract writing to project a persuasive voice as is used in specific research disciplines.

Bl_labs_symposium_2016_209
Alannah Fitzgerald and Chris Mansfield receiving the Runner Up Teaching and Learning Award on behalf of the FLAX team.

British Library Labs Staff Award runner up
Phil Spence, Chief Operating Officer at the British Library announced that the runner up of the British Library Labs Staff Award as...

SHINE 2.0 - A Historical Search Engine

Led by Andy Jackson (Web Archiving Technical Lead at the British Library) and Gil Hoggarth (Senior Web Archiving Engineer at the British Library)

Shine
SHINE

SHINE is a state-of-the-art demonstrator for the potential of Web Archives to transform research. The current implementation of SHINE exposes metadata from the Internet Archive's UK domain web archives for the years 1996- 2013. This data was licensed for use by the British Library by agreement with JISC. SHINE represents a high level of innovation in access and analysis of web archives, allowing sophisticated searching of a very large and loosely-structured dataset and showing many of the characteristics of "Big Social Data". Users can fine-tune results to look for file-types, results from specific domains, languages used and geo-location data (post-code look-up). The interface was developed by Web Archive technical development alongside the AHRC-funded Big UK Domain Data for the Arts and Humanities project. An important concept in its design and development was that it would be researcher-led and SHINE was developed iteratively with research case studies relating to use of UK web archives.

Bl_labs_symposium_2016_298
Andy Jackson, Receiving the Runner up Staff Award on behalf of the SHINE team

The lead institution for SHINE was the University of London, with Professor Jane Winters as principle investigator, and former British Library staff members Peter Webster and Helen Hockx were also instrumental in developing the project and maintaining researcher engagement through the project. 

10 November 2016

British Library Labs Symposium 2016 - Competition and Award Winners

Add comment

The 4th annual British Library Labs Symposium took place on 7th November 2016 and was a resounding success! 

More than 220 people attended and the event was a fantastic experience, showcasing and celebrating the Digital Scholarship field and highlighting the work of BL Labs and their collaborators. The Symposium included a number of exciting announcements about the winners of the BL Labs Competition and BL Labs Awards, who are presented in this blog post. Separate posts will be published about the runners up of the Competition and Awards and posts written by all of the winners and runners up about their work are also scheduled for the next few weeks - watch this space!

BL Labs Competition winner for 2016

Roly Keating, Chief Executive of the British Library announced that the overall winner of the BL Labs Competition for 2016 was...

SherlockNet: Using Convolutional Neural Networks to automatically tag and caption the British Library Flickr collection
By Karen Wang and Luda Zhao, Masters students at Stanford University, and Brian Do, Harvard Medicine MD student

Machine learning can extract information and insights from data on a massive scale. The project developed and optimised Convolutional Neural Networks (CNN), inspired by biological neural networks in the brain, in order to tag and caption the British Library’s Flickr Commons 1 million collection. In the first step of the project, images were classified with general categorical tags (e.g. “people”, “maps”). This served as the basis for the development of new ways to facilitate rapid online tagging with user-defined sets of tags. In the second stage, automatically generate descriptive natural-language captions were provided for images (e.g. “A man in a meadow on a horse”). This computationally guided approach has produced automatic pattern recognition which provides a more intuitive way for researchers to discover and use images. The tags and captions will be made accessible and searchable by the public through the web-based interface and text annotations will be used to globally analyse trends in the Flickr collection over time.

Bl_labs_symposium_2016_131
SherlockNet team presenting at the Symposium

Karen Wang is currently a senior studying Computer Science at Stanford University, California. She also has an Art Practice minor. Karen is very interested in the intersection of computer science and humanities research, so this project is near and dear to her heart! She will be continuing her studies next year at Stanford in CS, Artificial Intelligence track.

Luda Zhao is currently a Masters student studying Computer Science at Stanford University, living in Palo Alto, California. He is interested in using machine learning and data mining to tackle tough problems in a variety of real-life contexts, and he's excited to work with the British Library to make art more discoverable for people everywhere.

Brian Do grew up in sunny California and is a first-year MD/PhD student at Harvard Medical School. Previously he studied Computer Science and biology at Stanford. Brian loves using data visualisation and cutting edge tools to reveal unexpected things about sports, finance and even his own text message history.

SherlockNet recently posted an update of their work and you can try out their SherlockNet interface and tell us what you think.

BL Labs Awards winners for 2016

Research Award winner

Allan Sudlow, Head of Research Development at the British Library announced that the winner of the Research Award was...

Scissors and Paste

By Melodee Beals, Lecturer in Digital History at Loughborough University and historian of migration and media

Bl_labs_symposium_2016_162
Melodee Beals presenting Scissors & Paste

Scissors and Paste utilises the 1800-1900 digitised British Library Newspapers, collection to explore the possibilities of mining large-scale newspaper databases for reprinted and repurposed news content. The project has involved the development of a suite of tools and methodologies, created using both out-of-the-box and custom-made project-specific software, to efficiently identify reprint families of journalistic texts and then suggest both directionality and branching within these subsets. From these case-studies, detailed analyses of additions, omissions and wholesale changes offer insights into the mechanics of reprinting that left behind few if any other traces in the historical record.

Melodee Beals joined the Department of Politics, History and International Relations at Loughborough University in September 2015. Previously, Melodee has worked as a pedagogical researcher for the History Subject Centre, a teaching fellow for the School of Comparative American Studies at the University of Warwick and a Principal Lecturer for Sheffield Hallam University, where she acted as Subject Group Leader for History. Melodee completed her PhD at the University of Glasgow.

Commercial Award winner

Isabel Oswell, Head of Business Audiences at the British Library announced that the winner of the Commercial Award was...

Curating Digital Collections to Go Mobile

By Mitchel Davis, publishing and media entrepreneur

Bl_labs_symposium_2016_178
Mitchell Davis presenting Curating Digital Collections to Go Mobile

As a direct result of its collaborative work with the British Library, BiblioLabs has developed BiblioBoard, an award-winning e-Content delivery platform, and online curatorial and multimedia publishing tools to support it. These tools make it simple for subject area experts to create visually stunning multi-media exhibits for the web and mobile devices without any technical expertise. The curatorial output is almost instantly available via a fully responsive web site as well as through native apps for mobile devices. This unified digital library interface incorporates viewers for PDF, ePub, images, documents, video and audio files allowing users to immerse themselves in the content without having to link out to other sites to view disparate media formats.

Mitchell Davis founded BookSurge in 2000, the world’s first integrated global print-on-demand and publishing services company (sold to Amazon.com in 2005 and re-branded as CreateSpace). Since 2008, he has been founder and chief business officer of BiblioLabs- the creators of BiblioBoard. Mitchell is also an indie producer and publisher who has created several award winning indie books and documentary films over the past decade through Organic Process Productions, a small philanthropic media company he founded with his wife Farrah Hoffmire in 2005.

Artistic Award winner

Jamie Andrews, Head of Culture and Learning at the British Library announced that the winner of the Artistic Award was... 

Here there, Young Sailor

Written and directed by writer and filmmaker Ling Low and visual art by Lyn Ong

Hey There, Young Sailor combines live action with animation, hand-drawn artwork and found archive images to tell a love story set at sea. Inspired by the works of early cinema pioneer Georges Méliès, the video draws on late 19th century and early 20th century images from the British Library's Flickr collection for its collages and tableaux. The video was commissioned by Malaysian indie folk band The Impatient Sisters and independently produced by a Malaysian and Indonesian team.

Bl_labs_symposium_2016_192
Ling Low receives her Award from Jamie Andrews

Ling Low is based between Malaysia and the UK and she has written and directed various short films and music videos. In her fiction and films, Ling is drawn to the complexities of human relationships and missed connections. By day, she works as a journalist and media consultant. Ling has edited a non-fiction anthology of human interest journalism, entitled Stories From The City: Rediscovering Kuala Lumpur, published in 2016. Her journalism has also been published widely, including in the Guardian, the Telegraph and Esquire Malaysia.

Teaching / Learning Award winner

Ria Bartlett, Lead Producer: Onsite Learning at the British Library announced that the winner of the Teaching / Learning Award was...

Library Carpentry

Founded by James Baker, Lecturer at the Sussex Humanities Lab, who represented the global Library Carpentry Team (see below) at the Symposium

Bl_labs_symposium_2016_212
James Baker presenting Library Carpentry

Library Carpentry is software skills training aimed at the needs and requirements of library professionals. It takes the form of a series of modules that are available online for self-directed study or for adaption and reuse by library professionals in face-to-face workshops. Library Carpentry is in the commons and for the commons: it is not tied to any institution or person. For more information on Library Carpentry see http://librarycarpentry.github.io/

James Baker is a Lecturer in Digital History and Archives at the School of History, Art History and Philosophy and at the Sussex Humanities Lab. He is a historian of the long eighteenth century and contemporary Britain. James is a Software Sustainability Institute Fellow and holds degrees from the University of Southampton and latterly the University of Kent. Prior to joining Sussex, James has held positions of Digital Curator at the British Library and Postdoctoral Fellow with the Paul Mellon Centre for Studies of British Art. James is a convenor of the Institute of Historical Research Digital History seminar and a member of the History Lab Plus Advisory Board.

 The Library Carpentry Team is regularly accepting new members and currently also includes: 

Carpentry
The Library Carpentry Team

British Library Labs Staff Award winner

Phil Spence, Chief Operating Officer at the British Library announced that the winner of the British Library Labs Staff Award was...

Libcrowds

Led by Alex Mendes, Software Developer at the British Library

LibCrowds is a crowdsourcing platform built by Alexander Mendes. It aims to create searchable catalogue records for some of the hundreds of thousands of items that can currently only be found in printed and card catalogues. By participating in the crowdsourcing projects, users will help researchers everywhere to access the British Library’s collections more easily in the future.

Bl_labs_symposium_2016_247
Nora McGregor presenting LibCrowds on behalf of Alex Mendes

The first project series, Convert-a-Card, experimented with a new method for transforming printed card catalogues into electronic records for inclusion in our online catalogue Explore, by asking volunteers to link scanned images of the cards with records retrieved from the WorldCat database. Additional projects have recently been launched that invite volunteers to transcribe cards that may require more specific language skills, such as the South Asian minor languages. Records matched, located, transcribed or translated as part of the crowdsourcing projects were uploaded to the British Library's Explore catalogue for anyone to search online. By participating users can have a direct impact on the availability of research material to anyone interested in the diverse collections available at the British Library.

Alex Mendes has worked at the British Library for several years and recently completed a Bachelor’s degree in Computer Science with the Open University. Alex enjoys the consistent challenges encountered when attempting to find innovative new solutions to unusual problems in software development.

AlexMendes
Alex Mendes

If you would like to find out more about BL Labs, our Competition or Awards please contact us at labs@bl.uk   

03 November 2016

SherlockNet update - 10s of millions more tags and thousands of captions added to the BL Flickr Images!

Add comment

SherlockNet are Brian Do, Karen Wang and Luda Zhao, finalists for the Labs Competition 2016.

We have some exciting updates regarding SherlockNet, our ongoing efforts to using machine learning techniques to radically improve the discoverability of the British Library Flickr Commons image dataset.

Tagging

Over the past two months we’ve been working on expanding and refining the set of tags assigned to each image. Initially, we set out simply to assign the images to one of 11 categories, which worked surprisingly well with less than a 20% error rate. But we realised that people usually search from a much larger set of words, and we spent a lot of time thinking about how we would assign more descriptive tags to each image.

Eventually, we settled on a Google Images style approach, where we parse the text surrounding each image and use it to get a relevant set of tags. Luckily, the British Library digitised the text around all 1 million images back in 2007-8 using Optical Character Recognition (OCR), so we were able to grab this data. We explored computational tools such as Term Frequency – Inverse Document Frequency (Tf-idf) and Latent Dirichlet allocation (LDA), which try to assign the most “informative” words to each image, but found that images aren’t always associated with the words on the page.

To solve this problem, we decided to use a 'voting' system where we find the 20 images most similar to our image of interest, and have all images vote on the nouns that appear most commonly in their surrounding text. The most commonly appearing words will be the tags we assign to the image. Despite some computational hurdles selecting the 20 most similar images from a set of 1 million, we were able to achieve this goal. Along the way, we encountered several interesting problems.

Similar images
For all images, similar images are displayed
  1. Spelling was a particularly difficult issue. The OCR algorithms that were state of the art back in 2007-2008 are now obsolete, so a sizable portion of our digitised text was misspelled / transcribed incorrectly. We used a pretty complicated decision tree to fix misspelled words. In a nutshell, it amounted to finding the word that a) is most common across British English literature and b) has the smallest edit distance relative to our misspelled word. Edit distance is the fewest number of edits (additions, deletions, substitutions) needed to transform one word into another.
  2. Words come in various forms (e.g. ‘interest’, ‘interested’, ‘interestingly’) and these forms have to be resolved into one “stem” (in this case, ‘interest’). Luckily, natural language toolkits have stemmers that do this for us. It doesn’t work all the time (e.g. ‘United States’ becomes ‘United St’ because ‘ates’ is a common suffix) but we can use various modes of spell-check trickery to fix these induced misspellings.
  3. About 5% of our books are in French, German, or Spanish. In this first iteration of the project we wanted to stick to English tags, so how do we detect if a word is English or not? We found that checking each misspelled (in English) word against all 3 foreign dictionaries would be extremely computationally intensive, so we decided to throw out all misspelled words for which the edit distance to the closest English word was greater than three. In other words, foreign words are very different from real English words, unlike misspelled words which are much closer.
  4. Several words appear very frequently in all 11 categories of images. These words were ‘great’, ‘time’, ‘large’, ‘part’, ‘good’, ‘small’, ‘long’, and ‘present’. We removed these words as they would be uninformative tags.

In the end, we ended up with between 10 and 20 tags for each image. We estimate that between 30% and 50% of the tags convey some information about the image, and the other ones are circumstantial. Even at this stage, it has been immensely helpful in some of the searches we’ve done already (check out “bird”, “dog”, “mine”, “circle”, and “arch” as examples). We are actively looking for suggestions to improve our tagging accuracy. Nevertheless, we’re extremely excited that images now have useful annotations attached to them!

SherlockNet Interface

Sherlocknet-interface
SherlockNet Interface

For the past few weeks we’ve been working on the incorporation of ~20 million tags and related images and uploading them onto our website. Luckily, Amazon Web Services provides comprehensive computing resources to take care of storing and transferring our data into databases to be queried by the front-end.

In order to make searching easier we’ve also added functionality to automatically include synonyms in your search. For example, you can type in “lady”, click on Synonym Search, and it adds “gentlewoman”, “ma'am”, “madam”, “noblewoman”, and “peeress” to your search as well. This is particularly useful in a tag-based indexing approach as we are using.

As our data gets uploaded over the coming days, you should begin to see our generated tags and related images show up on the Flickr website. You can click on each image to view it in more detail, or on each tag to re-query the website for that particular tag. This way users can easily browse relevant images or tags to find what they are interested in.

Each image is currently captioned with a default description containing information on which source the image came from. As Luda finishes up his captioning, we will begin uploading his captions as well.

We will also be working on adding more advanced search capabilities via wrapper calls to the Flickr API. Proposed functionality will include logical AND and NOT operators, as well as better filtering by machine tags.

Captioning

As mentioned in our previous post, we have been experimenting with techniques to automatically caption images with relevant natural language captions. Since an Artificial Intelligence (AI) is responsible for recognising, understanding, and learning proper language models for captions, we expected the task to be far harder than that of tagging, and although the final results we obtained may not be ready for a production-level archival purposes, we hope our work can help spark further research in this field.

Our last post left off with our usage of a pre-trained Convolutional Neural Networks - Recurrent Neural Networks (CNN-RNN) architecture to caption images. We showed that we were able to produce some interesting captions, albeit at low accuracy. The problem we pinpointed was in the training set of the model, which was derived from the Microsoft COCO dataset, consisting of photographs of modern day scenes, which differs significantly from the BL Flickr dataset.

Through collaboration with BL Labs, we were able to locate a dataset that was potentially better for our purposes: the British Museum prints and drawing online collection, consisting of over 200,000 print drawing, and illustrations, along with handwritten captions describing the image, which the British Museum has generously given us permission to use in this context. However, since the dataset is directly obtained from the public SPARQL endpoints, we needed to run some pre-processing to make it usable. For the images, we cropped them to standard 225 x 225 size and converted them to grayscale. For caption, pre-processing ranged from simple exclusion of dates and author information, to more sophisticated “normalization” procedures, aimed to lessen the size of the total vocabulary of the captions. For words that are exceeding rare (<8 occurrences), we replaced them with <UNK> (unknown) symbols denoting their rarity. We used the same neuraltalk architecture, using the features from a Very Deep Convolutional Networks for Large-Scale Visual Recognition (VGGNet) as intermediate input into the language model. As it turns out, even with aggressive filtering of words, the distribution of vocabulary in this dataset was still too diverse for the model. Despite our best efforts to tune hyperparameters, the model we trained was consistently over-sensitive to key phrases in the dataset, which results in the model converging on local minimums where the captions would stay the same and not show any variation. This seems to be a hard barrier to learning from this dataset. We will be publishing our code in the future, and we welcome anyone with any insight to continue on this research.

Captions
Although there were occasion images with delightfully detailed captions (left), our models couldn’t quite capture useful information for the vast majority of the images(right). More work is definitely needed in this area!

The British Museum dataset (Prints and Drawings from the 19th Century) however, does contain valuable contextual data, and due to our difficulty in using it to directly caption the dataset, we decided to use it in other ways. By parsing the caption and performing Part-Of-Speech (POS) tagging, we were able to extract nouns and proper nouns from each caption. We then compiled common nouns from all the images and filtered out the most common(>=500 images) as tags, resulting in over 1100 different tags. This essentially converts the British Museum dataset into a rich dataset of diverse tags, which we would be able to apply to our earlier work with tag classification. We trained a few models with some “fun” tags, such as “Napoleon”, “parrots” and “angels”, and we were able to get decent testing accuracies of over 75% on binary labels. We will be uploading a subset of these tags under the “sherlocknet:tags” prefix to the Flickr image set, as well as the previous COCO captions for a small subset of images(~100K).

You can access our interface here: bit.ly/sherlocknet or look for 'sherlocknet:tag=' and 'sherlocknet:category=' tags on the British Library Flickr Commons site, here is an example, and see the image below:

Sherlocknet tags
Example Tags on a Flickr Image generated by SherlockNet

Please check it out and let us know if you have any feedback!

We are really excited that we will be there in London in a few days time to present our findings, why don't you come and join us at the British Library Labs Symposium, between 0930 - 1730 on Monday 7th of November, 2016?

26 September 2016

British Library Labs Staff Awards 2016: Looking for entries now!

Add comment

Four-light-bulbsNominate a British Library staff member or team who has been instrumental in doing something exciting, innovative and cool with the British Library’s digital collections or data.

The 2016 British Library Labs Staff Award will recognise a team or current member of staff at the British Library that has played a key role in innovative work with the Library’s digital collections or data. This is the first time that the British Library is bestowing this Award and it will highlight some of the work the Library does and the people who do it. 

Perhaps the project you know about demonstrated the development of new knowledge or was an activity that delivered commercial value to the library. Did the person create an artistic work that inspired, stimulated, amazed and provoked? Do you know of a project developed by the Library where quality learning experiences for learners were developed using the Library’s digital content? 

You may nominate a current member of British Library staff, a team, or yourself(s) if you work at the Library, for the Award using this form.

The deadline for submission is 12:00 (BST), Monday 24th October 2016.

The winner(s) will be announced on Monday 7th November 2016 at the British Library Labs Annual Symposium where they will be asked to talk about their work.

The Staff Award complements the British Library Labs Awards, introduced in 2015, which recognises outstanding work that has been done in the broader community. Last year’s winners drew attention to artistic, research, and entrepreneurial activities that used our digital collections.

British Library Labs is a project within the Digital Scholarship department at the British Library that supports and inspires the use of the Library's digital collections and data in exciting and innovative ways. It is funded by the Andrew W. Mellon Foundation.

If you have any questions, please contact us at labs@bl.uk.

@bl_labs #bldigital @bl_digischol

09 September 2016

BL Labs Symposium (2016): book your place for Mon 7th Nov 2016

Add comment

Bl_labs_logo

Posted by Hana Lewis, BL Labs Project Officer.

The BL Labs team are pleased to announce that the fourth annual British Library Labs Symposium will be held on Monday 7th November, from 9:30 - 17:30 in the British Library Conference Centre, St Pancras. The event is free, although you must book a ticket in advance. Don't miss out!

The Symposium showcases innovative projects which use the British Library’s digital content, and provides a platform for development, networking and debate in the Digital Scholarship field.

Melissa
Professor Melissa Terras will be giving the keynote at this year's Symposium

This year, Dr Adam Farquhar, Head of Digital Scholarship at the British Library, will launch the Symposium. This will be followed by a keynote from Professor Melissa Terras, Director of UCL Centre for Digital Humanities. Roly Keating, Chief Executive of the British Library, will present awards to the BL Labs Competition (2016) finalists, who will also give presentations on their winning projects. 

After lunch, Stella Wisdom, Digital Curator at the British Library, will announce the winners of the Shakespeare Off the Map 2016 competition, which challenged budding designers to use British Library digital collections as inspiration in the creation of exciting interactive digital media. Following, the winners will be announced of the BL Labs Awards (2016)which recognises projects that have used the British Library’s digital content in exciting and innovative ways. Presentations will be given by the winners in each of the Awards’ categories: Research, Commercial, Artistic and Teaching / Learning. A British Library Staff Award will also be presented this year, recognising an outstanding individual or team who have played a key role in innovative work with the British Library's digital collections.  

The Symposium's endnote will be followed by a networking reception which will conclude the event, at which delegates and staff can mingle and network over a drink.  

So book your place for the Symposium today!

For any further information please contact labs@bl.uk

 

06 September 2016

BL Labs Awards (2016): deadline extended to Monday 12 September!

Add comment

The BL Labs Awards formally recognises outstanding and innovative work that has been created using the British Library’s digital collections and data.

The closing date for entering the BL Labs Awards (2016) has just been extended and people can take advantage of this opportunity to submit until 0900 BST on Monday 12th September 2016. 

This year, the BL Labs Awards is commending fantastic digital-based projects in four key areas:

  • Research - A project or activity which shows the development of new knowledge, research methods, or tools.
  • Commercial - An activity that delivers or develops commercial value in the context of new products, tools, or services that build on, incorporate, or enhance the Library's digital content.
  • Artistic - An artistic or creative endeavour which inspires, stimulates, amazes and provokes.
  • Teaching / Learning - Quality learning experiences created for learners of any age and ability that use the Library's digital content.

After the submission deadline, the entries will be shortlisted. Selected shortlisted entrants will be notified via email by midnight BST on Wednesday 21st September 2016. A prize of £500 will be awarded to the winner and £100 to the runner up of each Awards category at the Labs Symposium on 7th November 2016 at the British Library, St Pancras, courtesy of the Andrew W. Mellon Foundation.

The talent of the BL Labs Awards winners and runners of 2015 has led to the production a remarkable and varied collection of innovative projects. Last year, the Awards commended work in three main categories – Research, Creative/Artistic and Entrepreneurship:

All

Image:
(Top-left) Spatial Humanities research group at the University Lancaster plotting mentions of disease in newspapers on a map in Victorian times;
(Top-right) A computer generated work of art, part of 'The Order of Things' by Mario Klingemann;
(Bottom-left) A bow tie made by Dina Malkova inspired by a digitised original manuscript of Alice in Wonderland;
(Bottom-right) Work on Geo-referencing maps discovered from a collection of digitised books at the British Library that James Heald is still involved in.

For any further information about BL Labs or our Awards, please contact us at labs@bl.uk.

22 August 2016

SherlockNet: tagging and captioning the British Library’s Flickr images

Add comment

Finalists of the BL Labs Competition 2016, Karen Wang, Luda Zhao and Brian Do, inform us on the progress of their SherlockNet project:

Sherlock

This is an update on SherlockNet, our project to use machine learning and other computational techniques to dramatically increase the discoverability of the British Library’s Flickr images dataset. Below is some of our progress on tagging, captioning, and the web interface.

Tags

When we started this project, our goal was to classify every single image in the British Library's Flickr collection into one of 12 tags -- animals, people, decorations, miniatures, landscapes, nature, architecture, objects, diagrams, text, seals, and maps. Over the course of our work, we realised the following:

  1. We were achieving incredible accuracy (>80%) in our classification using our computational methods.
  2. If our algorithm assigned two tags to an image with approximately equal probability, there was a high chance the image had elements associated with both tags.
  3. However, these tags were in no way enough to expose all the information in the images.
  4. Luckily, each image is associated with text on the corresponding page.

We thus wondered whether we could use the surrounding text of each image to help expand the “universe” of possible tags. While the text around an image may or may not be directly related to the image, this strategy isn’t without precedent: Google Images uses text as its main method of annotating images! So we decided to dig in and see how this would go.

As a first step, we took all digitised text from the three pages surrounding each image (the page before, the page of, and the page after) and extracted all noun phrases. We figured that although important information may be captured in verbs and adjectives, the main things people will be searching for are nouns. Besides, at this point this is a proof of principle that we can easily extend later to a larger set of words. We then constructed a composite set of all words from all images, and only kept words present in between 5% and 80% of documents. This was to get rid of words that were too rare (often misspellings) or too common (words like ‘the’, ‘a’, ‘me’ -- called “stop words” in the natural language processing field).

With this data we were able to use a tool called Latent Dirichlet Allocation (LDA) to find “clusters” of images in an automatic way. We chose the original 12 tags after manually going through 1,000 images on our own and deciding which categories made the most sense based on what we saw; but what if there are categories we overlooked or were unable to discern by hand? LDA solves this by trying to find a minimal set of tags where each document is represented by a set of tags, and each tag is represented by a set of words. Obviously the algorithm can’t provide meaning to each tag, so we provide meaning to the tag by looking at the words that are present or absent in each tag. We ran LDA on a sample of 10,000 images and found tags clusters for men, women, nature, and animals. Not coincidentally, these are similar to our original tags and represent a majority of our images.

This doesn’t solve our quest for expanding our tag universe though. One strategy we thought about was to just use the set of words from each page as the tags for each image. We quickly found, however, that most of the words around each image are irrelevant to the image, and in fact sometimes there was no relation at all. To solve this problem, we used a voting system [1]. From our computational algorithm, we found the 20 images most similar to the image in question. We then looked for the words that were found most often in the pages around these 20 images. We then use these words to describe the image in question. This actually works quite well in practice! We’re now trying to combine this strategy (finding generalised tags for images) with the simpler strategy (unique words that describe images) to come up with tags that describe images at different “levels”.

Image Captioning

We started with a very ambitious goal: given only the image input, can we give a machine -generated, natural-language description of the image with a reasonably high degree of accuracy and usefulness? Given the difficulty of the task and of our timeframe, we didn’t expect to get perfect results, but we’ve hoped to come up with a first prototype to demonstrate some of the recent advances and techniques that we hope will be promising for research and application in the future.

We planned to look at two approaches to this problem:

  • Similarity-based captioning. Images that are very similar to each other using a distance metric often share common objects, traits, and attributes that shows up in the distribution of words in their captions. By pooling words together from a bag of captions of similar images, one can come up with a reasonable caption for the target image.
  • Learning-based captioning. By utilising a CNN similar to what we used for tagging, we can capture higher-level features in images. We then attempt to learn the mappings between the higher-level features and their representations in words, using either another neural network or other methods.

We have made some promising forays into the second technique. As a first step, we used a pre-trained CNN-RNN architecture called NeuralTalk to caption our images. As the models are trained on the Microsoft COCO dataset, which consists of pictures and photograph that differs significantly from the British Library's  Flickr dataset, we expect the transfer of knowledge to be difficult. Indeed, the resulting captions of some ~1000 test images show that weakness, with the black-and-white exclusivity of the British Library illustration and the more abstract nature of some illustrations being major roadblocks in the qualities of the captioning. Many of the caption would comment on the “black and white” quality of the photo or “hallucinate” objects that did not exist in the images. However, there were some promising results that came back from the model. Below are some hand-pick examples. Note that this was generated with no other metadata; only the raw image was given.

S1 S2 S3
From a rough manual pass, we estimate that around 1 in 4 captions are of useable quality: accurate, contains interesting and useful data that would aid in search discovery, catalogisation etc., with occasional gems (like the elephant caption!). More work will be directed to help us increase this metric.

Web Interface

We have been working on building the web interface to expose this novel tag data to users around the world.

One thing that’s awesome about making the British Library dataset available via Flickr, is that Flickr provide an amazing API for developers. The API exposes, among other functions, the image website’s search logic via tags as well as free text search using the image title and description, and the capability to sort by a number of factors including relevance and “interestingness”. We’ve been working on using the Flickr API, along with AngularJS and Node.js to build a wireframe site. You can check it out here.

If you look at the demo or the British Library's Flickr album, you’ll see that each image has a relatively sparse set of tags to query from. Thus, our next steps will be adding our own tags and captions to each image on Flickr. We will pre-pend these with a custom namespace to distinguish them from existing user-contributed and machine tags, and utilise them in queries to find better results.

Finally, we are interested in what users will use the site for. For example, we could track user’s queries and which images they click on or save. These images are presumably more relevant to these queries, and we rank them higher in future queries. We also want to be able to track general analytics like the most popular queries over time. Thus incorporating user analytics will be the final step in building the web interface.

We welcome any feedback and questions you may have! Contact us at teamsherlocknet@gmail.com

References

[1] Johnson J, Ballan L, Fei-Fei L. Love Thy Neighbors: Image Annotation by Exploiting Image Metadata. arXiv (2016)