THE BRITISH LIBRARY

Digital scholarship blog

6 posts categorized "Law"

03 November 2016

SherlockNet update - 10s of millions more tags and thousands of captions added to the BL Flickr Images!

Add comment

SherlockNet are Brian Do, Karen Wang and Luda Zhao, finalists for the Labs Competition 2016.

We have some exciting updates regarding SherlockNet, our ongoing efforts to using machine learning techniques to radically improve the discoverability of the British Library Flickr Commons image dataset.

Tagging

Over the past two months we’ve been working on expanding and refining the set of tags assigned to each image. Initially, we set out simply to assign the images to one of 11 categories, which worked surprisingly well with less than a 20% error rate. But we realised that people usually search from a much larger set of words, and we spent a lot of time thinking about how we would assign more descriptive tags to each image.

Eventually, we settled on a Google Images style approach, where we parse the text surrounding each image and use it to get a relevant set of tags. Luckily, the British Library digitised the text around all 1 million images back in 2007-8 using Optical Character Recognition (OCR), so we were able to grab this data. We explored computational tools such as Term Frequency – Inverse Document Frequency (Tf-idf) and Latent Dirichlet allocation (LDA), which try to assign the most “informative” words to each image, but found that images aren’t always associated with the words on the page.

To solve this problem, we decided to use a 'voting' system where we find the 20 images most similar to our image of interest, and have all images vote on the nouns that appear most commonly in their surrounding text. The most commonly appearing words will be the tags we assign to the image. Despite some computational hurdles selecting the 20 most similar images from a set of 1 million, we were able to achieve this goal. Along the way, we encountered several interesting problems.

Similar images
For all images, similar images are displayed
  1. Spelling was a particularly difficult issue. The OCR algorithms that were state of the art back in 2007-2008 are now obsolete, so a sizable portion of our digitised text was misspelled / transcribed incorrectly. We used a pretty complicated decision tree to fix misspelled words. In a nutshell, it amounted to finding the word that a) is most common across British English literature and b) has the smallest edit distance relative to our misspelled word. Edit distance is the fewest number of edits (additions, deletions, substitutions) needed to transform one word into another.
  2. Words come in various forms (e.g. ‘interest’, ‘interested’, ‘interestingly’) and these forms have to be resolved into one “stem” (in this case, ‘interest’). Luckily, natural language toolkits have stemmers that do this for us. It doesn’t work all the time (e.g. ‘United States’ becomes ‘United St’ because ‘ates’ is a common suffix) but we can use various modes of spell-check trickery to fix these induced misspellings.
  3. About 5% of our books are in French, German, or Spanish. In this first iteration of the project we wanted to stick to English tags, so how do we detect if a word is English or not? We found that checking each misspelled (in English) word against all 3 foreign dictionaries would be extremely computationally intensive, so we decided to throw out all misspelled words for which the edit distance to the closest English word was greater than three. In other words, foreign words are very different from real English words, unlike misspelled words which are much closer.
  4. Several words appear very frequently in all 11 categories of images. These words were ‘great’, ‘time’, ‘large’, ‘part’, ‘good’, ‘small’, ‘long’, and ‘present’. We removed these words as they would be uninformative tags.

In the end, we ended up with between 10 and 20 tags for each image. We estimate that between 30% and 50% of the tags convey some information about the image, and the other ones are circumstantial. Even at this stage, it has been immensely helpful in some of the searches we’ve done already (check out “bird”, “dog”, “mine”, “circle”, and “arch” as examples). We are actively looking for suggestions to improve our tagging accuracy. Nevertheless, we’re extremely excited that images now have useful annotations attached to them!

SherlockNet Interface

Sherlocknet-interface
SherlockNet Interface

For the past few weeks we’ve been working on the incorporation of ~20 million tags and related images and uploading them onto our website. Luckily, Amazon Web Services provides comprehensive computing resources to take care of storing and transferring our data into databases to be queried by the front-end.

In order to make searching easier we’ve also added functionality to automatically include synonyms in your search. For example, you can type in “lady”, click on Synonym Search, and it adds “gentlewoman”, “ma'am”, “madam”, “noblewoman”, and “peeress” to your search as well. This is particularly useful in a tag-based indexing approach as we are using.

As our data gets uploaded over the coming days, you should begin to see our generated tags and related images show up on the Flickr website. You can click on each image to view it in more detail, or on each tag to re-query the website for that particular tag. This way users can easily browse relevant images or tags to find what they are interested in.

Each image is currently captioned with a default description containing information on which source the image came from. As Luda finishes up his captioning, we will begin uploading his captions as well.

We will also be working on adding more advanced search capabilities via wrapper calls to the Flickr API. Proposed functionality will include logical AND and NOT operators, as well as better filtering by machine tags.

Captioning

As mentioned in our previous post, we have been experimenting with techniques to automatically caption images with relevant natural language captions. Since an Artificial Intelligence (AI) is responsible for recognising, understanding, and learning proper language models for captions, we expected the task to be far harder than that of tagging, and although the final results we obtained may not be ready for a production-level archival purposes, we hope our work can help spark further research in this field.

Our last post left off with our usage of a pre-trained Convolutional Neural Networks - Recurrent Neural Networks (CNN-RNN) architecture to caption images. We showed that we were able to produce some interesting captions, albeit at low accuracy. The problem we pinpointed was in the training set of the model, which was derived from the Microsoft COCO dataset, consisting of photographs of modern day scenes, which differs significantly from the BL Flickr dataset.

Through collaboration with BL Labs, we were able to locate a dataset that was potentially better for our purposes: the British Museum prints and drawing online collection, consisting of over 200,000 print drawing, and illustrations, along with handwritten captions describing the image, which the British Museum has generously given us permission to use in this context. However, since the dataset is directly obtained from the public SPARQL endpoints, we needed to run some pre-processing to make it usable. For the images, we cropped them to standard 225 x 225 size and converted them to grayscale. For caption, pre-processing ranged from simple exclusion of dates and author information, to more sophisticated “normalization” procedures, aimed to lessen the size of the total vocabulary of the captions. For words that are exceeding rare (<8 occurrences), we replaced them with <UNK> (unknown) symbols denoting their rarity. We used the same neuraltalk architecture, using the features from a Very Deep Convolutional Networks for Large-Scale Visual Recognition (VGGNet) as intermediate input into the language model. As it turns out, even with aggressive filtering of words, the distribution of vocabulary in this dataset was still too diverse for the model. Despite our best efforts to tune hyperparameters, the model we trained was consistently over-sensitive to key phrases in the dataset, which results in the model converging on local minimums where the captions would stay the same and not show any variation. This seems to be a hard barrier to learning from this dataset. We will be publishing our code in the future, and we welcome anyone with any insight to continue on this research.

Captions
Although there were occasion images with delightfully detailed captions (left), our models couldn’t quite capture useful information for the vast majority of the images(right). More work is definitely needed in this area!

The British Museum dataset (Prints and Drawings from the 19th Century) however, does contain valuable contextual data, and due to our difficulty in using it to directly caption the dataset, we decided to use it in other ways. By parsing the caption and performing Part-Of-Speech (POS) tagging, we were able to extract nouns and proper nouns from each caption. We then compiled common nouns from all the images and filtered out the most common(>=500 images) as tags, resulting in over 1100 different tags. This essentially converts the British Museum dataset into a rich dataset of diverse tags, which we would be able to apply to our earlier work with tag classification. We trained a few models with some “fun” tags, such as “Napoleon”, “parrots” and “angels”, and we were able to get decent testing accuracies of over 75% on binary labels. We will be uploading a subset of these tags under the “sherlocknet:tags” prefix to the Flickr image set, as well as the previous COCO captions for a small subset of images(~100K).

You can access our interface here: bit.ly/sherlocknet or look for 'sherlocknet:tag=' and 'sherlocknet:category=' tags on the British Library Flickr Commons site, here is an example, and see the image below:

Sherlocknet tags
Example Tags on a Flickr Image generated by SherlockNet

Please check it out and let us know if you have any feedback!

We are really excited that we will be there in London in a few days time to present our findings, why don't you come and join us at the British Library Labs Symposium, between 0930 - 1730 on Monday 7th of November, 2016?

28 January 2016

Book Now! Nottingham @BL_Labs Roadshow event - Wed 3 Feb (12.30pm-4pm)

Add comment Comments (0)

Do you live in or near Nottingham and are available on Wednesday 3 Feb between 1230 - 1600? Come along to the FREE UK @BL_Labs Roadshow event at GameCity and The National Video Game Arcade, Nottingham (we have some places left and booking is essential for anyone interested) and:

 

BL Labs Roadshow in Nottingham - Wed 3 Feb (1200 - 1600)
BL Labs Roadshow at GameCity and The National Video Game Arcade, Nottingham, hosted by the Digital Humanities and Arts (DHA) Praxis project based at the University of Nottingham, Wed 3 Feb (1230 - 1600)
  • Discover the digital collections the British Library has, understand some of the challenges of using them and even take some away with you.
  • Learn how researchers found and revived forgotten Victorian jokes and Political meetings from our digital archives.
  • Understand how special games and computer code have been developed to help tag un-described images and make new art.
  • Find out about a tool that links digitised handwritten manuscripts to transcribed texts and one that creates statistically representative samples from the British Library’s book collections.
  • Consider how the intuitions of a DJ could be used to mix and perform the Library's digital collections.
  • Talk to Library staff about how you might use some of the Library's digital content innovatively.
  • Get advice, pick up tips and feedback on your ideas and projects for the 2016 BL Labs Competition (deadline 11 April) and Awards (deadline 5 September). 

Our hosts are the Digital Humanities and Arts (DHA) Praxis project at the University of Nottingham who are kindly providing food and refreshments and will be talking about two amazing projects they have been involved in:

ArtMaps: putting the Tate Collection on the map project
ArtMaps: Putting the Tate Collection on the map

Dr Laura Carletti will be talking about the ArtMaps project which is getting the public to accurately tag the locations of the Tate's 70,000 artworks.

The 'Wander Anywhere' free mobile app developed by Dr Benjamin Bedwell.
The 'Wander Anywhere' free mobile app developed by Dr Benjamin Bedwell.

Dr Benjamin Bedwell, Research Fellow at the University of Nottingham will talk about the free mobile app he developed called 'Wander Anywhere'.  The mobile software offers users new ways to experience art, culture and history by guiding them to locations where it downloads stories intersecting art, local history, architecture and anecdotes on their mobile device relevant to where they are.

For more information, a detailed programme and to book your place, visit the Labs and Digital Humanities and Arts Praxis Workshop event page.

Posted by Mahendra Mahey, Manager of BL Labs.

The BL Labs project is funded by the Andrew W. Mellon Foundation.

27 January 2016

Come to our first @BL_Labs Roadshow event at #citylis London Mon 1 Feb (5pm-7.30pm)

Add comment Comments (0)

Labs Roadshow at #citylis London, Mon 1 Feb (5pm-7.30pm)

Live in or near North-East London and are available on Monday 1 Feb between 1700 - 1930? Come along to the first FREE UK Labs Roadshow event of 2016 (we have a few places left and booking is essential for anyone interested) and:

#citylis London BL Labs London Roadshow Event Mon 1 Feb (1730 - 1930)
#citylis at the Department for Information ScienceCity University London,
the first BL Labs Roadshow event Mon 1 Feb (1700 - 1930)
  • Discover the digital collections the British Library has, understand some of the challenges of using them and even take some away with you.
  • Learn how researchers found and revived forgotten Victorian jokes and Political meetings from our digital archives.
  • Understand how special games and computer code have been developed to help tag un-described images and make new art.
  • Talk to Library staff about how you might use some of the Library's digital content innovatively.
  • Get advice, pick up tips and feedback on your ideas and projects for the 2016 BL Labs Competition (deadline 11 April) and Awards (deadline 5 September). 

Our first hosts are the Department for Information Science (#citylis) at City University London. #citylis have kindly organised some refreshments, nibbles and also an exciting student discussion panel about their experiences of working on digital projects at the British Library, who are:

#citylis student panel  Top-left, Ludi Price and Top-right, Dimitra Charalampidou Bottom-left, Alison Pope and Bottom-right, Daniel van Strien
#citylis student panel.
Top-left, Ludi Price 
Top-right, Dimitra Charalampidou
Bottom-left, Alison Pope
Bottom-right, Daniel van Strien

For more information, a detailed programme and to book your place (essential), visit the BL Labs Workshop at #citylis event page.

Posted by Mahendra Mahey, Manager of BL Labs.

The BL Labs project is funded by the Andrew W. Mellon Foundation.

22 January 2016

BL Labs Competition and Awards for 2016

Add comment Comments (0)

Today the Labs team is launching the fourth annual Competition and Awards for 2016. Please help us spread the word by tweeting, re-blogging and telling anyone who might be interested about it!

British Library Labs Competition 2016

The annual Competition is looking for transformative project ideas which use the British Library’s digital collections and data in new and exciting ways. Two Labs Competition finalists will be selected to work 'in residence' with the BL Labs team between May and early November 2016, where they will get expert help, access to the Library’s resources and financial support to realise their projects.

Winners will receive a first prize of £3000 and runners up £1000 courtesy of the Andrew W. Mellon Foundation at the Labs Symposium on 7th November 2016 at the British Library in London where they will showcase their work.

The deadline for entering is midnight British Summer Time (BST) on 11th April 2016.

Labs Competition winners from previous years have produced an amazing range of creative and innovative projects. For example:

(Top-left)  Adam Crymble's Crowdsource Arcade (Bottom-left) Katrina Navickas' Political Meetings Mapper and (Right) Bob Nicholson's Mechanical Comedian.
(Top-left) Adam Crymble's Crowdsource Arcade and some specially developed games to help with tagging images
(Bottom-left) Katrina Navickas' Political Meetings Mapper and a photo from a Chartist re-enactment 
(Right) Bob Nicholson's Mechanical Comedian

A further range of inspiring and creative ideas have been submitted in previous years and some have been developed further.

British Library Labs Awards 2016

The annual Awards, introduced in 2015, formally recognises outstanding and innovative work that has been carried out using the British Library’s digital collections and data. This year, they will be commending work in four key areas:

  • Research - A project or activity which shows the development of new knowledge, research methods, or tools.
  • Commercial - An activity that delivers or develops commercial value in the context of new products, tools, or services that build on, incorporate, or enhance the Library's digital content.
  • Artistic - An artistic or creative endeavour which inspires, stimulates, amazes and provokes.
  • Teaching / Learning - Quality learning experiences created for learners of any age and ability that use the Library's digital content.

A prize of £500 will be awarded to the winner and £100 for the runner up for each category at the Labs Symposium on 7th November 2016 at the British Library in London, again courtesy of the Andrew W. Mellon Foundation.

The deadline for entering is midnight BST on 5th September 2016.

The Awards winners for 2015 produced a remarkable and varied collection of innovative projects in  Research, Creative/Artistic, Entrepreneurship categories and a special Jury's prize:

(Top-left) Spatial Humanities research group at the University Lancaster,  (Top-right) A computer generated work of art, part of  'The Order of Things' by Mario Klingemann,  (Bottom-left) A bow tie made by Dina Malkova  and (Bottom-right) work on Geo-referenced maps at the British Library that James Heald is still involved in.
(Top-left) Spatial Humanities research group at the University Lancaster plotting mentions of disease in newspapers on a map in Victorian times,
(Top-right) A computer generated work of art, part of 'The Order of Things' by Mario Klingemann,
(Bottom-left) A bow tie made by Dina Malkova inspired by a digitised original manuscript of Alice in Wonderland
(Bottom-right) Work on Geo-referencing maps discovered from a collection of digitised books at the British Library that James Heald is still involved in.
  • Research: “Representation of disease in 19th century newspapers” by the Spatial Humanities research group at Lancaster University analysed the British Library's digitised London based newspaper, The Era through innovative and varied selections of qualitative and quantitative methods in order to determine how, when and where the Victorian era discussed disease.
  • Creative / Artistic:  “The Order of Things” by Mario Klingemann involved the use of semi-automated image classification and machine learning techniques in order to add meaningful tags to the British Library’s one million Flickr Commons images, creating thematic collections as well as new works of art.
  • Entrepreneurship: “Redesigning Alice” by Dina Malkova produced a range of bow ties and other gift products inspired by the incredible illustrations from a digitised British Library original manuscript of Alice's Adventures Under Ground by Lewis Carroll and sold them through the Etsy platform and in the Alice Pop up shop at the British Library in London.
  • Jury's Special Mention: Indexing the BL 1 million and Mapping the Maps by volunteer James Heald describes both the work he has led and his collaboration with others to produce an index of 1 million 'Mechanical Curator collection' images on Wikimedia Commons from the British Library Flickr Commons images. This gave rise to finding 50,000 maps within this collection partially through a map-tag-a-thon which are now being geo-referenced.

A further range of inspiring work has been carried out with the British Library's digital content and collections.

If you are thinking of entering, please make sure you visit our Competition and Awards pages for further details.

Finally, if you have a specific question that can't be answered through these pages, feel free to contact us at labs@bl.uk, or why not come to one of the 'BL Labs Roadshow 2016' UK events we have scheduled between February and April 2016 to learn more about our digital collections and discuss your ideas?

We really look forward to reading your entries!

Posted by Mahendra Mahey, Manager of British Library Labs.

The British Library Labs project is funded by the Andrew W. Mellon Foundation.

 

12 November 2015

The third annual British Library Labs Symposium (2015)

Add comment Comments (0)

The third annual BL Labs Symposium took place on Monday 2nd November and the event was a great success!

The Labs Symposiums showcase innovative projects which use the British Library's digital content and provide a platform for development, networking and debate in the Digital Scholarship field.

The videos for the event are available here.

This year’s Symposium commenced with a keynote from Professor David De Roure, entitled “Intersection, Scale and Social Machines: The Humanities in the digital world”, which addressed current activity in digital scholarship within multidisciplinary and interdisciplinary frameworks.

DSL_6178

 Professor David De Roure giving the Symposium keynote speech

Caroline Brazier, the Chief Librarian of the British Library, then presented awards to the two winners of the British Library Labs Competition (2015) – Dr Adam Crymble and Dr Katrina Navickas, both lecturers of Digital History at the University of Hertfordshire.  

   DSL_6204

(L-R): Caroline Brazier, Chief Librarian; Competition winners Katrina Navickas and Adam Crymble; Dr Adam Farquhar, Head of Digital Scholarship 

After receiving their awards, it was time for Adam and Katrina to showcase their winning projects.

Adam’s project, entitled “Crowdsourcing Arcade: Repurposing the 1980s arcade console for scholarly image classification”, takes the crowdsourcing experience off the web and establishes it in a 1980s-style arcade game.

PB021291

Presentation by Dr Adam Crymble, BL Labs Competition (2015)  winner 

Katrina’s project, “Political Meetings Mapper: Bringing the British Library maps to life with the history of popular protest”, has developed a tool which extracts notices of meetings from historical newspapers and plots them on layers of historical maps from the British Library's collections.

PB021332

Presentation by Dr Katrina Navickas, BL Labs Competition (2015)  winner 

After lunch, the Symposium continued with Alice's Adventures Off the Map 2015 competition, produced and presented by Stella Wisdom, Digital Curator at the British Library. Each year, Off the Map challenges budding designers to use British Library digital collections as inspiration to create exciting interactive digital media.

The winning entry was "The Wondering Lands of Alice", created by Off Our Rockers, a team of six students from De Montfort University in Leicester: Dan Bullock, Freddy Canton, Luke Day, Denzil Forde, Amber Jamieson and Braden May.

 

Video: Alice's Adventures Off the Map 2015 competition winner 'The Wondering Lands of Alice'

This was followed by the presentations of the British Library Labs Awards (2015), a session celebrating BL Labs’ collaborations with researchers, artists and entrepreneurs from around the world in the innovative use of the British Library's digital collections.

The winners were: 

BL Labs Research Award (2015) – “Combining Text Analysis and Geographic Information Systems to investigate the representation of disease in nineteenth-century newspapers”, by The Spatial Humanities project at Lancaster University: Paul Atkinson, Ian Gregory, Andrew Hardie, Amelia Joulain-Jay, Daniel Kershaw, Cat Porter and Paul Rayson.  

The award was presented to one of the project collaborators, Ian Gregory, Professor of Digital Humanities at Lancaster University.

PB021372

Professor Ian Gregory  receiving the BL Labs Research Award (2015), on behalf of the Spatial Humanties project, from Dr Aquiles Alencar-Brayner

 

BL Labs Creative/Artistic Award (2015) – “The Order of Things” by Mario Klingemann, New Media Artist.

PB021381

Mario Klingemann receiving the BL Labs Creative/Artistic Award (2015) from Nora McGregor

  

BL Labs Entrepreneurial Award (2015) –“Redesigning Alice: Etsy and the British Library joint project” by Dina Malkova, designer and entrepreneur.

PB021398

Dina Malkova receiving the BL Labs Entrepreneurial Award (2015) from Dr Rossitza Atanassova

 

Jury’s Special Mention Award – “Indexing the BL 1 million and Mapping the Maps” by James Heald, Wikipedia contributor.

PB021417

James Heald receiving the Jury's Special Mention Award (2015) from Dr Mia Ridge

The Symposium concluded with a thought provoking panel session, “The Ups and Downs of Open”, chaired by George Oates, Director of Good, Form & Spectacle Ltd. George was joined by panelists Dr Mia Ridge, Digital Curator at the British Library, Jenn Phillips-Bacher, Web Manager at the Wellcome Library, and Paul Downey, Technical Architect at the Government Digital Service (GDS). The session discussed the issues, challenges and value of memory organisations opening up their digital content for use by others. 

PB021425

Panel session (L-R): George Oates; Jenn Phillips-Bacher; Paul Downey; Mia Ridge

The BL Labs team would like to thank everyone who attended and participated in this year’s Symposium, making the event the most successful one to date – and we look forward to seeing you all at next year’s BL Labs Symposium on Monday 7th of November 2016!

Posted by Mahendra Mahey, Manager of British Library Labs.

The British Library Labs Project is funded by the Andrew W. Mellon Foundation.

05 October 2015

British Library Labs Symposium (2015)

Add comment Comments (0)

  Bl_labs_logo

The BL Labs team are excited to announce that the third annual British Library Labs Symposium (2015) is taking place on Monday 2nd November 2015, from 09:30 –17:00 in the British Library Conference Centre, St Pancras. The event is free, although you must book a ticket. Don’t delay, as last year’s event was a sell out!

The Symposium showcases innovative projects which use the British Library’s digital content, and provides a platform for development, networking and debate in the Digital Scholarship field.

This year, Dr Adam Farquhar, Head of Digital Scholarship at the British Library, will launch the Symposium. This will be followed by a keynote from Professor David De Roure, Professor of e-Research at the University of Oxford and Director of the Oxford e-Research Centre. The British Library’s Chief Librarian, Caroline Brazier, will then present awards to the two British Library Labs Competition (2015) winners, who will follow with presentations on their winning projects.

After lunch, Stella Wisdom, Digital Curator at the British Library, will announce the winners of the Alice’s Adventures Off the Map competition, which challenged budding designers to use British Library digital collections as inspiration in the creation of exciting interactive digital media.

Following, the winners will be announced of the British Library Labs Awards (2015), which recognises projects that have used the British Library’s digital content in exciting and innovative ways. Presentations will be given by the winners in each of the Awards’ three categories: Research, Creative/Artistic and Entrepreneurial.  

The afternoon will end with a thought provoking panel session discussing the issues of opening up digital content for memory organisations, chaired by George Oates, Director of Good, Form & Spectacle Ltd.

The Symposium will conclude with a networking reception in the Chaucer and Foyer area.

Don’t forget to book your place for the Symposium today!

For any further information, please contact labs@bl.uk

Posted by Mahendra Mahey, Manager of British Library Labs.

The British Library Labs Project is funded by the Andrew W. Mellon Foundation.