The traditional, somewhat stereotypical image of the researcher of things past has not changed much in recent times. There is nothing easier than to imagine a scholar sitting at a scarcely illuminated wooden desk, surrounded by piles of old hardbound volumes, spending hours on end rummaging through the sheets in search of a clue.

In the field of eighteenth-century studies, this is certainly still the case. Scholars often go on a pilgrimage to prestigious repositories such as the British Library. However, in the last fifteen years or so, technology has started to offer attractive alternatives to the pleasure of travelling to London. Powered by Gale-Cengage, the Eighteenth-Century Collections Online (commonly referred to as ECCO) is a well-known resource that provides access to English-language and foreign-language publications printed in Britain, Ireland and the American colonies during the eighteenth century. This extensive collection contains over 180,000 titles (200,000 volumes) and allows full-text searching of some 32 million pages. These are digital editions based on the Eighteenth Century microfilming that started in 1981 and the English Short Title Catalogue.

New ECCO home page

Moving away from its classic web-1.0 design, the Gale-Cengage team recently decided to revamp the layout of ECCO – indeed, of their entire portfolio of archive products, which include among others the Seventeenth- and Eighteenth-Century Burney Newspapers Collection. The aim is to make the Gale Primary Sources experience more consistent and intuitive for the user. At the head of this delicate operation are product managers Doran Steele and Megan Sullivan, who lead a nine-person team of software developers, content engineers, researchers and designers. Not quite the IT-only type of personnel, Doran and Megan are scholars themselves, respectively holding degrees in History and Information Science and a remarkable passion for all things past. They are responsible for the maintenance of the existing ECCO interface, as well as the development of the upcoming design refresh.

During a recent interview they gave to the authors of this post, Doran and Megan declared their objective of evolving ECCO in line ‘with user expectations of modern online research experiences’. Their driving force was stated very clearly as a bottom-up process. ‘This redesign’, they explained, ‘is informed by user feedback and market research’. A beta version of the new site has been available since the second half of 2018 to enable the Gale-Cengage team to gather feedback about the new design. The product managers specified that the final transition to the ‘new’ ECCO will only be completed once they feel confident that the new experience ‘successfully meets the needs of our users’. The final goal is a better user experience, ‘one that is faster and more intuitive’. To achieve this, a range of new features have been included, such as more filters on search results; results more relevant to the search queries; data visualization tools; improved subject indexing; more options for adjusting the image; and the ability to download in a text format the OCR (optical character recognition) version of a volume. The latter feature will be a particularly welcome innovation for scholars that often need to look up the occurrence of a single word or cut and paste long chunks of text.

New ECCO search results screen

The options for adjusting the page view are another significant novelty. The beta version boasts new settings to quickly select the preferred zoom level, as well as sliders to increase or decrease the brightness and contrast of the page. These improvements are particularly welcome considering that the quality of the scans remains unchanged. The page quality is not directly related to ECCO. The portal simply allows the consultation of the digitised microfilms included in the first collection (also known as ECCO 1, comprising over 154.000 texts) and the digitisation of a second, smaller collection of books (ECCO 2, over 52.000 titles). This raises an important issue. A plethora of relatively unknown, yet precious eighteenth-century material remains difficult to consult because, on top of the uneven quality in the texts that came out of eighteenth-century printing presses, the original microfilming technology that was employed for the first collection yielded relatively low-resolution results. This causes some hiccups with OCR recognition, thus discouraging the use of quantitative methodologies. But the issue is all the more salient when the category of eighteenth-century visuals is taken into account. At a time when British engravers multiplied in numbers to illustrate the newly-discovered wonders of the natural world or the archaeological remains of Roman cities in England, illustrations became an essential aspect of the eighteenth-century book market and reading experience. While for essential texts such as William Stukeley’s Itinerarium curiosum (1724) or Eleazar Albin and William Derham’s A Natural History of Birds (1734) more refined scans can be found elsewhere, a large number of texts is digitally available only through ECCO 1. Scholars interested in images are either to focus on well-known texts that have been digitised by other providers – with serious consequences in terms of canonicity – or eventually need to plan a visit to major libraries to consult the relevant volumes in person, somehow defeating the very idea of digital reading. Either way, the study of visual culture is somewhat inhibited. Nevertheless, the ‘new’ ECCO promises to enhance the user experience and to offer even more opportunities to engage with outstanding repositories of primary material. If you already had a chance to use the new version, we encourage you to get in touch with Doran and Megan: as your feedback and suggestions can improve ECCO even further.

New ECCO image viewer screen

This post is by Alessio Mattana, Teaching Assistant in Eighteenth-Century Literature at the University of Leeds (on Twitter as @mattanaless), and Dr Giacomo Savani, Teaching Assistant in Ancient History at the University of Leeds (on Twitter as @GiacomoSavani).

A guest post by Geradine Meany, Professor of Cultural Theory in the School of English, Drama and Film and Derek Greene, Assistant Professor at the School of Computer Science, both at the University College Dublin who are organising a FREE workshop and roundtable together with the BL Labs team on Thursday 20 February 2019 at the British Library in London.

How do you set about finding specific references and thematic associations in the massive digital resource represented by the British Library Nineteenth Century Book Corpus, originally digitised through a collaboration with Microsoft?

The Contagion, Biopolitics and Cultural Memory project at UCD Dublin set out to illuminate culturally and historically specific understandings of disease and contagion that appear within the fiction in the corpus. In order to do so, the project team extracted over 35,000 unique volumes out of a total of 65,000 in English and built a searchable interface of 12.3 million individual pages of text, which can be filtered and sorted using the corpus metadata (e.g. author, title, year, etc). The interface incorporates an index of the topical catalogue of volumes used by the British Library from 1823-1985 (within Alston index). Using a combination of OCR text recognition and manual annotation, we have extracted data the two top levels of the index, covering over 98% of the English language texts in the corpus. So for the first time it is possible to reliably identify and extract fiction, drama, history, topography, etc, from the corpus.

Extracting data from 35,000 digitised books

To allow researchers to further filter the corpus to identify texts from niche topic areas, the interface supports the semi-automatic creation of word lexicons, built upon modern “word embedding” natural language processing methods. By combining the resulting lexicons with existing corpus metadata and the data extracted from digitised version of the Alston Index, researchers can efficiently create and export small topical sub-corpora for subsequent close reading.

The Contagion project team is currently using information retrieval and word embeddings to identify texts for close reading. This combination allows us to track key trends pertaining to illness and contagion in the corpus, and interpret these findings with particular reference to current and historical debates surrounding biopolitics, medical culture and migration. Clusters of associations between contagion, poverty and morality are identifiable within the corpus. However, to date our research indicates that Victorians were more worried about religious contamination from migrants and minorities than they were about contagious diseases.

A key feature of the project is the intersection of methodologies and concepts from English literature, automated text mining, and medical humanities. This involves using data analytics as a mode of interpretation not a substitute for it, a way of engaging with the extent and complexity of cultural production in the nineteenth century. Cultural data resists giving definitive yes or no answers to the questions put to it by researchers, but the more cultural data we analyse the better we can map the processes of cultural change and continuity, in all their complexity. The process of tracking themes, topics, and associations enabled by the new interface offers an opportunity to work with and far beyond the existing canon of nineteenth century fiction, itself radically expanded by the last 20 years of scholarship. The identification within the corpus of a very large collection of 3 volume novels indicates that the popular novel is very well represented, for example, while the ability to identify and extract ‘Collected Works’ indicates which writers their contemporaries expected to remain central to the tradition of fiction.

On February 20th 2019, the FREE ‘Reading 35,000 Books’ workshop and roundtable will present the project’s work to date, and will also include discussion by scholars of nineteenth century literature and the British Library Labs of the future development and use of the new searchable interface, including exporting topical sub-corpora for further research.

The event is supported by the Irish Research Council.

 

In this post Tom Derrick, Digital Curator, Two Centuries of Indian Print, explains the Library's recent use of Transkribus for automated text recognition of Bengali printed books.

Are you working with digitised printed collections that you want to 'unlock' for keyword search and text mining? Maybe you have already heard about Transkribus but thought it could only be used for automated recognition of handwritten texts. If so you might be surprised to hear it also does a pretty good job with printed texts too. You might be even more surprised to hear it does an impressive job with printed texts in Indian scripts! At least that is what we have found from recent testing with a batch of 19th century printed books written in Bengali script that have been digitised through the British Library’s Two Centuries of Indian Print project.

Transkribus is a READ project and available as a free tool for users who want to automate recognition of historical documents. The British Library has already had some success using Transkribus on manuscripts from our India Office collection, and it was that which inspired me to see how it would perform on the Bengali texts, which provides an altogether different type of challenge.

For a start, most text recognition solutions either do not support Indian scripts, or do not reach close to the same level of recognition as they do with documents written in English or other Latin scripts. In part this is down to supply and demand. Mainstream providers of tools have prioritised Western customers, yet there is also the relative lack of digitised Indian texts that can be used to train text recognition engines.

These text recognition engines have also been well trained on modern dictionaries and a collection of historical texts like the Bengali books will often contain words which are no longer in use. Their aged physicality also brings with it the delights of faded print, blotchy paper and other paper-based gremlins that keeps conservationists in work yet disrupts automated text recognition. Throw in an extensive alphabet that contains more diverse and complicated character forms than English and you can start to piece together how difficult it can be to train recognition engines to achieve comparable results with Bengali texts.

So it was with more with hope than expectation I approached Transkribus. We began by selecting 50 pages from the Bengali books representing the variety of typographical and layout styles within the wider collection of c. 500,000 pages as much as possible. Not an easy task! We uploaded these to Transkribus, manually segmenting paragraphs into text regions and automating line recognition. We then manually transcribed the texts to create a ground truth which, together with the scanned page images, were used to train the recurrent neural network within Transkribus to create a model for the 5,700 transcribed words.

                                 View of a segmented page from one of the British Library's Bengali books along with its transcription, within the Transkribus viewer. 

The model was tested on a few pages from the wider collection and the results clearly communicated via the graph below. The model achieved an average character error rate (CER) of 21.9%, which is comparable to the best results we have seen from other text recognition services. Word accuracy of 61% was based on the number of words that were misspelled in the automated transcription compared to the ground truth. Eventually we would like to use automated transcriptions to support keyword searching of the Bengali books online and the higher the word accuracy increases the chances of users pulling back all relevant hits from their keyword search. We noticed the results often missed the upper zone of certain Bengali characters, i.e. the part of the character or glyph which resides above the matra line that connects characters in Bengali words. Further training focused on recognition of these characters may improve the results.

Graph showing the learning curve of the Bengali model using the Transkribus HTR tool.      

Our training set of 50 pages is very small compared to other projects using Transkribus and so we think the accuracy could be vastly improved by creating more transcriptions and re-training the model. However, we're happy with these initial results and would encourage others in a similar position to give Transkribus a try.

 

 

Members of the Two Centuries of Indian Print team have just returned from a fascinating trip to Delhi where we took part in a packed programme of activities organised as part of the Association for Asian Studies conference.

We spent most of the week with a group of archivists brought together from a variety of academic and cultural institutions across India and as far away as Cambodia and Australia. What united us was a shared passion for preserving South Asian heritage. As part of the program we led a workshop on Digitisation Standards as practiced by the British Library which also considered the key challenges organisations face when digitising cultural heritage material, including everything from selecting material and scanning, through to post-processing, online display and user engagement. The workshop also featured a paper on the IFLA guidelines for digitisation and (what we hope) was fun activity in which archivists were presented with different case studies of archival collections and asked to consider a digitisation strategy. It certainly sparked a lot of conversation! See photo below

 

Workshop participants taking part in a group activity

 

Undeterred by the inhospitable weather occupying Delhi, we ventured out and were fortunate enough to receive some very thorough and illuminating tours of the Archives and Research Centre for Ethnomusicology, Centre for Art and Archaeology, The National Archives, Indira Gandhi National Centre for the Arts, and Sangeet Natak Akademi where we learned about their respective collections, conservation facilities and digitisation projects.

 

Taking part in a tour of the audiovisual lab at the Archives and Research Centre for Ethnomusicology 

 

This marked the end of a trip which has connected us with inspiring professionals who we hope to collaborate on more events in the near future.

Our thanks go out to the organisers of what turned out to be a very engaging week of activities, to the American Institute of Indian Studies, to Ashoka University, and to the hosts of our workshop, the India International Centre.

 

Dr Mia Ridge writes: in 2017, we made XML and image files from a four-year, AHRC-funded project: The Italian Academies 1525-1700 available through the Library's open data portal. The original data structure was quite complex, so we would be curious to hear feedback from anyone reusing the converted form for research or visualisations.

In this post, Dr Lisa Sampson, Reader in Early Modern Italian Studies at UCL, and Dr Jane Everson, Emeritus Professor of Italian literature, RHUL, provide further information about the project...

New research opportunities for students of Renaissance and Baroque culture! The Italian Academies database is now available for download. It's in a format called XML which represents the original structure of the database.

This dedicated database results from an eight-year project, funded by the Arts and Humanities Research Council UK, and provides a wealth of information on the Italian learned academies. Around 800 such institutions flourished across the peninsula over the sixteenth and seventeenth centuries, making major contributions to the cultural and scientific debates and innovations of the period, as well as forming intellectual networks across Europe. This database lists a total of 587 Academies from Venice, Padua, Ferrara, Bologna, Siena, Rome, Naples, and towns and cities in southern Italy and Sicily active in the period 1525-1700. Also listed are more than 7,000 members of one or more academies (including major figures like Galileo, as well as women and artists), and almost 1,000 printed works connected with academies held in the British Library. The database therefore provides an essential starting point for research into early modern culture in Italy and beyond. It is also an invitation to further scholarship and data collection, as these totals constitute only a fraction of the data relating to the Academies.

Laura Terracina, nicknamed Febea, of the Accademia degli Incogniti, Naples

The database is designed to permit searches from many different perspectives and to allow easy searching across categories. In addition to the three principal fields – Academies, People, Books – searches can be conducted by title keyword, printer, illustrator, dedicatee, censor, language, gender, nationality among others. The database also lists and illustrates the mottoes and emblems of the Academies (where known) and similarly of individual academy members. Illustrations from the books entered in the database include frontispieces, colophons, and images from within texts.

Emblem of the Accademia degli Intronati, Siena

The database thus aims to promote research on the Italian Academies in disciplines ranging from literature and history, through art, science, astronomy, mathematics, printing and publishing, censorship, politics, religion and philosophy.

The Italian Academies project which created this database began in 2006 as a collaboration between the British Library and Royal Holloway University of London, funded by the Arts and Humanities Research council and led by Jane Everson. The objective was the creation of a dedicated resource on the publications and membership of the Italian learned Academies active in the period between 1525 and 1700. The software for the database was designed in-house by the British Library and the first tranche of data was completed in 2009 listing information for academies in four cities (Naples, Siena, Bologna and Padua). A second phase, listing information for many more cities, including in southern Italy and Sicily, developed the database further, between 2010 and 2014, with a major research grant from the AHRC and collaboration with the University of Reading.

The exciting possibilities now opened up by the British Library’s digital data strategy look set to stimulate new research and collaborations by making the records even more widely available, and easily downloadable, in line with Open Access goals. The Italian Academies team is now working to develop the project further with the addition of new data, and the incorporation into a hub of similar resources.

The Italian Academies project team members welcome feedback on the records and on the adoption of the database for new research (contact: www.italianacademies.org).

The original database remains accessible at http://www.bl.uk/catalogues/ItalianAcademies/Default.aspx 

An Introduction to the database, its aims, contents and objectives is available both at this site and at the new digital data site: https://data.bl.uk/iad/

Jane E. Everson, Royal Holloway University of London

Lisa Sampson, University College, London

Vittoria’s 'World of Stories' - the BL Labs Learning and Teaching Award Winner 2017 - is a project led by parents at Vittoria Primary School through the PTA, with the support of school staff. The aim of the project is to collect and share traditional tales from around the world and creative work by current pupils through workshops, the production of a book, school assemblies, readings and performances, and via the creation of audio, text and images for the school website during the current academic year. The illustrations for the project are drawn from the British Library’s Flickr collection which are displayed alongside pupils’ artwork.

The front cover of Vittoria primary school's 'World of Stories'

Our school is a diverse community of learners: pupils’ families come from a wide range of ethnic and cultural backgrounds. Languages spoken by pupils at home include Arabic, Bengali, Vietnamese, Russian, Chechen, Turkish and Somali. One of the pedagogical goals of the project was to make visible the similarities between well-loved traditional tales and explore how different cultures use the same cast of characters - heroes and heroines, tricksters and magicians, villains and monsters – in order to speak across generations about what it means to be human. We wanted to promote and celebrate the diversity of the multi-cultural community which makes up our school, and show parents and children that the characters and stories they love are shared by others from different cultures.   

 The stories in the book include original works by pupils, gathered through a story-writing competition with winning entries selected by the PTA committee. We also asked parents to nominate traditional tales for inclusion in the collection, and held a bi-lingual (English and Arabic) story-sharing workshop for parents organised by the PTA. During the workshop, parents spoke about well-known traditional tales which they remembered from childhood and discussed the contrasts and similarities between characters and narratives from different cultures. For example, the section of the book which presents ‘bogeyman’ type monsters was developed from discussions in the workshop. We discovered that the Beast from Beauty and the Beast is called ‘Al-Ba’ati’ in Sudan, where the story is known as ‘Jamila wal Ba’ati’. Sudanese parents discussed how ‘Al-Ba’ati’ is used to encourage good behaviour in children, which prompted another parent to share her family’s stories of ‘The Boogerman’ who plays a similar role in persuading children to stay in bed at bedtime.

One of the British Library's Flickr images, used as an illustration

The project also links with our work within the classroom to develop children’s reading skills, through promoting a love of reading and books at home. By showing that we value and celebrate the oral culture of storytelling between parents and children, and by collecting and translating tales from languages other than English, we aim to encourage parents to read with their children and support their learning at home.

The project has had a positive effect within the school community, by promoting dialogue and interaction between parents from different cultures through the parents’ workshop, and provided a vehicle to celebrate pupils’ achievements to the school community. Parents have also bought copies of the book to share with family and friends. One of our parent contributors took copies of the book to share with older generations of her family in Sudan during a recent visit, and we hope that other parents will do the same.

During the next phase of the project we will be organising a series of readings and performances using the book with different year groups and making audio recordings which we will publish on the school website for parents to download and listen to with their children at home.

If this blog post has stimulated your interest in working with the British Library's digital collections, start a project and enter it for one of the BL Labs 2018 Awards! Join us on 12 November 2018 for the BL Labs annual Symposium at the British Library.

Posted by BL Labs on behalf of Vittoria Primary school

Making the British Library’s valuable collection of early Bengali books more accessible to researchers and the general public around the world rests heavily on the collaborative work undertaken across different teams of the library and partners in the UK and abroad. The commitment and passion of the project team has relied on the contribution and expertise of collaborators, as well as the forward thinking vision of the library, partners and fundraisers.

Receiving the BL Labs Staff Award 2017 is a great opportunity to thank everyone involved. 

Members of the Two Centuries of Indian Print team receiving the British Library Labs award at the Symposium on 30th October 2017

 

Tom Derrick (Digital Curator) was in India at the same time the team received their Award

The Two Centuries of Indian Print project is a partnership between the British Library, the School of Cultural Texts and Records (SCTR) at Jadavpur University, Srishti Institute of Art, Design and Technology, and the Library at SOAS University of London, among others. It has also involved collaborations with the National Library of India, and other institutions in India.

The AHRC Newton-Bhabha Fund and the Department for Business, Energy and Industrial Strategy have generously funded the work undertaken so far by the project, focusing on early printed Bengali books. Many are unavailable in other library collections or are extremely difficult to locate and access. The project has undertaken a variety of initiatives from the digitisation of books and enhancement of the catalogue records in English and Bengali, to stimulating the use of digital humanities tools and techniques, running a programme of digital skills sharing and capacity building workshops, and hosting the South Asia Series seminars. All of these initiatives greatly contribute to the discovery and study of the collection. The project is also conducting ground breaking work in finding a solution to Optical Character Recognition (OCR) in Bangla script. OCR is not available for South Asian languages currently and harnessing viable Optical Character Recognition technology would enable full text search of the books, paving the way for researchers to use natural language processing techniques to perform large scale analysis across a large corpus of text covering a diverse range of topics relating to Indian society, religion, and politics to name but a few. Doing so will increase the possibilities for new discoveries in this academic field. 

However, despite its status as one of the most widely spoken languages in the world, Bangla script has been greatly underserved by providers of OCR solutions. This is due in part to the orthographical and typographical variances that have taken place in recent centuries that make building a dictionary and character ‘classifier’ more challenging. Due to the wide date range of the books we are digitising, these issues affect the quality of OCR. The physical condition of our historical books, including faded text, presents additional difficulties for creating machine readable versions of the books. 

To overcome these obstacles, the project team has been advancing the development of OCR for Bangla through the organisation of an international competition which reviewed the state-of-the-art in commercial and open source text recognition tools. The results of the competition will be announced at the ICDAR 2017 conference in Kyoto later this month. Watch this space! The competition dataset has been made openly available for download and reuse for any researchers or institutions who would like to experiment with OCR for Bengali.

A page from the Animal Biographies, VT 1712 showing its transcription produced for the ICDAR 2017 competition

The project has organised two Skills Exchange Programmes, hosting mid-career Library professionals from the the National Library of India at the British Library for a week, providing a packed programme of tours and talks from all areas of the Library. The project has also conducted digital skills sharing and capacity building workshops for library professionals and archivists from cultural heritage institutions in India. The first workshop took place at Jadavpur University, Kolkata, in December 2016. Library and information professionals from cultural heritage institutions in Bengal took part in a one-day event to learn more about how information technology is transforming humanities research today and in turn Library services, as well as the methods for interrogating humanities-related datasets.

Afterthe success of this first workshop another event was held in July 2017, at which more than 30 library professionals discussed OCR developments for Bangla, trying out different tools and discussing digital scholarship techniques and projects. Most recently, the project’s digital curator facilitated a workshop around Digitisation Standards at the International Conference of Asian Libraries in Delhi. The workshops continue in earnest in the new year with another digital humanities skills workshop planned for January 2018 to be held in partnership with the Srishti Institute of Art, Design, and Technology.

Attendees of the workshop held at Jadavpur University in December 2016 taking part in a group activity to discuss the application of digital humanities methods to library collections

The Project Team also held a two day Academic Symposium on South Asian book history at Jadavpur University in the summer, with 17 speakers from India, wider South Asia, and the UK. Attendance was between 50-70 people a day and feedback was very good.  We plan to have a publication arising from this Symposium, and to upload a video to our project webspace. The project also hosts a popular series of talks based around the Two Centuries of Indian Print project and the British Library’s South Asia collections. The seminars take place fortnightly at the British Library. So far we have hosted a range of academics and researchers, from PhD students to senior academics from the UK and abroad, who share cutting-edge research with discussion chaired by curators and specialists in the field. The seminars have been a great success attracting large attendances and speakers from around the world. We also host a number of show and tells of our material to raise awareness for our collection and to engage in community outreach.

Everyone on the project is thrilled to have won this award and we will be working hard in 2018 to continue bringing the Two Centuries of Indian Print project to the attention and use of researchers and the general public.

Submit a project for one of the BL Labs 2018 Awards! Join us on 12 November 2018 for the BL Labs annual Symposium at the British Library.

Posted by BL Labs on behalf of The Two Centuries of Indian Print team.

Applications of Distributed Version Control Technologies Toward the Creation of 50,000 Digital Scholarly Editions

Jonathan Reeve

The British Library maintains a collection of roughly 50,000 digital texts, scanned from public-domain books, most of which were originally published in the 19th century. As scanned books, their text format is Analyzed Layout and Text Object (ALTO) Extensible Markup Language (XML), a verbose markup format created by Optical Character Recognition (OCR) software, and one which is only marginally human-readable. Our project, Git-Lit, converts each text to the plain text format Markdown, creates version-controlled repositories for each using the distributed version control system Git, and posts the repositories to the project management platform GitHub, where they can be edited by anyone. Along the way, websites for each text, optimized for readability, are automatically generated via GitHub Pages. These websites integrate with the annotation platform Hypothes.is, enabling them to be annotated. In this way, Git-Lit aims to make this collection of British Library electronic texts discoverable, readable, editable, annotatable, and downloadable.

A Screenshot of the Website Automatically Generated from the British Library Electronic Text

The biggest advantage of using a distributed version control system like Git is that it leverages the kinds of decentralized collaboration workflows that have long been in use in software development. Open-source software and web development, for which Git and GitHub were originally designed, is a much-studied methodology, long proven to be more effective than closed-source methods. Rather than maintain a central silo for serving code and electronic texts, the decentralized approach ensures a plurality of textual versions. Since anyone may copy ("fork") a project, modify it, and create their own version, there is no one central, canonical text, but many. Each version may freely borrow ("pull") from others, request that others integrate their changes ("pull request"), and discuss potential changes ("issues") using the project management subsystems of GitHub. This workflow streamlines collaboration, and encourages external contributions. Furthermore, since each change ("commit") requires a description of the commit, and reasons for it, the Git platform enforces the kind of editorial documentation necessary for scholarly editing. We like to think of git-based editing, therefore, as scholarly editing, and GitHub-based collaboration as a democratization of scholarly editing.

Furthermore, since GitHub allows instant editing of texts in the web browser, it is a simple and intuitive method of crowdsourcing the text cleanup process. Since OCRd texts are often full of errors, GitHub allows any reader to correct an obvious OCR error she or he finds. The analogous process of reporting errors to centralized text repositories like Project Gutenberg has been known to take several years. On GitHub, however, it is instantaneous.

Not the least advantage of this setup is the automated creation of websites from the plain text sources. Not only does this transform the markdown to a clean, readable edition of the text, but it provides integration with the annotation platform Hypothes.is. Hypothes.is allows for social annotation of a text, making it ideal for classroom use. Professors may assign a British Library text as a course reading, and may require their students annotate it, an activity which can generate discussions in the limitless virtual margins of this electronic textual space.

The Git-Lit project has so far posted around 50 texts to GitHub, as prototypes, with the full corpus of roughly 50,000 texts soon to come. After the full corpus is processed in this way, we'll begin enhancing some of the metadata. So far, we have developed techniques for probabilistically inferring the language of each text, and using Ben Schmidt's document vectorization method, Stable Random Projection, we have been able to probabilistically infer Library of Congress classifications, as well. This enables the automatic generation of sub-corpora like PR (British Literature), or PZ (American Literature).

In the coming year, we hope to integrate the Git-Lit transformed British Library texts into a structured database, further enhancing the discoverability of its texts. We have just received a micro-grant from NYC-DH to help launch Corpus-DB, a project also aiming to produce textual corpora, and through Corpus-DB, we will soon create a SQL database containing the metadata, our enhanced and inferred metadata, and other aggregated book data gleaned from public APIs. This will soon allow readers and computational text analysts the ability to download groups of British Library electronic texts. Users interested in downloading, say, all novels set in London, will be able to get a complete full-text dump of all public-domain novels in this category by visiting a URL such as api.corpus-db.org/novels/setting/London. We expect that this will greatly streamline the corpus creation process that takes up so much of the time of a computational text analysis.

Both Git-Lit and Corpus-DB are open-source projects, open to contributions from anyone, regardless of skill. If you'd like to contribute to our project in some way, get in contact with us, and we'll tell you how you can help.

Jonathan Reeve

Jonathan Reeve is a third-year graduate student in the Department of English and Comparative Literature at Columbia University, where he specializes in computational literary analysis. Find his recent experiments at jonreeve.com.

If this blog post has stimulated your interest in working with the British Library's digital collections, start a project and enter it for one of the BL Labs 2018 Awards! Join us on 12 November 2018 for the BL Labs annual Symposium at the British Library to find out who wins.

Posted by BL Labs on behalf of Jonathan Reeve