UK Web Archive blog

By Jason Webber, Web Archive Engagement Manager, British Library

Welcome to the UK Web Archive 'Summer of Sport' season! Over the next few months we will show the many ways that sport is represented in the web archive.

Let's start with some of the more quirky and unusual 'sports' played in the UK:

Cheese Rolling Championship

Brave competitors chase a wheel of cheese down the terrifyingly steep Cooper's hill (1 in 1 in places) in Gloucestershire. First one to the bottom is the winner! The prize is a 7-8lb wheel of Double Gloucester cheese!

The official Cheese Rolling Championship website in 2008.

The Chap Olympiad

What sport can there be for the well dressed 'person-about-town'? The 'Chap Olympiad' of course.

"A series of challenges ensue ranging from the frantic and frenetic to the barely mobile. The Tea Pursuit and Umbrella Jousting (where participants clamber aboard a bike holding an umbrella and a briefcase) see what is possibly the first use of Boris bikes as part of a sporting contest. The Tug Of Hair pits two teams against each other, pulling on a twenty feet long moustache until one team topples over. In Well-Dressage, individuals mount hobbyhorses and prance around to music while Not Tennis is the epitome of anti-sport with two players invited to do anything but play tennis."

Photos of the Chap Olympiad from the Londonist website in 2016.

Bog Snorkeling

If an athlete is not afraid of a spot of mud, what better event than the Bog Snorkeling Championships! Competitors aim to complete two consecutive lengths of a 60 yards (55 m) water-filled trench cut through a peat bog in the shortest time possible, wearing traditional snorkel, diving mask and flippers.

"Event rules state that no recognised swimming strokes are allowed at the event so it all comes down to honing down the perfect technique to power through the murky water."

Bog Snorkeling on the Visit Wales website from April 2013. 

World Conker Championships

Threading a piece of string through a horse chestnut seed and hitting another one has been a long standing feature of school playgrounds. Conkers, however, is a serious business and over a thousand are used in each World Championship contest!

Photo of the World Conker Championship 2016.

BBC News article on the World Conker Championship in 2004.

Summary
We aim to capture all aspects of UK life including the sporting life. If you have a UK sport website that you would like to suggest for the web archive, nominate it here.

#WebArchiveSummerOfSport

By Steven Dryden, British Library LGBTQ+ Staff Network & Ash Green CILIP LGBTQ+ Network

As you’ll have read on this blog, the collaboration with UK Web Archive (UKWA), British Library and CILIP LGBTQ+ Network to develop LGBTQ+ content within the UK Web Archive was launched during summer 2020.

LGBTQ+ content was already part of the UK Web Archive before the collaboration began, with many sites in other collections overlapping LGBTQ+ themes. For example, Black and Asian Britain (blackgayblog.com), Gender Equality (Beyond the Binary), Sport (Graces Cricket Club). And some sites cut across many collections, highlighting the intersectional nature of the UK Web Archive. For example, Gal-Dem features in the News Sites; Zines and Fanzines; Black and Asian Britain; Gender Equality; Women's Issues; Unfinished Business: The Fight for Women’s Rights collections, as well as LGBTQ+ Lives Online. LGBTQ+ Lives Online, much like the lived experience of the LGBTQ+ does not sit in isolation, disconnected from other aspects of UK offline and online life. LGBTQ+ people play a part in all aspects of the UK community, and are not solely defined by their gender or sexual orientation.

This UK Web Archive collection doesn’t stand in isolation either, it enriches the scope of work already begun at The British Library.LGBTQ Histories aims to explore the experiences and stories encountered in the collections, posing questions about the lived experience of LGBTQ+ people throughout history.The LGBTQ+ Lives Online collection of the UK Web Archive plays a part in CILIP LGBTQ+ Network’s ambition to raise the profile of LGBTQ+ people, support the development of LGBTQ+ information resources and the work of LGBTQ+ Library, information and knowledge workers.

LGBTQ+ Lives Online Collection

The collection currently contains over 400 sites and web pages in the main collection, with more of these being added to sub-collections every week. Many of the sites were already in the UKWA before the collaboration began, but were not linked to sub-collections. We are still at the stage where we are developing the structure of sub-collections but our initial indexes cover:

• Activism/Pride
• Arts, Literature, Music & Culture
• Business/Commerce
• Education
• History
• Medicine and Community
• Policy and Legislative Change
• Religion
• Social Organisations
• Sport

Since the launch of this collaborative project, we have been focused on a number of areas to both develop the project and to preserve sites within the collection. This includes:

• Identifying sites already in the UK Web Archive to be added to the LGBTQ+ Lives Online sub-collections.
• Identifying new sites not already in the UKWA to be included in the collection.
• Spreading the word about the project as widely as possible via blog posts and articles such as this; social media; emails targeting specific LGBTQ+, library, and broader diversity organisations and networks.

You can browse through the collection here, and nominate a UK published site or webpage with a focus on LGBTQ+ lives to be included in the collection via: https://www.webarchive.org.uk/en/ukwa/info/nominate. We would especially like to see more nominations that reflect the multicultural nature of UK LGBTQ+ communities and the many diaspora communities based here, including UK sites written in languages other than English.

Though it can often be challenging for us to archive social media accounts, we are able to collect LGBTQ+ Twitter accounts. We have experimented with other methods of archiving social media but this is on a selective basis, but we would welcome nominations and projects that might address these challenges and how they might impact on archiving LGBTQ+ experience in the UK,

How can you access these archived websites?

Under the Non-Print Legal Deposit Regulations 2013, the UKWA  can archive UK published websites, but are only able to make the archived version available to people outside the Legal Deposit Libraries Reading Rooms, if the website owner has given permission. The UK Legal Deposit Libraries are the British Library, National Library of Scotland, National Library of Wales, Bodleian Libraries, Cambridge University Library and Trinity College Dublin Library.  

Some of the websites in UKWA have already had permission granted, these include Out Stories Bristol, Trans Ageing and Care, Bi Cymru/Wales and Queer Zine Library. As the content of UKWA has mixed access, the message ‘Viewable only on Library premises’ will appear under the title of the website if you need to visit a Legal Deposit Library to view content. If there is no message underneath then the archived version of the website should be available on your personal device.

Due to the coronavirus pandemic, the reading rooms were closed for a number of weeks but are starting to reopen. This blog post gives an overview of opening hours and how to book a visit at the six UK Legal Deposit Libraries:

https://blogs.bl.uk/webarchive/2020/09/ukwa-available-in-reading-rooms-again.html 

Previous blog posts about the project can be viewed via the following links.

LGBTQ+ Lives Online project introduction

LGBTQ+ Lives Online: Introducing the Lead Curators

By Andy Jackson, Web Archiving Technical Lead at the British Library

On the 10th of September the 2020 Domain Crawl got underway. The annual Domain Crawl usually takes about three months to complete, it visits UK published websites on a UK Top Level Domain (TLD) like .uk, .cymru, .scot, .london etc., any web content hosted on a server registered in the UK as well as all the records manually created by the UK Web Archive teams across the UK Legal Deposit Libraries. 

Update on crawl management

Due to the billions of URLs involved, the Domain Crawl is the most technically difficult crawl we run. As the crawl frontier grows and grows, the strain starts to show, particularly on the disk space required to store all of the status information about the URLs that have been crawled or are awaiting crawling. Worst of all, we found some mysterious problems with how Heritrix3 manages this information, meant that we could not safely stop and restart long crawls. We could usually restart once, but if we restarted again strange errors would appear, and sometimes these would be serious enough to cause the whole crawl to fail. Fortunately, in the last year, we finally tracked this down and updated the Heritrix3 crawler so that it can be safely stopped and restarted multiple times. 

This has made managing the crawler much easier, as we can stop and restart the crawl with confidence if we need to change the software or hardware setup. This makes managing things like disk space much less stressful.

Update on the crawl performance 

In the initial phase of the crawl, we threw in the roughly 11 million web hostnames that we have seen in past crawls, which then got whittled down to about 7 million active hosts. After this bumpy start and some system tuning, the crawl settled down and has been pretty consistently processing 250-300 URLs per second.  This is acceptable, but isn’t quite as fast as we would like, so we are analysing the crawl while it runs to try and work out where the bottlenecks are.

What we have collected so far

The figure below shows the URLs collected over time.

Graph illustrating the number of URLs downloaded in the 2020 Domain Crawl

The rather jagged start shows where we were able to stop and start the crawl in order to tune the initial hardware setup, and the flatter ‘pauses’ later on are from other maintenance activities like growing the available disk space. The advantage of being able to re-tune the crawler as we go is shown by the way the line gets steeper over time, corresponding to the increased crawl rate.

In terms of bytes downloaded, we see a similar result:

Graph illustrating the number of TBs downloaded in the 2020 Domain Crawl

As you can see, we are rapidly approaching 90TB of downloaded data, which corresponds to roughly 50TB of compressed WARC.gz data.

Despite starting the crawl relatively late in the year (due to issues around the COVID-19 outbreak), we are making good and stable progress and are on track to download over two billion URLs by the end of the year.

Follow the UK Web Archive on Twitter for the latest updates on the Domain Crawl and other web archiving activities! 

By Hannah Connell, Collaborative PhD Student, King’s College London; British Library

I spoke about my position as a curator for the Russia in the UK curated collection as part of the recent Engaging with Web Archives conference (EWA), which was held online from the 21^st-22^nd of September 2020. This conference reflected the breadth of the web archiving community, bringing together speakers from researchers to librarians, as well as curators and web archiving teams from many different countries.

As always, it was inspiring to participate in such a welcoming event. Even online, the conference retained the collaborative atmosphere which has marked my experience of research in web archiving, allowing new researchers to interact with more experienced practitioners and encouraging questions and conversations between researchers, users and archivists.

The researcher-curated collection, Russia in the UK, is part of the UK Web Archive (UKWA). I was particularly pleased to have had the opportunity to present this curated collection, a resource on the Russian-speaking community in the UK, which was first started in November 2017. Such collections play an important role in making the wide range of material preserved in the UKWA more visible to researchers.

Curators are important to the preservation work of the UKWA. Curated collections are collected manually by curators and researchers with specialist knowledge in their field. The role of a curator in creating a UKWA collection involves identifying relevant websites to be included in a collection, and recording the metadata for these websites, including the translation and transliteration of titles and descriptions in other languages.

This collection is valuable both as a resource for further research, and as a means of questioning research practices. It is not possible to capture everything on the web, and collection curators ensure that a representative sample of websites for each thematic collection are selected. The practice of creating and maintaining a collection such as the Russia in the UK  ultimately influences the shape of the collection and the online representation of the diasporic community it will come to reflect. As such, it is important for researchers and users to understand the decisions taken by curators in selecting and capturing websites.

My paper for EWA focused on the creation of a curation guide for curators of new curated collections. This  draws on the ongoing process of curating the Russia in the UK collection, documenting both the provenance of this special collection and reflecting on this process as a model for future collections.  

In documenting the creation of this collection, I hope to enable future researchers to explore and contribute to this record of the online activity of the Russian diaspora in the UK, and to question and develop the curatorial and research practices behind the curation of collections.

You can watch Hannah Connell’s presentation on the EWA YouTube channel.

By Steven Dryden, British Library LGBTQ+ Staff Network & Ash Green CILIP LGBTQ+ Network

In July 2020 the British Library, the UK Web Archive and CILIP LGBTQ+ Network relaunched the LGBTQ+ Lives Online web archive collection. We have received many nominations for new sites to be collected by the UK Web Archive and work has begun to re-tag many of the websites that have been collected since the UK Web Archive began collecting the UK web in 2005.

To mark two months since the project began, LGBTQ+ Lives Online leads Steven Dryden, of the British Library, and Ash Green, of CILIP LGBTQ+ Network write about the relevance of the World Wide Web to them as members of the LGBTQ+ community, and some of their collection highlights:

Steven he/him/his

Steven Dryden

I first encountered the internet in Las Vegas. It was the summer of 1998, I was 17 and my family had migrated from Newcastle Upon Tyne to the western world’s party play pit in the Nevada desert. My friend, Lilian, was talking to someone in New York City about the band Depeche Mode through America Online (AOL).

Chat rooms were online spaces that allowed groups of people to join anonymously and had the options to talk and interact within a group or in private. Chatrooms quickly became a pivotal part of my small cohort of friends and I, the odd balls who didn’t quite fit, as we were forming our identities in those formative late teen years, and trying to find our place in the world.

Later the same year on October 12, 1998 Matthew Shepard would die. A gay student at the University of Wyoming, Shepherd was beaten, tortured, and left to die near Laramie on the night of October 6, 1998. AOL chatrooms formed the major part of how I found out about Shepherd, worked through my feelings about his murder, and was the first news story that I followed online.

The protections and general understanding of who the lesbian, gay, bisexual and transgender community are has undergone radical change in the 22 years since I first encountered the internet. I’m interested to see what survives online of the change in language relating to the community, and what evidence remains in the UK Web Archive of the online discussion. Some websites that interest me in these first months of the project include:

• The Campaign for Homosexual Equality: an organisation which led the way to legal reform in the UK, following the passing of the Sexual Offences Act 1967, which partial decriminalised homosexuality in England and Wales.

https://www.webarchive.org.uk/wayback/en/archive/20130505124828/http://www.c-h-e.org.uk/

• Around the Toilet: a community engaged art project exploring the accessibility and culture of toilets for the LGBTQ+ community

https://www.webarchive.org.uk/wayback/en/archive/20180606164959/https://aroundthetoilet.wordpress.com/

• Asexual Visibility and Education Network: founded in 2001 with two distinct goals: creating public acceptance and discussion of asexuality and facilitating the growth of an asexual community

https://www.webarchive.org.uk/wayback/en/archive/20150226230020/http://www.asexuality.org/home/

Ash (they/them)

Ash Green

When I was studying for my BA Information and Library Management degree in the early 1990s, the internet and World Wide Web weren’t as high profile as they are now. I loved tech back then, and was into programming and creating databases as part of the degree. But I didn’t really understand what the lecturers were talking about when they mentioned the internet. At the time I had no idea how important it would be to my coming out just over 20 years later, and what a positive impact it would have.

Thinking about the lead up to my coming out in 2017, without access to sites and forums related to trans/gender non-conforming lives in particular, I doubt I would have come out at all. But when I decided to look for guidance online, I found a huge amount of information that was overwhelming at first, but eventually this helped me understood where I fitted into the world. They included medical sites; statements from WHO and other health organisations highlighting that being trans wasn’t a mental health issue; personal blogs and forums, talking about experiences and a variety of perspectives on what it means to be trans; finding out about non-binary, genderfluid, and genderqueer people experiences (I had no idea what these words meant); LGBTQ+ events; makeup and style tips; sites for face-to-face support groups and meetups, and sites for exhibitions such as the Museum of Transology and the Transworkers photography exhibition, which helped me understand that being trans is much broader than mainstream media would have the world believe.

Many sites were useful, but at the same time I came across quite a few that were more "Yes, this miracle herbal treatment really does change your hormones", and "You're only valid if you fit into trans box X or Y" that put my critical, digital literacy and research experience into practice. I also found supportive friends and allies, and I was able to share useful sites and sources of information I’d discovered to give them a better understanding of my experience. It’s important that these sites should be a part of the UK Web Archive LGBTQ+ Lives Online collection. Not only because they have a relevance to the UK Web Archive in general, but from a personal perspective I feel that if they had such an impact on helping me find where I fit into the world, how many other people have they also had a similar positive impact upon?

The sites I’ve chosen below from the UK Web Archive have all had a personal impact upon myself.

• Museum of Transology: The UK’s most significant collection of objects representing trans, non-binary and intersex people’s lives. 

https://www.webarchive.org.uk/wayback/en/archive/20201003091027/https://www.museumoftransology.com/

• OutStories Bristol: Collecting and preserving the social history and recollections of LGBT+ people living in or associated with Bristol, England.

https://www.webarchive.org.uk/wayback/en/archive/10000101000000/https://outstoriesbristol.org.uk/

• Outline Surrey: Outline provides support to people with their sexuality and gender identity, including but not limited to the lesbian, gay, bi-sexual and trans community of Surrey, primarily through a helpline, website and support groups.

https://www.webarchive.org.uk/wayback/en/archive/20160107134238/http://www.outlinesurrey.org/

Get involved with preserving UK LGBTQ+ Lives Online with the UK Web Archive

We can’t curate the whole of the UK web on our own, we need your help to ensure that information, discussions, personal experiences and creative outputs related to the LGBTQ+ community are preserved for future generations. Anyone can suggest UK published websites to be included in the UK Web Archive by filling in our nominations form:

https://www.webarchive.org.uk/en/ukwa/nominate

By Helena Byrne, Curator of Web Archives at the British Library

Creepy crawlers - British Library digitised image from page 79 of "The Child's Book of Poetry. A selection of poems, ballads and hymns"

Halloween and the UK Web Archive

From the start of October all the shops and supermarkets were filling up with Halloween costumes, decorations and lots of fun sized confectionery that are easy to share with some of the trick-o-treaters who might be knocking on your door. It is not clear yet how the coronavirus pandemic will impact any of the informal celebrations that take place every year. No doubt the UK Web Archive crawlers are picking up lots of Halloween and 5th of November themed webpages as part of the 2020 Domain Crawl.

Halloween in the UK is often perceived to be a cultural import from North America. A YouGov poll in 2019 showed that only 30% of people surveyed were planning to celebrate the occasion. This Shine graph shows how the popularity of the term on the archived .uk web has increased in popularity over time. 

Click on a point in the graph to see a sample of how the phrase is used.

Halloween History

The tradition of Halloween actually goes back centuries and was widely celebrated by people in Ireland, Britain and northern France. During pagan times, the 1st of November was officially the start of winter, this season was  associated with death as the crops, wildlife and many people died due to the cold and lack of sunlight during this period. Because of this day’s association with death, it was believed that the ghosts of the dead returned to earth on the night of the 31st October. It was during the Reformation that the tradition of celebrating Halloween died out in Britain, especially in England. 

A recent YouGov poll has shown that Guy Fawkes Night is more popular in Great Britain than Halloween. 

The 5th of November

The commemoration on the 5th of November goes by many names, traditionally it was Guy Fawkes Night but is sometimes referred to as Bonfire Night or Fireworks Night. But there seems to be some regional differences in what term is used and how the night is celebrated. 

What do you call this commemoration?

This is a question we visited back in 2017 and as you can see in the Shine graph in more recent years the term Bonfire Night was used more widely on the archived .uk web. 

Get creative with Halloween at the British Library

Our Assistant Web Archivist, Carlos Lelkes-Rarugal, has designed some short animated videos using recordings from the British Library Sound Archive and images from the Ghosts & Ghoulish Scenes, British Library Flickr. See these on the UK Web Archive, Digital Scholarship and the Sound Archive’s Wildlife Department Twitter accounts.

This and other sounds can be experienced in the Sound Archive at the British Library which has over 260,000 wildlife sound recordings from all over the world. You can hear a selection of some of these recordings on the British Library, Sound & Vision blog, the latest blog post Going Batty for Halloween, gives an overview of the history of bats and Halloween. 

The Digital Scholarship’s latest blog post, Mind Your Paws and Claws, encourages you to use these images and sounds for various creative projects. The Ghosts & Ghoulish Scenes Flickr Album was previously used in the Gothic Off the Map competition. 

Get involved with preserving the UK web

The UK Web Archive aims to archive, preserve and give access to the historic UK web space. We endeavour to include important aspects of British culture and events that shape society. 

Anyone can suggest UK websites to be included in the UK Web Archive by filling in our nominations form: www.webarchive.org.uk/nominate 

We have a Festivals collection, but are there any local Halloween or 5th of November events near you that haven’t been added yet? Equally, if these events have now been cancelled, we would like to add some of these online cancellation notifications to our collection Coronavirus (COVID-19) UK. Browse through what we have so far and please nominate more content!

By Rosita Murchan, Web Archivist, Public Record Office of Northern Ireland (PRONI)

Screenshot of the PRONI Web Archive homepage

The Public Record Office of Northern Ireland (PRONI) Web Archive has been building its collection of websites for almost ten years, focusing initially on capturing the websites of our Government departments and local councils but also websites deemed historically or culturally important to Northern Ireland.

Our collection has grown in both size and scope and we now have over 240 captured sites that range from Instagram and twitter feeds to local history group pages to significant inquest sites with one terabyte of data being captured each year.

Unlike the rest of UK where legal deposit libraries are entitled to copy published material from the internet under the 2013 Non-Print Legal Deposit Regulations (NPLD),  PRONI has no legal deposit status and rather works on a permissions based approach where we write to website owners informing them of our intention to archive their website and then operate on a ‘silence is consent’ approach crawling a site anyway and taking down websites should the owner request it. 

In an attempt to continue to expand our collection we have been very lucky to have been invited to collaborate with the UK Web Archive team based at the British Library on some of their projects in the last year and have added a Northern Ireland perspective on the topics of Brexit and the General Election 2019 and more recently a NI Covid-19 viewpoint.

Part and parcel of this collaboration included us getting access to and being able to use and add to the UK Web Archive ACT Web Archiving tool. For me this was a great opportunity to see and use another web archiving tool, especially a custom piece of software for an institution as reputable as the British Library and it has been a fantastic opportunity for us to archive sites that would normally be outside our remit, as a result we have been able to add to further research for Northern Ireland.

We really hope to continue this partnership with the British library going forward, not only as a method of increasing the amount of NI archived sites but also as a way to continue to improve and learn from their expertise.

You can watch Rosita Murchan’s presentation on the EWA YouTube Channel.

By Brendan Power, Digital Preservation Librarian, Library of Trinity College Dublin

At the recent conference, ‘Engaging with Web Archives: Opportunities, Challenges and Potentialities’, I presented a paper on a collaborative project between The Library of Trinity College Dublin, the University of Dublin, the Bodleian Libraries, the University of Oxford, and the British Library. The project was carried out in 2015/16 and aimed to identify, collect, and preserve online resources related to the 1916 Easter Rising and the diverse ways it was commemorated and engaged with throughout its centenary in 2016. The Bodleian Libraries primarily collected UK websites under the provisions of the 2013 Non-Print Legal Deposit Regulations (NPLD), while The Library of Trinity College Dublin focused on websites in the .ie domain. Since no legislation exists in the Republic of Ireland to ensure that the .ie domain is preserved, websites within the .ie domain were collected on a voluntary basis, that is, with the express formal permission of the website owners through the signing of a license agreement.

We aimed to reflect the variety of ways that the Irish and British states, cultural and educational institutions, as well as communities and individuals, approached the centenary events. These included official commemorative websites, the websites of museums, archives, heritage, cultural, and education institutions, along with traditional and alternative news media websites, blogs, and community websites. These resources will be invaluable primary resources to analyse how people interpreted and engaged with the Easter Rising in its centenary year. Researchers have reflected on the events organised on the fiftieth anniversary of the Easter Rising in 1966 and how these events were framed, the aspects that were championed, and the critical viewpoints denied expression. In a similar way, the records created throughout the centenary will be an essential resource for researchers in analysing how the generations of 2016 engaged with the legacy of the Easter Rising and the approaches, themes, and tone adopted.

The resulting web archive collection contains over 318 seeds, i.e. websites or sub-sections of these. Of these 318 websites, 112 (35%) were selected by The Library of Trinity College Dublin, 190 (60%) by the Bodleian Libraries, and 16 (5%) by curators at the British Library. 118 (37%) of the websites were from the .ie domain, 172 (54%) were from the .uk domain and 28 (9%) were associated with other areas, predominantly the USA. For all websites outside the UK (146), formal permission was sought from the website owners, resulting in 61 licenses to archive and make the archived copies publicly available. We received no response from 83 website owners, and 2 organisations agreed in principle to inclusion in the web archive but were not in a position to sign the license agreement required to allow us to archive the website as they could not affirm that they controlled the copyright of all the content that was to be archived. This meant an overall permissions rate of 42%, with the rate for websites in the .ie domain being even higher, at 51%.

Since the project was completed there have been many helpful reminders of the impact that such work has. This included one organisation that had created a website dedicated to an Easter Rising project which was no longer live on the web. The person that was responsible for the website had left the organisation and their replacement had no access to the materials that had been on the website. They had discovered an e-mail from me back in 2016 inviting them to participate in the web archive. Once they contacted me, I was able to direct them to the UK web archive and, as the organisation had signed the license agreement, they were able to access the archived website immediately from their office. This access had saved them both the time and staff resources that would have been expended in order to recreate some of the resources that were available on the archived website. It serves as an example of what embedding sustainability into a project can save in terms of time and staff resources and demonstrated the positive economic impact that organisations can derive by participation in cultural heritage initiatives such as web archives.

The co-curators of this collection have also previously published a paper on the collection in the academic journal, Internet Histories called Capturing commemoration: the 1916 Easter Rising web archive project.

You can watch Brendan Power’s presentation on the EWA YouTube Channel.