UK Web Archive blog

Information from the team at the UK Web Archive, the Library's premier resource of archived UK websites

The UK Web Archive, the Library's premier resource of archived UK websites

3 posts categorized "eResources"

21 September 2023

How YouTube is helping to drive UK Web Archive nominations

By Carlos Lelkes-Rarugal, Assistant Web Archivist, British Library

Screenshot of the UK Web Archive website 'Save a UK website' page.
https://www.webarchive.org.uk/nominate

There currently exists a plethora of digital platforms for all manner of online published works; YouTube itself has become more than just a platform for sharing videos, it has evolved into a platform for individuals and organisations to reach a global audience and convey powerful messages. Recently, a popular content creator on YouTube, Tom Scott, produced a short video helping to outline the purpose of Legal Deposit and by extension, the work being carried out by UKWA.

Watch the video here: https://www.youtube.com/watch?v=ZNVuIU6UUiM

Tom Scott’s video, titled "This library has every book ever published", is a concise and authentic glimpse into the work being done by the British Library, one of the six UK Legal Deposit Libraries. The video highlighted some of the technology being used that enables preservation at scale, which also highlighted the current efforts in web archiving. Dr Linda Arnold-Stratford (Head of Liaison and Governance for the Legal Deposit Libraries) stated, “The Library collection is around 170 million items. The vast majority of that is Legal Deposit”. Ian Cooke (Head of Contemporary British and Irish Publications) highlighted that with the expansion of Legal Deposit to include born-digital content that “the UK Web Archive has actually become one of the largest parts of the collection. Billions of files, about one and a half terabytes of data”.

At the time of writing, the video has had over 1.4 million views. In addition, as the video continued to gain momentum, something remarkable happened. UKWA started receiving an influx of email nominations from website owners and members of the public. This was unexpected and the volume of nominations that have since come through has been impressive and unprecedented. 

The video has led to increased engagement with the public; with nominations representing an eclectic mix of websites. The comments on the video have been truly positive. We are grateful to Tom for highlighting our work, but we are also thankful and humbled that so many commentators have left encouraging messages, which are a joy to read. The British Library has the largest web archive team of all the Legal Deposit Libraries, but this is still a small team of three curators and four technical experts where we do everything in-house from curation to the technical side. Web archiving is a difficult task but we are hopeful that we can continue to develop the web archive by strengthening our ties to the community by bringing together our collective knowledge.

If you know of a UK website that should be included in the archive, please nominate it here:  https://www.webarchive.org.uk/en/ukwa/info/nominate

12 July 2023

UK Web Archive Technical Update - Summer 2023

By Andy Jackson, Web Archive Technical Lead, British Library

This is a summary of what’s been going on since the 2023 Q1 report.

At the end of the last quarter, we launched the 2023 Domain Crawl. This started well (as described in the 2023 Q1 report) but a few days later it became clear the crawl was going a bit too well. We were collecting so quickly, we started to run out of space on the temporary store we use as a buffer for incoming content.

The full story of how we responded to this situation is quite complicated, so I wrote up the detailed analysis in a separate blog post. But in short, we took the opportunity to move to a faster transfer process and switch to a widely-used open source tool called Rclone. After about a week of downtime, the crawl was up and running again, and we were able to keep up and store and index all the new WARC files as they come in.

Since then, the crawl has been running pretty well, but there have been some problems…

2023-07-05-dc-storage-and-queues
2023 Domain Crawl Storage and Queues

The crawler uses disk space in two main ways: the database of queues of URLs to visit (a.k.a. the crawl frontier), and the results of the crawl (the WARCs and logs). The work with Rclone helped us get the latter under control, with the move from /mnt/gluster/dc2023 to sharing the main /opt drive and uploading directly to Hadoop. These uploads run daily, leading to a saw-tooth pattern as free space gets rapidly released before being slowly re-consumed.

But the frontier shares the same disk space, and can grow very large during a crawl. So it’s important we keep an eye on things to make sure we don’t run out of space. In the past, before we made some changes to Heritrix itself, it was possible for a domain crawl to consume huge amounts of disk space. Once, we hit over 100TB for the frontier, which becomes very difficult to manage. In recent domain crawls, our configuration changes we’ve managed to get this down to more like 10TB.

But, as you can see, around the 13th of June, we hit some kind of problem, where the apparent number of queues in the frontier started rapidly increasing, as did the rate at which we were consuming disk space. We deleted some crawler checkpoints to recover some space, as we very rarely need to restart the crawl from anything other than the most recent daily checkpoint, but this only freed-up modest amounts of space. Fortunately, the aggressive frontier growth seemed to subside before we ran out of space, and the crawl is now stable again.

Unfortunately, it’s not clear what happened. Based on previous crawls, it seems unlikely that the crawler suddenly discovered many more millions of web hosts at this point in the crawl. In the past, the number of queues has been consistently up to around 20 million at most, so this leap to over 30 million is surprising. It is possible we hit some weird web structures, but it’s difficult to tell as we do don’t yet have reliable tools for quickly analysing what’s going on in this situation.

Suspiciously, just prior to this problem, we resolved a different issue with the system used to record what URLs had been seen already. This had been accidentally starved of resources, causing problems when the crawler was trying to record what URLs had been seen. This lead to the gaps in the crawl monitoring data just prior to the frontier growth, as the system stopped working and required some reconfiguration. It’s possible this problem left the crawler in a bit of a confused state, leading to mis-management of the frontier database. Some analysis of the crawl will be needed to work out what happened.

In the laster quarter, the new URL search feature was deployed on our BETA service. Following favourable feedback on the new feature, the main https://www.webarchive.org.uk/ service has been updated to match. We hope you find the direct URL search useful.

We’ve also updated the code that recognises whether a visitor is in a Legal Deposit reading room, as it wasn’t correctly identifying readers at Cambridge University Library. Finally, there was an issue with how the CAPTCHAs on the contact and nomination forms were being validated, which has also been resolved.

Our colleagues from Webrecorder delivered the initial set of changes to the ePub renderer, making it easier to cite a paragraph of one of our Legal Deposit eBooks. Given how long the ePub format has been around, it is perhaps surprising that support for ‘obvious’ features like citations and printing are still quite immature, inconsistent and poorly-standardised. To make citation possible, we have ended up adopting the same approach as Calibre’s Reference Mode and implemented a web-based version that integrates with out access system.

We’ve also worked on updating the service documentation based on feedback from our Legal Deposit Library partners, resolved some problems with how the single-concurrent-use locks were being handled and managed, and implemented most of the translations for the Welsh language service. The translations should be complete shortly, and and updated service can be rolled out, including the second set of changes from Webrecorder (focused on searching the text of ePub documents).

Replication to NLS

The long process of establishing a replica of our holdings at the National Library of Scotland (NLS) is finally nearing completion. We have an up-to-date replica, and have been attempting to arrange the transfer of the servers. This turned out to be a bit more complicated that we expected, so has been delayed, but should be completed in the next few weeks.

Minor Updates

For curators, one small but important fix was improving how the W3ACT curation tool validates URLs. This was thought to have been fixed already, but the W3ACT software was not using URL validation consistently and this meant it was still blocking the creation of crawl target records with top-level domains like .sport (rather than the more familiar .uk or .com etc.). As of June 23rd, we released version 2.3.5 of W3ACT that should finally resolve this issue.

Apart from that, we also updated Apache Airflow to version 2.5.3, and leveraged our existing Prometheus monitoring system to send alerts if any of our SSL certificates are about to expire.

30 September 2022

Celebrating Sporting Heritage Day 2022

By Helena Byrne, Curator of Web Archives, The British Library

NSHD-Facebook-Banner-Sport-Icons-2.jpg-564x339

This blog post gives an overview of our sports related activities for the year to celebrate Sporting Heritage Day 2022 

2022 has been, and continues to be, a really busy year for international sport especially in the UK. The Winter Olympics in Beijing and the Commonwealth Games in Birmingham were  always scheduled to take place in 2022 years in advance. But as the Covid-19 pandemic caused disruption to many events in 2020 and 2021 many sporting events were postponed. The UEFA Women's Euros and the Rugby League World Cup, both hosted by England, were moved from 2021 to 2022, meaning that 2022 was even busier than normal in terms of major sporting events.

Sports has always been an Important part of the UK Web Archive so 2022 has been a busy year for us so far. Since 2017, sports has been grouped into three separate collections. 

Sports Collection - https://www.webarchive.org.uk/en/ukwa/collection/1768 

Sports: Football - https://www.webarchive.org.uk/en/ukwa/collection/1490 

Sports: International Events - https://www.webarchive.org.uk/en/ukwa/collection/2315 

The UK Web Archive regularly publishes blog posts about sport, which can be found here: https://blogs.bl.uk/webarchive/sports/

2022 Winter Olympics and Paralympics

As members of the International Internet Preservation Consortium (IIPC) both the British Library and the National Library of Scotland contributed to the IIPC Content Development Group (CDG) 2022 Winter Olympics and Paralympics collection.

The Olympics took place in Beijing from 4 to 20 February 2022, while the Paralympics were also in Beijing from 4 to 13 March 2022. 

The collection archived 863 items which included whole websites, subsections or individual pages from websites. These items are from 38 countries and 24 different languages are represented in the collection. Topics covered both events on and off the sporting field.

Browse the collection here:

https://archive-it.org/collections/18422 

UEFA Women’s Euro England 2022

The UEFA Women's Euro 2022 competition took place across England from July 6 to July 31, 2022. Although the event is over we are still collecting websites about the Euros from around the UK till the end of October. 

This collection covers both the sporting and cultural achievements of the event. There are over 275 items in the UEFA Women’s Euro England 2022 collection.

So far we have published seven blog posts about the Women’s Euros and there are still more to come. They can be found on the UK Web Archive blog with the sports tag here:

https://blogs.bl.uk/webarchive/sports/ 

Browse the collection here: https://www.webarchive.org.uk/en/ukwa/collection/4278

Commonwealth Games Birmingham 2022

Commonwealth Games Birmingham 2022 ran from 28 July to 8 August. Although the sporting events are over the cultural programme is continuing for a number of weeks. This means that UKWA still has an open call for nominations for this collection.

The collection covers both the sporting and cultural achievements as well as the social impact of this mega event. So far there are 434 items in the Commonwealth Games Birmingham 2022 collection.

Browse the collection here: https://www.webarchive.org.uk/en/ukwa/collection/4228 

Rugby League World Cup 2021

The Rugby League World Cup 2021 will take place from 15 October to 19 November 2022 across England. 

This event is unique in that the men's, women's and wheelchair competition all take place alongside each other. You can nominate your UK published Rugby League World Cup content here: https://www.webarchive.org.uk/nominate 

Updates on this collection will be published on the UK Web Archive blog and Twitter account

When published this collection will sit as a subsection of the Sports: International Events collection on the UKWA Topics & Themes page and will be available here: https://www.webarchive.org.uk/en/ukwa/collection/2315 

Access to the collections 

All of the archived content in the IIPC CDG 2022 Winter Olympics and Paralympics collection is open access. CDG collaborative collections are archived using the Archive-It platform meaning that all archived content is open access, although a publisher may  request its removal under the Internet Archives’ general terms and conditions

All CDG collections can be viewed here: https://archive-it.org/home/IIPC 

UK Web Archive Content has a mix of on-site and remote access due to the Non-Print Legal Deposit Regulations implemented in 2013. The full manifest of  content selected for UK Web Archive collections is visible on the website but access to individual archived websites depends on permission being granted by website publishers.  A note under each title informs users whether they can view the archived website online or whether they need to visit a UK Legal Deposit Library to view the archived content. 

All curated collections can be found on the Topics and Themes page of the UK Web Archive website: https://www.webarchive.org.uk/en/ukwa/category 

Get involved

The UK Web Archive is a partnership of the six UK legal Deposit Libraries and works with other external partners in order to expand  our subject expertise. We can’t curate the whole of the UK web on our own, however - we need your help to ensure that information, discussions and creative output related to sports is preserved for future generations.

Anyone can suggest UK published websites to be included in the UK Web Archive by filling in our nomination form: https://www.webarchive.org.uk/nominate