UK Web Archive blog

Information from the team at the UK Web Archive, the Library's premier resource of archived UK websites

The UK Web Archive, the Library's premier resource of archived UK websites


News and views from the British Library’s web archiving team and guests. Posts about the public UK Web Archive, and since April 2013, about web archiving as part as non-print legal deposit. Editor-in-chief: Jason Webber. Read more

17 September 2020

Arnhem75 - a special collection of websites added to the UK Web Archive


By Marja Kingma, Curator of Germanic Collections, the British Library.


Arnhem75 blog image
Book cover of 75 Years Battle of Arnhem by Laurens van Aggelen



The idea to create a collection of websites about the commemoration of Arnhem75 came to RAF Museum historian Harry Raffal and myself whilst attending the seminar ‘The Arnhem Spirit - 75 years of Brits in Arnhem’, on 15 May 2019, organised by the Dutch Embassy in London. The event was part of a programme in which the Netherlands, Britain and other former Allied countries commemorated Operation Market Garden, the code name for the battle for the bridge across the Rhine at Arnhem that took place in September 1944. Allied forces consisted of British, American and Polish troops, with help from Dutch resistance.

The Battle of Arnhem 1944 is of great significance to the UK and interest in it remains strong on both sides of the North Sea.

We wanted to create a lasting memory of these events and a special collection in the UK Web Archive on the subject seemed like a good idea.


What is included?

We kept the scope of the project quite narrow; only websites with a focus on the commemorations that took place in Britain and the Netherlands in 2019 are included, with the exception of some websites that deal with the historic facts regarding the Battle to give it some context.

So far over 150 individual websites within the UK web domain have been identified, of which 64 were selected to go into the collection. These sites are limited to the UK web domain, so have .uk in their domain name, or if they don’t must be hosted in the UK, or owned by UK organisations or individuals with a postal address in the UK.


How can you access these archived websites?

Under the Non-Print Legal Deposit Regulations 2013, we can archive UK websites but we are only able to make them available to people outside the UK Legal Deposit Libraries reading rooms, if the website owner has given permission. The UK Legal Deposit Libraries are the British Library, National Library of Scotland, National Library of Wales, Bodleian Libraries, Cambridge University Library and Trinity College Dublin Library.

For this collection you can view what has been selected through the UK Web Archive website but will need to visit a UK Legal Deposit Library reading room to view the archived content. The reading rooms across the Legal Deposit Libraries are starting to reopen now, with some restrictions, as you can read in this blog:


How Can I Get Involved?

You can help expand this collection by sending us a URL you think may be eligible for inclusion in the collection Arnhem75. Please go to to nominate a website and we’ll take it from there.

Occasionally websites from non UK domains can be included, if they have a strong link to the UK and the website owners have given their permission to be included in the collection. Dutch organisations that were involved in the Arnhem75 commemorations are encouraged to get in touch.

We look forward to your suggestions!


10 September 2020

Launching the UK Web Archive 2020 Annual Domain Crawl

By Helena Byrne, Curator of Web Archives at the British Library

Today (10th September 2020) the UK Web Archive team will be pushing the big red button to kickstart the annual Domain Crawl of the UK webspace. The current coronavirus pandemic will no doubt feature strongly in this year’s crawl. This will complement the curated collection that the web archive teams across the UK Legal Deposit Libraries are contributing. The British Library along with the National Library of Scotland are also selecting websites for the International Internet Preservation Consortium (IIPC) Content Development Group (CDG) Novel Coronavirus (COVID-19) collection. 

What we collect

The UK Web Archive has been archiving UK published websites on a selective basis since 2005 and in 2020 is celebrating #15YearsOfUKWA. Domain Crawl 2020 is the seventh that has taken place. It wasn’t till after the implementation of the Non-Print Legal Deposit Regulations (NPLD) in April 2013, that we were able to run a broad crawl over the UK webspace. This includes anything with a .uk or other UK geographic Top Level Domain (TLD) such as .scot, .cymru or .london etc. It also includes websites on other TLDs that have been registered in the UK or that have been manually selected. 

NPLD came into effect on the 6th April 2013 and the British Library hosted a special event to launch the first Domain Crawl. This was widely covered in the national press and you can still watch back a short video from the event on The Guardian website

How much data is collected in the Domain Crawl?

The Domain Crawl usually runs for three months of the year and each year starts at a different time of year to avoid seasonal biases. Roughly 5-10 million hosts (websites) are archived every year. However, the amount of data collected each year varies. Also the way the data is collected and stored over time changes. We compress the data we store and as technology develops the amount of data that can be compressed into one terabyte changes. Last year 63.7 TB of compressed data was collected bringing the total collected during Domain Crawls from 2013 to 2019 to 477.62 TB. 

UKWA Domain Crawl 2013-2019 (1)

When can I view this content?

Due to the enormous amounts of data that is collected each year from the annual Domain Crawl and our Frequent Crawls, there is a significant lag from when the content is archived and made available through the UK Web Archive website. The Frequent Crawl data collected from 2013-2019 was 250.34 TB bringing the combined total to 727.96 TB of compressed data. To make searching content easier the website allows you search across all the Selectively Crawled content from 2005 to 2013 as well as the Frequent Crawl content from 2013 to 2017 and the Domain Crawl content 2013 to 2015. 

Under the Non-Print Legal Deposit (NPLD) Regulations 2013, we can archive all UK published websites but we are only able to make them available to people outside the Legal Deposit Libraries Reading Rooms, if the website owner has given permission.

Due to the NPLD Regulations, access to the archived content is a mix of open and onsite access. The ‘Viewable only on Library premises’ message on individual records indicates that you have to visit one of the six UK Legal Deposit Libraries.  The UK Legal Deposit Libraries are the British Library, National Library of Scotland, National Library of Wales, Bodleian Libraries, Cambridge University Library and Trinity College Dublin Library.

Follow the UK Web Archive on Twitter for the latest updates on the domain crawl and other web archiving activities! 

08 September 2020

UKWA available in reading rooms again

By Jason Webber, Web Archive Engagement Manager, The British Library

Much of the UK Web Archive content is only available in the reading rooms of UK Legal Deposit Libraries as current legislation regulates access. All libraries were closed for many months during the COVID-19 lockdown, however, a phased reopening has begun. 

Below is some basic information of what current access is available at Legal Deposit Libraries with links to more detail. Note opening times were correct at the time of publishing this article, library websites should be checked for current opening times.

British Library reading room

British Library

London, St Pancras

Tuesday – Saturday 11.00 – 15.00

Boston Spa

Tuesday – Friday 11.00 – 15.00

You’ll need to pre-book online for whatever you would like to see at the Library. At the moment you can pre-book:

National Library of Scotland

Edinburgh reading rooms

Our Edinburgh reading rooms have reopened to existing and new library card holders, on a pre-booked basis only, with revised opening hours. Readers must book and preorder items 24 hours in advance.

General Reading Room and Special Collections Reading Room:

Tuesday-Saturday, 10.00-16.00

Kelvin Hall

We anticipate that the Library at Kelvin Hall in Glasgow, will reopen around mid-September.

National Library of Wales

The Reading Room is open to the public with a restricted service. You will have to book your place online before your visit. For more details on this and to read strict guidelines regarding the nature of the restricted service and what is expected of you go to Guidelines on re-opening.

(Reading Rooms only)

Monday - Friday: 10:00-12.30 and 13.30-16.00

Saturday: Closed

Bodleian Library

The Bodleian Libraries have begun a phased reopening to staff, students and Bodleian Reader Card holders.

To help us keep you safe, and make sure we follow government and University guidelines, you'll need to book your visit in advance.

Weston Library (and several others)

Monday - Friday 1000-1600

Cambridge University Libraries

Cambridge University Library is now open for limited services from Monday-Friday. Book a visit to view non-borrowable material in the Main reading room or a Special Collections reading room. Please read more about our phased reopening of the UL and Faculty and Departmental Libraries.

Monday-Friday 10:15 -15:45 for limited services.

Trinity College, Dublin

Library reading rooms are now open for current staff and students. Face coverings are required. "Click and Collect" items will now be delivered to Library buildings. Goldsmith Hall is no longer used for collections or returns.

Monday-Friday 0930-1700