UK Web Archive blog

Information from the team at the UK Web Archive, the Library's premier resource of archived UK websites

The UK Web Archive, the Library's premier resource of archived UK websites

2 posts from May 2020

29 May 2020

Using Webrecorder to archive UK political party leaders' social media after the UK General Election 2019

This blog post is is by Nicola Bingham, Helena Byrne, Carlos Lelkes-Rarugal and Giulia Carla Rossi

Introduction to Webrecorder

The UK Web Archive aims to capture the whole of the UK web space at least once a year, and targeted websites at more frequent intervals. We conduct this activity under the auspices of the Legal Deposit Regulations 2013 which enable us to capture, preserve and make accessible the UK Web for the benefit of researchers now and in the future.

Along with many cultural and heritage institutions that perform at-scale web archiving, we use Heritrix 3, the state of the art crawler developed by the Internet Archive and maintained and improved by an international community of web archiving technologists.

Heritrix copes very well with large scale, bulk crawling but is not optimised for high fidelity crawling of dynamic content, and in particular does not archive social media content very well.

Researchers are increasingly turning their attention to social media as a significant witness to our times, therefore we have a requirement to capture this content, in certain circumstances and in line with our collection development policy. Usually this will be around public events such as General Elections where much of the campaigning over recent years has been played out online and increasingly on social media in particular. 

For this reason we have looked at alternative web archiving tools such as Webrecorder to complement our existing toolset. 

Webrecorder was developed by Ilya Kreymer under the auspices of Rhizome (a non-profit organisation based in New York which commissions, presents and preserves digital art), under its digital preservation program. It offers a browser based version, which offers free accounts up to 5GB storage and a Desktop App

Webrecorder was already well known to us at the UK Web Archive although we had not used it until recently. It is a web archiving service which creates an interactive copy of web pages that the user explores in their browser including content revealed by interactions such as playing video and audio, scrolling, clicking buttons etc. This is a much more sophisticated method of acquisition than that used by Hertrix which essentially only follows HTML links and doesn’t handle dynamic content very well. 

What we planned to do

The UK General Election Campaign ran from the 6th of November 2019 when Parliament was dissolved, until polling day on the 12th of December 2019. On the 13th of December 2019 the UK Web Archive team, based at the British Library attempted to archive various social media accounts of the main political party leaders. Seventeen political leaders from the four home nations were identified and a selection of three social media accounts were targeted: Twitter, Facebook and Instagram. Not all leaders have accounts on all three platforms, but in total forty four social media accounts were archived. These accounts are identified in the table below by an X. 

List of UK political political part leaders' social media accounts archived
Image credit: Carlos Lelkes-Rarugal



How we did it

On the 13th of December, 2019 we ran the Webrecorder Desktop App across twelve office PCs. Many were running the Webrecorder Autopilot function over the accounts, but we had mixed success, in that not all accounts captured the same amount of data. As the Autopilot functionality didn’t work well on all accounts, a combination of automated and manual capture processes were used where necessary. It took the team a lot longer than expected to archive the accounts therefore some were archived on a range of dates the following week.    


Large political party’s vs smaller party’s social media accounts

The two largest political party leaders, Jeremy Corbyn and Boris Johnson, have many more social media followers than the other home nations party leaders. This meant that it was more difficult to get a comprehensive capture of Corbyn and Johnson’s Twitter accounts than, for example, Arlene Foster’s. The more popular Twitter accounts took many hours to crawl; Corbyn’s took almost ten hours to archive thirteen day’s worth of Tweets (which only took us up to 1st December). 


Technical Issues

We experienced several technical issues with crawling, mainly concerned with issues around  IP addresses, the app crashing, and Autopilot working on some computers and not others. It was hard to get the app restarted after it crashed, so some time was lost when this happened. Different computers with the same specs ran differently. The Autopilot capture for Jeremy Corbyn’s and Boris Johnson’s Twitter accounts were started at the same time but Corbyn’s ran uninterrupted while Johnson’s crashed when it reached 475 MB. Although Corbyn’s account was crawled for nearly ten hours it only collected 93 MB of data. In contrast, Nigel Farage’s Twitter page was crawled for over four hours and only produced 506 MB. It is important to check the size of crawled data, as the hours the Webrecorder Desktop App is running on Autopilot does not necessarily translate into a high fidelity crawl. 


Added complications when using multiple devices with the same user profile:

Complications arose mainly from the auditing and collating of WARC files; performing QA and keeping track of which jobs were successful and those that were not. 

Initially, all participants in this project had planned to use their own work PC or work laptop and a local desktop installation of Webrecorder. However, an hour or so into the process(early in the day), it soon became apparent that there would not be enough time to archive all of the social media accounts within our time frame, given the volume of social media accounts and the unanticipated time it would take to archive each one. For example, it took one instance of a desktop Webrecorder application almost ten hours to archive Jeremy Corbyn’s Twitter account (only able to capture Tweets up to a month prior to the day of archiving).

It was then decided that we could potentially, and experimentally, run multiple parallel Webrecorder applications across a number of office desktop PCs; PCs that were free and available for us to use. This was possible because of the IT Architecture in place, allowing users to log into any office machine with the correct credentials and making their personal desktop load up along with all their files and user settings, regardless of the PC they log into. 

The British Library’s IT system, which incorporates a lot of the Windows ecosystem, gives each user their own dedicated central work directory where they are given a virtual hard drive and  their own storage space for all their documents and any other work related files. This allowed one user to be logged into several office PCs at the same time and therefore run a separate desktop Webrecorder application running on each machine. This was indeed very helpful as it allowed each machine to focus on one particular social media account, which in many cases took hours to archive. 

Having multiple Webrecorder jobs greatly increased our capacity to archive by removing the previous bottleneck, that was, one webrecorder job per user. Instead, this was increased to several webrecorder jobs per user.

Work flow of gathering WARC files from Webrecorder
Image credit: Carlos Lelkes-Rarugal



Having multiple Webrecorder jobs added complications down the line, not necessarily impacting the archiving process, but rather, complicating the auditing and collating of WARC files. When a user had several Webrecorder jobs running concurrently, each job would still be downloading to the same user work directory (the user’s virtual hard drive). So if a user had many parallel jobs running, this would create multiple WARC files in the same folder (but with different names, so no clashes), WARC files being produced by the different desktop PC that the user had logged in to. This was quite an elaborate setup because once a job had completed, the entire contents of the Webrecrder folder (where the WARCs were stored) was copied to a USB so that an initial Quality Assurance (QA) could be performed on the completed job on a more capable laptop. The difficulty was in finding the WARC file that corresponded to the completed job, which was somewhat convoluted as there would have been multiple WARC files with this type of file-naming convention:


As you can imagine, taking a copy of Webrecorder’s folder contents not only has the completed job, but also the instances of other WARC files from other incomplete jobs. Coupled with multiple jobs per PC, and multiple PCs per user; keeping track of what had completed and which WARCs were either corrupted or not up to standard, was quite demanding. 


Review of the data collected 

File size of data collected from UK political party leaders' social media accounts
Image credit: Carlos Lelkes-Rarugal


How to access this data

The archived social media accounts can be accessed through the UK General Election 2019 collection in a UK Legal Deposit Library Reading Room. The UK Legal Deposit Libraries are the British Library, National Library of Scotland, National Library of Wales, Bodleian Libraries, Cambridge University Library and Trinity College Dublin Library.  

The 2019 collection is part of a time series of UK General Elections dating from 2005. They can be accessed over the Internet on the Topics and Themes page of the UK Web Archive website. All the party leaders' social media accounts are tagged into the subsection UK Party Leaders Social Media Accounts (access to individual websites depends on whether we have an additional permission to allow ‘open’ access). More information about what is included in the UK General Election 2019 collection is available through the UK Web Archive blog



Overall, undertaking this experiment was an interesting experience for our small team of British Library Web Archive Curators. Many valuable lessons were learnt on how best to utilise Webrecorder in our current practice. The major takeaway was that it was a lot more time consuming than we expected. Instead of taking up one working day, it took nearly a whole week to archive our targeted social media accounts with Webrecorder. Our usual practise is to archive social media accounts with the Heritrix crawler, which works reasonably well with Twitter but is less suited to capturing other platforms. For a long time, we were unable to capture any Facebook content with Heritrix, mainly due to the platform’s publishing model, however the way the platform is published has changed recently allowing us limited success. Archiving social media will always remain challenging for the UK Web Archive, for myriad technical, ethical and legal reasons. The sheer scale of the UK’s social media output is too large for us to capture adequately (and indeed, this may not even be desirable) and certainly too large a task for us to tackle with manual, high fidelity tools such as Webrecorder. However, our recent experience during the 2019 UK General Election has convinced us that using Webrecorder to capture significant events is a worthwhile exercise, as long as we target selected, in scope accounts on a case by case basis. 


27 May 2020

Web Archiving the UK General Election 2019

By Jennie Grimshaw, Curator for Official Publications, The British Library

The 2019 general election was a turning point in British political history.  It saw the resurgence of the Conservative vote from 8.8% at the May 2019 European Parliament election to 23.6% eight months later. It saw the collapse of the “Red Wall” in the North and the Midlands as seats such as Sedgfield,  Labour since 1935, and Workington, Labour 1918-1976 and 1979-2019, turned Conservative. It saw the breaking of the Parliamentary deadlock over Brexit with the return to power of pro-Leave Conservative Prime Minister Boris Johnson, with a majority of 80, which then enabled him to “Get Brexit Done”

Polling station sign

To help researchers trace how use of the Internet for political campaigning and communication has evolved over time, the British Library, the National Libraries of Scotland and Wales, and the Bodleian Library have collaborated to create a web archive collection for all UK general elections since 2005, using more or less the same categories – candidates web presence, national and local political party websites, online news and commentary, interest group manifestos and comment and analysis by think tanks. This collection is the fifth in the time series and is complemented by the EU Referendum and Brexit collections. In 2019 Northern Ireland political party and candidate sites were selected by the Public Record Office of Northern Ireland (PRONI).

Sadly, our ability to harvest social media sites is very limited due to technical and legal issues. We can gather Twitter feeds, but not Facebook pages as the site deliberately blocks the crawls and it has proved impossible to negotiate access.  Due to the limitations of our crawl software, we cannot always gather dynamic content, Wix-based sites, documents stored in the cloud or videos. This may explain why some worthy candidate sites you might expect to see have not been archived.

However, the UK Web Archive General Election 2019 does offers researchers a respectable total of 2237 sites, including:

  • UK–based news and comment sites , such as Politics Home, Political UK, the Commentator, Unherd, Reaction, CAPX and  CAPX 2019 Election Archive as well as Twitter feeds of selected  political journalists. We include satirical sites such as the spoof Conservative Manifesto created by a company called Concerned Citizens Ltd and the Daily Reckless, which offers satirical songs by Tommy Mackay.
  • Candidates campaign websites and Twitter feeds and local constituency party sites. The collection includes all candidates standing in Scottish and Welsh constituencies. Due to staff resource limitations, we can only capture a sample of the websites and Twitter feeds of candidates standing in English constituencies. We cover three inner London and three outer London boroughs, and one rural and one urban constituency from each of the English regions.  We have used the same English constituencies for every election since 2005.
  • Websites and Twitter feeds of 100 national political parties, ranging from fringe groups such as the Animal Welfare Party, Arthur Horner of the Welsh Communist Party, Britain First and the anarchist Class War Party to the major national parties (Conservative, Labour, Liberal Democrats, and Greens) and political parties in the devolved administrations (Plaid Cymru, Scottish Nationalists and the Northern Ireland parties). We also capture major political party blogs such as Conservative Home, Conservative Woman, Labour List and Liberal Democrat Voice.
  • Social media sites of the main party leaders captured in depth using Web Recorder. We can use this software only very sparingly as operating it is very resource and time intensive, but it can capture sites our regular crawler cannot reach.
  • Interest groups, seeking to influence party policies through engagement with candidates and publication of manifestos and lists of “asks”.  We have selected about 340 sites, ranging from the manifestos of campaigning charities such as Age UK and trade associations such as Airlines UK and the Association of the British Pharmaceutical Industry to unions, religious groups, such as the Evangelical Alliance and Muslim Council of Britain, and pressure groups such as the Campaign to Protect Rural England, Actionaid and Anti-Slavery International. Health charities and environmental groups are particularly prominent at this election and professional associations such as the medical royal colleges are also well represented. The voices of disabled people and minority groups are also heard through manifestos and comment from Leonard Cheshire Disability, SCOPE, Disability Rights UK, MENCAP, RNIB, Operation Black Vote and the Muslim Public Affair’s Committee’s Operation Muslim Vote 2019.
  • Thank tanks and academic research centres providing in-depth comment and analysis. We have sought to include both right- and left-wing views, and comment from political, legal and economic viewpoints. Targets include the Centre for Labour and Social Studies (CLASS), the London School of Economics British Politics and Policy blog, the Centre for Constitutional Change, Demos’ Manifesto for consensus politics, the Democratic Audit blog, the British Future think tank, Full Fact, the Institute for Fiscal Studies the King’s Fund and the Institute for Government, etc.

We hope that this collection will preserve the voices and illustrate the concerns and priorities of a wide spectrum of UK society and help to show how political parties and candidates engaged and  responded at this pivotal moment of UK history.

General Election 2019 collection. Note that you can view what is in this collection but many of the actual websites can only be viewed in the reading room of a UK Legal Deposit Library.