UK Web Archive blog

22 posts categorized "Collections"

19 February 2015

Building a 'Historical Search Engine' is no easy thing

Add comment Comments (0)

Over the last year the UK Web Archive has been part of the Big UK Domain Data for the Arts and Humanities project, with the ambitious goal of building a ‘historical search engine’ covering the early history of the UK web. This continues the work of the Analytical Access to the Domain Dark Archive project but at a greater scale, and moreover, with a much more challenging range of use cases. We presented the current prototype at the International Digital Curation Conference last week (written up by the DCC), and received largely positive feedback, at least in terms of how we have so far handled the scale of the collection.

What the researchers found
However, we are eagerly awaiting the results of the real test of this system, from the project’s bursary holders. Ten researchers have been funded as ‘expert users’ of the system, each with a genuine historical research question in mind. Their feedback will be critical in helping us understand the successes and failures of the system, and how it might be improved.

One of those bursary holders, Gareth Millward, has already talked about his experience, including this (somewhat mis-titled but otherwise excellent) Washington Post article “I tried to use the Internet to do historical research. It was nearly impossible.” Based on that, it seems like the results are something of a mixed bag (and from our informal conversations with the other bursary holders, we suspect that Gareth’s experiences are representative of the overall outcome). But digging deeper, it seems that this situation arises not simply because of problems with the technical solution, but because of conflicting expectations of how the search should behave.

For example, as Gareth states, if you search for RNIB using Google, the RNIB site and information about it is delivered right at the top of the results.

But does this reflect what our search engine should do?

Is a historical search engine like Google?
When Google ranks its results, it is making many assumptions. About the most important meanings of terms, the current needs of its users and the information interests of specific users (also known as the filter bubble). What assumptions should we make? Are we even playing the same game?

One of the most important things we have learned so far is that we are not playing the same game, and the information needs of our researchers might be very different to those of a normal search (and indeed different between different users). When a user searches for ‘iphone’, Google might guess that you care about the popular one, but perhaps a historian of technology might mean the late 1990’s Internet Phone by VocalTec. Terms change their meaning over time, and we must enable our researchers to discover and distinguish the different usages. As Gareth says “what is ‘relevant’ is completely in the eye of the beholder.”

Moreover, in a very fundamental way, the historians we have worked with are not searching for the one top document, or a small set of documents about a specific topic. They look to the web archive as a refracting lens onto the society that built it, and are using these documents as intermediaries, carrying messages from the past and about the past. In this sense, caring about the first few hits makes no sense. Every result is equally important.

How results are sorted
To help understand these whole sets of results, we have endeavoured to add appropriate filtering and sorting options that can be used to ‘slice and dice’ the data down into more manageable chunks. At the most basic level (and contrary to the Washington Post article), the results are sorted, and the default is to sort by ascending harvest date. The contrast with a normal search engine is perhaps no more stark than here – where BING or Google will generally seek to bring you the most recent hits, we focus on the past, something that is very difficult to achieve using a normal search engine.

With so many search options, perhaps the biggest challenge has been to present them to our users in a comprehensible way. For example, the problem where the RNIB advertisements for a talking watch were polluting the search results can be easily remedied if you combine the right search terms. The text of the advert is highly consistent, and therefore it is possible to precisely identify those advertisements by searching for the text “in associate with the RNIB”. This means it is possible to refine a search for RNIB to make sure we exclude those results (as you can see below).


The problems are even more marked when it comes to trying to allow network analysis to be exploited. We do already extract links from the documents, and so it is already possible to show how the number of sites linking to the RNIB has changed over time, but it is not yet clear how best to expose and utilize that information. At the moment, the best solution we have found is to present this network links as additional search facets. For example, here are the results for the sites that linked to in 2000, which you can contrast with those for 2010.

Refining searches further
Currently, we expect that refining a search on the web archive will involve a lot this kind of operation, combining new search terms and clauses to help focus in on the documents of interest. Therefore, looking further ahead, we envisage that future iterations of this kind of service might take the research queries and curatorial annotations we collect and start to try to use that information to semi-automatically classify resources and better predict user needs.

A ‘Macroscope’ rather than a search engine
Despite the fact that it helps get the overall idea across, calling this system a ‘historical search engine’ turns out to be rather misleading. The actual experience and ‘information needs’ of our researchers are very different from that case. This is why we tend to refer to this system as a Macroscope (see here for more on macroscopes), or as a Web Observatory. Sometimes a new tool needs a new term.

Throughout all of this, the most crucial part has been to find ways of working closely with our users, so we can all work together to understand what a ‘Macroscope’ might mean. We can build prototypes, and use our users’ feedback to guide us, but at the same time those researchers have had to learn how to approach such a complex, messy dataset.
Both the questions and the answers have changed over time, and all parties have had their expectations challenged. We look forward to continuing to build a better Macroscope, in partnership with that research community.

By Dr Andrew Jackson, Web Archiving Technical Lead, The British Library

28 January 2015

Spam as a very ephemeral (and annoying) genre…

Add comment Comments (0)

Spam is a part of modern life. Who hasn’t received any recently, is a lucky person indeed. But only try to put your email out there in the open and you’ll be blessed with endless messages you don’t want, from people you don’t know, from places you’ve never heard about! And then just delete, de-le-te, block sender command…

Imagine though someone researching our web lives in say 50 years and this part of our daily existence is nowhere to be found. Spam is the ugly sister of the Web Archive, it is unlikely we’ll keep spam messages in our inboxes, and almost certainly no institution will keep them for posterity. And yet they are such great research materials. They vary in topics, they can be funny, they can be dangerous (especially to your wallet), and they make you shake your head in disbelief…

We all know the spam emails about people who got stuck somewhere and they can’t pay the bill and ask for a modest sum of £2,500 or so. Theses always make me think: if I had spare £2,500, it’d be Bora Bora here I come, but that’s just selfish me! Now these are taken to a new level. It’s about giving us the money that is inconveniently placed in a bank somewhere far, far away:

Charity spree

From Mrs A.J., a widow of a Kuwait embassy worker in Ivory Coast with a very English surname:

…Currently, this money is still in the bank. Recently, my doctor told me I would not last for the next eight months due to cancer problem. What disturbs me most is my stroke sickness. Having known my condition I decided to donate this fund to a charity or the man or woman who will utilize this money the way I am going to instruct here godly.

Strangely two weeks a Libyan lady, who is also a widow, is writing to me that she also suffered a stroke and all she wants to shower me with money as part of her charity spree:

Having donated to several individuals and charity organization from our savings, I have decided to anonymously donate the last of our family savings to you. Irrespective of your previous financial status, please do accept this kind and peaceful offer on behalf of my beloved family.


Mr. P. N. ‘an accountant with the ministry of Energy and natural resources South Africa’ was straight to the point:

… presently we discovered the sum of 8.6 million British pounds sterling, floating in our suspense Account. This money as a matter of fact was an over invoiced Contract payment which has been approved for payment Since 2006, now we want to secretly transfer This money out for our personal use into an overseas Account if you will allow us to use your account to Receive this fund, we shall give you 30% for all your Effort and expenses you will incure if you agree to Help.

My favourite is quite light-hearted. Got it from a 32 year old Swedish girl:

My aim of writing you is for us to be friends, a distance friend and from there we can take it to the next level, I writing this with the purest of heart and I do hope that it will your attention. In terms of what I seek in a relationship, I'd like to find a balance of independence and true intimacy, two separate minds and identities forged by trust and open communication. If any of this strikes your fancy, do let me know...

So what I’m a girl too, with a husband and a kid? You never know what may be handy…

Blog post by Dorota Walker 
Assistant Web Archivist



Further reading: Spam emails received by Please note that the quotations come from the emails and I left the original spelling intact.

11 November 2014

Collecting First World War Websites – November 2014 update

Add comment Comments (0)

Earlier in 2014 we blogged about the new Special Collection of websites related to World War One that we’ve put together to mark the Centenary. As today is Armistice Day, commemorating the cessation of hostilities on the Western Front, it seems fitting to see what we have collected so far.


The collection has been growing steadily over the past few months and now totals 111 websites. A significant subset of the WW1 special collection comes from the output of the Heritage Lottery Funded projects. The collection also includes websites selected by subject specialists at the British Library and nominations from members of the public.

A wide variety of websites have been archived so far which can broadly be categorised into a few different types:

Critical reflections
They include critical reflections on British involvement in armed conflict more generally, for example the Arming All Sides website, which features a discussion of the Arms trade around WW1 and, an invaluable academic resource on the history of naval conflict in the First and Second World Wars.

Artistic and literary
The First World War inspired a wealth of artistic and literary output. For example the website dedicated to Eugene Burnand (1850-1921) a Swiss artist who created a series of pencil and pastel portraits depicting various ‘military types’ of all races and nationalities drawn into the conflict on all sides. Burnand was a man of great humanity and his subjects included typical men and women who served in the War as well as those of more significant military rank.

The Collection also includes websites of contemporary artists who in connection with the Centenary are creating work reflecting on the history of the conflict. One such artist is Dawn Cole whose work on WW1 has focused on the archive of WW1 VAD Nurse Clarice Spratling’s diaries, creating a project of live performance, spoken word and art installations.

Similar creative reflections from the world of theatre, film and radio can be seen in the archive. See for example Med Theatre: Dartmoor in WW1, an eighteen-month project investigating the effect the First World War had on Dartmoor and its communities. Pals for Life is a project based in the north-west aiming to create short films enabling local communities to learn about World War One. Subterranean Sepoys, is a radio play resulting from the work of volunteers researching the forgotten stories of Indian soldiers and their British Officers in the trenches of the Western Front in the first year of the Great War.

Community stories
The largest number of websites archived so far comprise projects produced by individuals or local groups telling stories of the War at a community level across the UK. The Bottesford Parish 1st World War Centenary Project focusses on 220 local recruits who served in the War using wartime biographies, memorabilia and memories still in the community to tell their stories.

The Wylye Valley 1914 project has been set up by a Wiltshire-based local history group researching the Great War and the sudden dramatic social and practical effects this had on the local population. In 1914 24,000 troops descended suddenly on the Wylye Valley villages, the largest of which had a population of 500, in response to Kitcheners’ appeals for recruits. These men arrived without uniform, accommodation or any experience of organisation. The project explores the effects of the War on these men and the impact on the local communities.

An important outcome of commemorations of the Centenary of WW1 has been the restoration and transcription of war memorials across the UK. Many local projects have used the opportunity to introduce the stories of those who were lost in the conflict. Examples include the Dover War Memorial Project; the Flintshire War Memorials Project ; Leicester City, County and Rutland War Memorials project and St. James Toxteth War memorials project.

Collecting continues
This shows just some of the many ways people are choosing to commemorate the First World War and demonstrates the continued fascination with it.

We will continue collecting First World War websites through the Centenary period to 2018 and beyond. If you own a website or know of a website about WW1 and would like to nominate it for archiving then we would love to hear from you. Please submit the details on our nominate form.

By Nicola Bingham, Web Archivist, The British Library

16 October 2014

What is still on the web after 10 years of archiving?

Add comment Comments (2)

The UK Web Archive started archiving web content towards the end of 2004 (e.g. The Hutton Enquiry). If we want to look back at the (almost) ten years that have passed since then, can we find a way to see how much we’ve achieved? Are the URLs we’ve archived still available on the live web? Or are they long since gone? If those URLs are still working, is the content the same as it was? How has our archival sliver of the web changed?

Looking Back
One option would be to go through our archives and exhaustively examine every single URL, and work out what has happened to it. However, the Open UK Web Archive contains many millions of archived resource, and even just checking their basic status would be very time-consuming, never mind performing any kind of comparison of the content of those pages.

Fortunately, to get a good idea of what has happened, we don’t need to visit every single item. Our full-text index categorizes our holdings by, among other things, the year in which the item was crawled. We can therefore use this facet of the search index to randomly sample a number of URLs from each year the archive has been in operation, and use those to build up a picture that compares those holdings to the current web.

URLs by the Thousand
Our search system has built-in support for randomizing the order of the results, so a simple script that performs a faceted search was all that was needed to build up a list of one thousand URLs for each year. A second script was used to attempt to re-download each of those URLs, and record the outcome of that process. Those results were then aggregated into an overall table showing how many URLs fell into each different class of outcome, versus crawl date, as shown below:


Here, ‘GONE’ means that not only is the URL missing, but the host that originally served that URL has disappeared from the web. ‘ERROR’, on the other hand, means that a server still responded to our request, but that our once-valid URL now causes the server to fail.

The next class, ‘MISSING’, ably illustrates the fate of the majority of our archived content - the server is there, and responds, but no longer recognizes that URL. Those early URLs have become 404 Not Found (either directly, or via redirects). The remaining two classes show URLs that end with a valid HTTP 200 OK response, either via redirects (‘MOVED’) or directly (‘OK’).

The horizontal axis shows the results over time, since late 2004, broken down by each quarter (i.e. 2004-4 is the fourth quarter of 2004). The overall trend clearly shows how the items we have archived have disappeared from the web, with individual URLs being forgotten as time passes. This is in contrast to the fairly stable baseline of ‘GONE’ web hosts, which reflects our policy of removing dead sites from the crawl schedules promptly.

Is OK okay?
However, so far, this only tells us what URLs are still active - the content of those resources could have changed completely. To explore this issue, we have to dig a little deeper by downloading the content and trying to compare what’s inside.

This is very hard to do in a way that is both automated and highly accurate, simply because there are currently no reliable methods for automatically determining when two resources carry the same meaning, despite being written in different words. So, we have to settle for something that is less accurate, but that can be done automatically.

The easy case is when the content is exactly the same – we can just record that the resources are identical at the binary level. If not, we extract whatever text we can from the archived and live URLs, and compare them to see how much the text has changed. To do this, we compute a fingerprint from the text contained in each resource, and then compare those to determine how similar the resources are. This technique has been used for many years in computer forensics applications, such as helping to identify ‘bad’ software, and here we adapt the approach in order to find similar web pages.

Specifically, we generate ssdeep ‘fuzzy hash’ fingerprints, and compare them in order to determine the degree of overlap in the textual content of the items. If the algorithm is able to find any similarity at all, we record the result as ‘SIMILAR’. Otherwise, we record that the items are ‘DISSIMILAR’.

Processing all of the ‘MOVED’ or ‘OK’ results in this way leads to this graph:


So, for all those ‘OK’ or ‘MOVED’ URLs, the vast majority appear to have changed. Very few are binary identical (‘SAME’), and while many of the others remain ‘SIMILAR’ at first, that fraction tails off as we go back in time.

Summarising Similarity
Combining the similarity data with the original graph, we can replace the ‘OK’ and ‘MOVED’ parts of the graph with the similarity results in order to see those trends in context:


Shown in this way, it is clear that very few archived resources are still available, unchanged, on the current web. Or, in other words, very few of our archived URLs are cool.

Local Vs Global Trends
While this analysis helps us understand the trends and value of our open archive, it’s not yet clear how much it tells us about other collections, or global trends. Historically, the UK Web Archive has focused on high-status sites and sites known to be at risk, and these selection criteria are likely to affect the overall trends. In particular, the very rapid loss of content observed here is likely due to the fact that so many of the sites we archive were known to be ‘at risk’ (such as the sites lost during the 2012 NHS reforms). We can partially address this by running the same kind of analysis over our broader, domain-scale collections. However, that would still bias things towards the UK, and it would be interesting to understand how these trends might differ across countries, and globally.

By Andy Jackson, Web Archiving Technical Lead, The British Library

07 October 2014

Thoughts on website selecting for the UK Web Archive

Add comment Comments (0)

Hedley Sutton, Asian & African Studies Reference Team Leader at The British Library gives his thoughts and experiences of web archiving.

A Reference Team Leader spends most of their day answering queries sent in by e-mail, fax and letter or manning Reading Room enquiry desks. Some, however, also help with contributing to the selection of sites for inclusion in the UK Web Archive.

The rise of digital
Digital content is of course increasingly important for researchers, and is certain to become ever more so as publishers slowly move away from print to online formats. The Library recognized this when it began to archive websites in 2004, aiming to harvest a segment of the vast national web domain by providing free access both to live sites and to snapshots of existing and defunct sites as they developed over time.

Those which have been fully ‘accessioned’, as it were, are available to view online, and can be found alphabetically by title, or subject/keyword, or in some cases grouped in themed collections such as the 2012 London Olympics or the ‘Credit crunch’. 

Websites of interest
I volunteered to become a selector in 2008, planning initially to concentrate on tracing websites within my own specialism of Asian and African studies. I soon discovered, however, that it was more rewarding (addictive, even) to look beyond conventional subject divisions to home in on all and anything that looked of potential interest to present and future users of the archive.

Worthy, unusual and not-quite-believe-it
Over the years this has ranged from the worthy (such as the UK Web Designers’ Association and the Centre for Freudian Analysis and Research), through the unusual (step forward the Federation of Holistic Therapists, the Fellowship of Christian Magicians, and the Society for the Assistance of Ladies in Reduced Circumstances), to the I-see-it-but-do-not-quite-believe-it (yes, I mean you, British Leafy Salads Association; no, don’t try and run away, Ferret Education and Research Trust; all power to you, Campaign Against Living Miserably). Being paid to spend part of your time surfing the web – what’s not to like?

Permission required
The only mildly disappointing aspect of selecting websites is the fact that at present only about 20% of recommended sites actually make it into the Open UK Web Archive. The explanation is simple – the Library requires formal permission from website owners before it can ingest and display their sites.

This is offset in part by the amendment to the Legal Deposit legislation that (since 2013) has allowed The British Library to archive all UK websites. These, however, can only be viewed in the Reading Rooms of the UK Legal Deposit Libraries.

If you know of a website that you feel should be in the Open UK Web Archive, please nominate it.

By Hedley Sutton - Asian & African Studies Reference Team Leader, The British Library

15 September 2014

Dot Scot: A new domain identity

Add comment Comments (0)

As all thoughts turn to Scotland and the Scottish Referendum which is taking place on the 18th of September it seems appropriate to highlight some recent developments in the digital sphere that will impact the Web Archiving Team over the coming months.

New top level domain (TLD) for Scotland
The Internet Corporation for Assigned Names and Numbers (ICANN) has released a suite of new top level domains (TLDs) this year. One of these is .scot (live since 15 July 2014) allowing organisations and individuals to create websites and email addresses identifying themselves as Scottish. The new TLD follows a near-decade long campaign by the Dot Scot Registry, a not-for-profit company created to apply for and operate the .scot domain as an online identity for Scots worldwide.

.scot is a community domain meaning anyone can apply for it, however for the first 60 days the domain was only available to launch ‘pioneers’, a cross section of organisations based in Scotland or part of the Scottish diaspora community. The first pioneer website to go live on 15th July was - a Highlands based Internet Service Provider who offer .scot domain registrations. Over 50 pioneers have signed up including the Scottish government, the Scouts in Scotland, Yes Scotland and Better Together.

Scotland beyond Britain
Individuals and groups outside of Scotland have also taken advantage of the new domain with the Louisiana Scots and the Clan Wallace among the first international organisations to launch websites using the new domain ahead of the general launch on 23rd September.

New-borns get a domain name
The Dot Scot Registry has come up with a unique idea to publicise the .scot TLD by reserving a few domain names for any Scottish baby born on 15 July for free. In a press release on their website, the organisation said: ‘It’s taken nine years to get to this point – and we want to celebrate this “birth” in as many ways as possible. So, if you know someone who had a baby in Scotland on 15 July 2014, contact our press team, and we’ll secure their .scot for them … It’s our little way of saying “welcome to the world and the digital future for Scotland.”’.

In the archive
The UK Web Archiving Team are already collecting .scot websites as part of our annual domain crawl along with .london websites, another of the TLDs released by ICANN this year. A phased release of the .cymru and .wales TLDs was launched this month by the UK internet registry, Nominet, with general availability due in March 2015. These websites will also be picked up by the British Library’s annual domain crawl.

Short lived?
One final point to make is that .scot might be superseded if the Scottish referendum on independence succeeds and Scotland were to leave the United Kingdom as it would get its own two letter country code TLD. Let’s see…..

Nicola Bingham, Web Archivist, The British Library

25 July 2014

Special Collection – Tour de France comes to Yorkshire

Add comment Comments (0)

As curator for sport in the British Library I have had a pretty exciting time in recent years, with plenty of sporting mega-events hitting the headlines in the UK, including the London Olympic Games and recently the Tour de France starting In Yorkshire.

The latter was celebrated by the Library in a number of ways: several members of staff actually biked from St Pancras to our Yorkshire site in Boston Spa (a two-day; 200 mile journey); while I (taking the train!) helped to create a small exhibition of cycling-related collections items in cases close to the newly refurbished Boston Spa reading room. Here I am with my colleague Robert Davies in front of the exhibition.

As with most of the significant events taking place in this country, the web archiving team wanted to make a record of the Tour of Yorkshire’s online presence for future researchers, so I was given a watching brief for relevant websites.



The Grand Depart
Everyone now knows that the Grand Depart was a resounding success in attracting enthusiastic spectators all along its route from Leeds to the Mall in London. The Tour organisers expected three million people to line the roads; they achieved more than double that! I anticipated a great response (similar to the success of the torch relay in 2012) so I was very keen to ensure that we archive the many different websites of the local councils and tourist offices through whose boroughs and counties the tour would pass. Many of these web sites had huge amounts of information on them, from details of local campsites, guest houses and B&Bs to special brochures with interactive maps and lists of events connected to the Tour. Opportunities for future tourism were clearly being optimised.

A mega event
It had to be borne in mind that the Grand Depart was not just a special event for the UK but formed part of a larger sociological and anthropological phenomenon: i.e., the mega-event, a phenomenon which is a growing area of research in a number of subject areas – not only in sport, where the development of organisations like the IOC and FIFA are of interest to sports sociologists and historians – but to economists and cultural observers. The local activity encouraged by such events, like the Tour-associated cultural festivals, and educational projects bear witness to their wide-ranging social impact.

Which websites to archive?
So all this had to be recorded if possible. Add to this the day-by-day; hour-by-hour reports of media organisations like broadcasters and newspapers and there were clearly a large number of websites waiting to be gathered. One aspect did seem to be missing, and that was the protest sites, which tend to be much in evidence with events like the Olympic Games. Contrary to this, most Tour websites were celebrating the Tour in every way possible. Where they did echo the Olympics was in their keen embracing of the successful outcomes of the latter - such as volunteering - with Asda sponsoring a volunteering website which called for route and crossing marshals, ‘dignitary managers’ and coordinators of all kinds.

The riders
The websites of the riders themselves proved problematic at first, as it was not clear until almost the last minute who was going to ride. In the end, as we know, Sir Bradley Wiggins bowed out, but we made sure that we kept a close eye on Chris Froome and Mark Cavendish, as well as the UK based teams like Team Sky, The British Cycling Organisation and the Tour de France organisation itself. It was a huge disappointment to see British hopes being dashed by falls but we can now follow Chris Froomes twitter feed, from his original expressions of excitement to his reports on his MRI scans ‘confirmed fractures to the left wrist and right hand’. While on his Facebook page, Mark Cavendish displays a picture of himself fresh from the operating theatre! Sad, but interesting, times.

The collection
Websites are marvellous research sources for the study of sport in particular. With their aid you can observe events as they take place from day to day, and get a marvellous feel for the atmosphere surrounding these exciting occasions. The process of archiving the Tour sites is not over. In the aftermath of such events the sites will often sum up their experiences, and others may even spring up in response to what has taken place. So the watching brief is certainly not over!

By Gill Ridgley, Lead Curator, Sociological and Cultural Studies, The British Library

23 July 2014

First World War Centenary – an online legacy in partnership with the HLF

Add comment Comments (0)

Earlier this year, we at the UK Web Archive were delighted to reach an agreement with the Heritage Lottery Fund (HLF) to enable the archiving of a very large and significant set of websites relating to the Centenary of the First World War.

Throughout the Centenary and beyond, we will be working with the HLF in order to take archival copies of the websites of all HLF-funded First World War Centenary projects, and to make them available to users in the Open UK Web Archive. The first of these archived sites are already available in the First World War special collection but we hope that this will eventually lead to more than 1,000.

HLF Funding
HLF is funding First World War projects throughout the Centenary, ranging from small community projects to major museum redevelopments. Grants start at £3,000 and funding is available through four different grants programmes: First World War: then and now (grants of £3,000 - £10,000), Our Heritage (grants of £10,000 - £100,000), Young Roots (Grants of £10,000 - £50,000 for projects led by young people) and Heritage Grants (grants of more than £100,000).


Include your website
If you have HLF funding for a First World War Centenary project, please send the URL (web address) to with your project reference number.

If you have a UK-based WW1 website NOT funded by HLF we would still encourage you to add it for permanent archiving through our Nominate form.

This set of archived websites will form a key part of our wider Centenary collection, and capture an important legacy of this most significant of anniversaries.

By Jason Webber, Web Archiving Engagement and Liaison Officer, The British Library