THE BRITISH LIBRARY

UK Web Archive blog

48 posts categorized "Web/Tech"

11 May 2018

Online Hours: Supporting Open Source

Add comment

Encouraging collaboration
Here at the UK Web Archive, we're very fortunate to be able to work in the open, with almost all code on GitHub. Some of our work has been taken up and re-used by others, which is great. We’d like to encourage more collaboration, but we've had trouble dedicating time to open project management, and our overall management process and our future plans are unclear. For example, we've experimented with so many different technologies over the years that our list of repositories give little insight into where we're going next. There are also problems with how issues and pull-requests have been managed: often languishing unanswered, waiting for us to get around to looking at them. This also applies to the IIPC repositories and other projects we are involved in, as well as the projects we lead.

I wanted to block out some time to deal with these things promptly, but also to find a way of making it a bit more, well, fun. A bit more social. Some forum where we can chat about our process and plans without the formality of having to write things up.

Taking inspiration from Jason Scott live-streamed CD-ripping sessions, we came up with the idea of something like Office Hours for Open Source -- a kind open open video conference or live stream, where we'll share our process, discuss issues relating to open source projects and have a forum where anyone can ask questions about what we’re up to.

Who is this for?
All welcome, from lurkers to those brimming with burning questions. Just remember that being *kind* beats being right.

Furthermore, if anyone else who manages open source projects like ours is also welcome to join and take the lead for a while! I can only cover the projects we’re leading, but there are many more that would be interesting to hear from.

When?
The plan is to launch the first Online Hours session on the 22nd of May, and then hold regular weekly slots every Tuesday from then on. We may not manage to run it every single week, but if it’s regular and frequent that should mean we can cope more easily with missing the odd one or two.

On the 22nd, we will run two sessions - one in the morning (for the west-of-GMT time-zones) and one in the evening (for the eastern half). Following that, we intend to switch between the two slots, making each a.m. and p.m. slot a fortnightly occurrence.

How?
The sessions will be webcast with a slack channel available for chat. See the IIPC Trello board for more information.

The IIPC (International Internet Preservation Consortium) have kindly agreed to help support this event and further Online Hours sessions. Running this initiative in a more open manner should raise the profile of our open source work both inside and outside of the IIPC, and encourage greater adoption of, and collaboration around, open source tools.

For full details, see the IIPC Trello Board card or ask a question in the NetPreserve Slack Channel #oh-sos (ask @NetPreserve to join the Slack).

See you there!

By Andrew Jackson, Web Archive Technical Lead, The British Library

 

04 May 2018

Star Wars in the Web Archive

Add comment

May the fourth be with you!

It's Star Wars day and I imagine that you are curious to know which side has won the battle of the UK web space?

Looking at the trends in our SHINE dataset (.uk websites 1996-2013 collected by Internet Archive) I first looked at the iconic match-up of Luke vs Darth.

Shine-darth-vader

Bad news, evil seems to have won this round mainly, it seems, due to the popularity of Darth Vader costume mentions on retail websites.

How about a more general 'Light Side vs Dark side'? 

Shine-lightside-v-darkside

It appears that discussing the 'dark side' of many aspects in life is a lot more fun and interesting than the 'light side'. 

How about just analysing the phrase 'may the force be with you'?

Shine-may the force be with you

This phrase doesn't seem to have been particularly popular on the UK web until it started to be used a lot on websites offering downloadable ringtones. Go figure.

Try using the trends feature on this dataset yourself here: www.webarchive.org.uk/shine/graph

Happy stars wars day!

by Jason Webber, Web Archive Engagement Manager, The British Library

@UKWebArchive

 

01 February 2018

A New Playback Tool for the UK Web Archive

Add comment

We are delighted to announce that the UK Web Archive will be working with Rhizome to build a version of pywb (Python Wayback) that we hope will greatly improve the quality of playback for access to our archived content.

What is playback of a web archive?

When we archive the web, just downloading the content is not enough. Data can be copied from the web into an archive in a variety of ways, but to make this archive actually accessible takes more than just opening downloaded files in a web browser. Technical details of pages and scripts coming out of the archive need to be presented in a way that enables them to work just like the originals, although they aren’t located on their actual servers anymore. Today’s web users have come to expect interactive features and dynamic layouts on all types of websites. Faithfully reproducing these behaviors in the archive has become an increasingly complex challenge, requiring web archive playback software that is on-par with the evolution of the web as a whole.

Why change?

Currently, we use the OpenWayback playback system, originally developed by the Internet Archive. But in more recent years, Rhizome have led the development of a new playback engine, called pywb (Python Wayback). This Python toolkit for accessing web archives is part of the Webrecorder project, and provides a modern and powerful alternative implementation that is being run as an open source project. This has led to rapid adoption of pywb, as the toolkit is already being used by the Portuguese Web Archive, perma.cc, the UK National Archives, the UK Parliamentary Archive, and a number of others.

Open development
To meet our needs we need to modify pywb, but as strong believers in open source development, all work will be in the open, and wherever appropriate, we will fold the improvements back into the core pywb project.

If all goes to plan, we expect to contribute the following back to pywb for others to use:

Other UKWA-specific changes, like theming, implementing our Legal Deposit restrictions, and deployment support, will be maintained separately.

Initially we will work with Rhizome to ensure our staff and curators can access our archived material via both pywb and OpenWayback. If the new playback tool performs as expected  we will move towards using pywb to support public access to all our web archives.

22 December 2017

What can you find in the (Beta) UK Web Archive?

Add comment

We recently launched a new Beta interface for searching and browsing our web archive collections but what can you expect to find?

UKWA have been collecting websites since 2005 in a range of different ways, using changing software and hardware. This means that behind the scenes we can't bring all of the material collected into one place all at once. What isn't there currently will be added over the next six months (look out for notices on twitter). 

What is available now?

At launch on 5 December 2017 the Beta website includes all of the websites that have been 'frequently crawled' (collected more often than annually) 2013-2016. This includes a large number of 'Special Collections' selected over this time and a reasonable selection of news media.

DC07-screenshot-brexit

What is coming soon?

We are aiming to add 'frequently crawled' websites from 2005-2013 to the Beta collection in January/February 2018. This will add our earliest special collections (e.g. 2005 UK General Election) and should complete all of the websites that we have permission to publicly display.

What will be available by summer 2018?

The largest and most difficult task for us is to add all of the websites that have been collected as part of the 'Legal Deposit' since 2013. We do a vast 'crawl' once per year of 'everything' that we can identify as being a UK website. This includes all .UK (and .london, .scot, .wales and .cymru) plus any website we identify as being on a server in the UK. This amounts to tens of millions of websites (and billions of individual assets). Due to the scale of this undertaking we thank you for your patience.

We would love to know your experiences of using the new Beta service, let us know here: www.surveymonkey.co.uk/r/ukwasurvey01

 By Jason Webber, Web Archive Engagement Manager, The British Library

 

10 November 2017

Driving Crawls With Web Annotations

Add comment

By Dr Andrew Jackson, Web Archive Technical Lead, The British Library

The heart of the idea was simple. Rather than our traditional linear harvesting process, we would think in terms of annotating the live web, and imagine how we might use those annotations to drive the web-archiving process. From this perspective, each Target in the Web Curator Tool is really very similar to a bookmark on an social bookmarking service (like PinboardDiigo or Delicious1), except that as well as describing the web site, the annotations also drive the archiving of that site2.

In this unified model, some annotations may simply highlight a specific site or URL at some point in time, using descriptive metadata to help ensure important resources are made available to our users. Others might more explicitly drive the crawling process, by describing how often the site should be re-crawled, whether robots.txt should be obeyed, and so on. Crucially, where a particular website cannot be ruled as in-scope for UK legal deposit automatically, the annotations can be used to record any additional evidence that permits us to crawl the site. Any permissions we have sought in order to make an archived web site available under open access can also be recorded in much the same way.

Once we have crawled the URLs and sites of interest, we can then apply the same annotation model to the captured material. In particular, we can combine one or more targets with a selection of annotated snapshots to form a collection. These ‘instance annotations’ could be quite detailed, similar to those supported by web annotation services like Hypothes.is, and indeed this may provide a way for web archives to support and interoperate with services like that.3

Thinking in terms of annotations also makes it easier to peel processes apart from their results. For example, metadata that indicates whether we have passed those instances through a QA process can be recorded as annotations on our archived web, but the actual QA process itself can be done entirely outside of the tool that records the annotations.

To test out this approach, we built a prototype Annotation & Curation Tool (ACT) based on Drupal. Drupal makes it easy to create web UIs for custom content types, and we were able to create a simple, usable interface very quickly. This allowed curators to register URLs and specify the additional metadata we needed, including the crawl permissions, schedules and frequencies. But how do we use this to drive the crawl?

Our solution was to configure Drupal so that it provided a ‘crawl feed’ in a machine readable format. This was initially a simple list of data objects (one per Target), containing all the information we held about that Target, and where the list could be filtered by crawl frequency (daily, weekly, monthly, and so on). However, as the number of entries in the system grew, having the entire set of data associated with each Target eventually became unmanageable. This led to a simplified description that just contains the information we need to run a crawl, which looks something like this:

[
    {
        "id": 1,
        "title": "gov.uk Publications",
        "seeds": [
            "https://www.gov.uk/government/publications"
        ],
        "schedules": [
            {
                "frequency": "MONTHLY",
                "startDate": 1438246800000,
                "endDate": null
            }
        ],
        "scope": "root",
        "depth": "DEEP",
        "ignoreRobotsTxt": false,
        "documentUrlScheme": null,
        "loginPageUrl": null,
        "secretId": null,
        "logoutUrl": null,
        "watched": false
    },
    ...



This simple data export became the first of our web archiving APIs – a set of application programming interfaces we use to try to split large services into modular components4.

Of course, the output of the crawl engines also needs to meet some kind of standard so that the downstream indexing, ingesting and access tools know what to do. This works much like the API concept described above, but is even simpler, as we just rely on standard file formats in a fixed directory layout. Any crawler can be used as long as it outputs standard WARCs and logs, and puts them into the following directory layout:

/output/logs/{job-identifer}/{launch-timestamp}/*.log
/output/warcs/{job-identifer}/{launch-timestamp}/*.warc.gz

Where the {job-identifer} is used to specify which crawl job (and hence which crawl configuration) is being used, and the {launch-timestamp} is used to separate distinct jobs launched using the same overall configuration, reflecting repeated re-crawling of the same sites over time.

In other words, if we have two different crawler engines that can be driven by the same crawl feed data and output the same format results, we can switch between them easily. Similarly, we can make any kind of changes to our Annotation & Curation Tool, or even replace it entirely, and as long as it generates the same crawl feed data, the crawler engine doesn’t have to care. Finally, as we’ve also standardised the crawler output, the tools we use to post-process our crawl data can also be independent of the specific crawl engine in use.

This separation of components has been crucial to our recent progress. By de-coupling the different processes within the crawl lifecycle, each of the individual parts is able to be move at it’s own pace. Each can be modified, tested and rolled-out without affecting the others, if we so choose. True, making large changes that affect multiple components does require more careful management of the development process, but this is a small price to pay for the ease by which we can roll out improvements and bugfixes to individual components.

A prime example of this is how our Heritrix crawl engine itself has evolved over time, and that will be the subject of the next blog post.

  1. Although, noting that Delicious is now owned by Pinboard, I would like to make it clear that we are not attempting to compete with Pinboard. 

  2. Note that this is also a feature of some bookmarking sites. But we are not attempting to compete with Pinboard. 

  3. I’m not yet sure how this might work, but some combination of the Open Annotation Specification and Memento might be a good starting point. 

  4. For more information, see the Architecture section of this follow-up blog post 

25 September 2017

Collecting Webcomics in the UK Web Archive

Add comment

By Jen Aggleton, PhD candidate in Education at the University of Cambridge

As part of my PhD placement at the British Library, I was asked to establish a special collection of webcomics within the UK Web Archive. In order to do so, it was necessary to outline the scope of the collection, and therefore attempt to define what exactly is and is not a digital comic. As anyone with a background in comics will tell you, comics scholars have been debating what exactly a comic is for decades, and have entirely failed to reach a consensus on the issue. The matter only gets trickier when you add in digital components such as audio and animation.

Under-construction

Due to this lack of consensus, I felt it was important to be very transparent about exactly what criteria have been used to outline the scope of this collection. These criteria have been developed through reference to scholarship on both digital and print comics, as well as my own analysis of numerous digital comics.

The scope of this collection covers items with the following characteristics:

  • The collection item must be published in a digital format
  • The collection item must contain a single panel image or series of interdependent images
  • The collection item must have a semi-guided reading pathway1

In addition, the collection item is likely to contain the following:

  • Visible frames
  • Iconic symbols such as word balloons
  • Hand-written style lettering which may use its visual form to communicate additional meaning

The item must not be:

  • Purely moving image
  • Purely audio

For contested items, where an item meets these categories but still does not seem to be a comic, it will be judged to be a comic if it self-identifies as such (e.g. a digital picturebook may meet all of these criteria, but self-identifies as a picturebook, not a comic).

Where the item is an adaptation of a print born comic, it must be a new expression of the original, not merely a different manifestation, according to FRBR guidelines: www.loc.gov/cds/FRBR.html.

1 Definition of a semi-guided reading pathway: The reader has autonomy over the time they spend reading any particular aspect of the item, and some agency over the order in which they read the item, especially the visual elements. However reading is also guided in the progression through any language elements, and likely to be guided in the order of movement from one image to another, though this pathway may not always be clear. This excludes items that are purely pictures, as well as items which are purely animation.

Alongside being clear about what the collection guidelines are, it is also important to give users information on the item acquisition process – how items were identified to be added to the collection. An attempt has been made to be comprehensive: including well known webcomics published in the UK and Ireland by award-winning artists, but also webcomics by creators making comics in their spare time and self-publishing their work. This process has, however, been limited by issues of discoverability and staff time.

Well known webcomics were added to the collection, along with webcomics discovered through internet searches, and those nominated by individuals after calls for nominations were sent out on social media. This process yielded an initial collection of 42 webcomic sites (a coincidental but nonetheless highly pleasing number, as surely comics do indeed contain the answers to the ultimate question of life, the universe, and everything). However, there are many more webcomics published by UK and Ireland based creators out there. If you know of a webcomic that should be added to our collection, please do nominate it at www.webarchive.org.uk/ukwa/info/nominate.

Jen Aggleton, PhD candidate in Education at the University of Cambridge, has recently completed a three month placement at the British Library on the subject of digital comics. For more information about what the placement has entailed, you can read this earlier blog.

09 August 2017

The Proper Serious Work of Preserving Digital Comics

Add comment

Jen Aggleton is a PhD candidate in Education at the University of Cambridge, and is completing a work placement at the British Library on the subject of digital comics. 

If you are a digital comics creator, publisher, or reader, we would love to hear from you. We’d like to know more about the digital comics that you create, find out links to add to our Web Archive collection, and find examples of comic apps that we could collect. Please email suggestions to Jennifer.Aggleton@BL.uk. For this initial stage of the project, we will be accepting suggestions until the end of August 2017.

I definitely didn’t apply for a three month placement at the British Library just to have an excuse to read comics every day. Having a number of research interests outside of my PhD topic of illustrated novels (including comics and library studies), I am always excited when I find opportunities which allow me to explore these strands a little more. So when I saw that the British Library were looking for PhD placement students to work in the area of 21st century British comics, I jumped at the chance.

Having convinced my supervisor that I wouldn’t just be reading comics all day but would actually be doing proper serious work, I temporarily put aside my PhD and came to London to read lots and lots of digital comics (for the purpose of proper serious work). And that’s when I quickly realised that I was already reading comics every day.

The reason I hadn’t noticed was because I hadn’t specifically picked up a printed comic or gone to a dedicated webcomic site every day (many days, sure, but not every day). I was however reading comics every day on Facebook, slipped in alongside dubiously targeted ads and cat videos. It occurred to me that lots of other people, even those who may not think of themselves as comics readers, were probably doing the same.

Forweb2-slytherinpic
(McGovern, E. My Life As A Background Slytherin, https://www.facebook.com/backgroundslytherin/photos/a.287354904946325.1073741827.287347468280402/338452443169904/?type=3&theater Reproduced with kind permission of Emily McGovern.)

This is because the ways in which we interact with comics have been vastly expanded by digital technology. Comics are now produced and circulated through a number of different platforms, including apps, websites and social media, allowing them to reach further than their traditional audience. These platforms have made digital comics simultaneously both more and less accessible than their print equivalents; many webcomics are available for free online, which means readers no longer have to pay between £8 and £25 for a graphic novel, but does require them to have already paid for a computer/tablet/smartphone and internet connection (or have access to one at their local library, provided their local library wasn’t a victim of austerity measures).

Alongside access to reading comics, access to publishing has also changed. Anyone with access to a computer and internet connection can now publish a comic online. This has opened up comics production to many whose voices may not have often been heard in mainstream print comics, including writers and characters of colour, women, members of the LGBTQ+ community, those with disabilities, and creators who simply cannot give up the stability of full-time employment to commit the time needed to chase their dream of being a comics creator. The result is a vibrant array of digital comics, enormously varying in form and having a significant social and cultural impact.

But digital comics are also far more fragile than their print companions, and this is where the proper serious work part of my placement comes in. Comics apps are frequently removed from app stores as new platform updates come in. Digital files become corrupted, or become obsolete as the technology used to host them is updated and replaced. Websites are taken down, leaving no trace (all those dire warnings that the internet is forever are not exactly true. For more details about the need for digital preservation, see an earlier post to this blog). So in order to make sure that all the fantastic work happening in digital comics now is still available for future generations (which in British Library terms could mean ten years down the line, or five hundred years down the line), we need to find ways to preserve what is being created.

One method of doing this is to establish a dedicated webcomics archive. The British Library already has a UK Web Archive, due to the extension of legal deposit in 2013 to include the collection of non-print items. I am currently working on setting up a special collection of UK webcomics within that archive. This has involved writing collections guidelines covering what will (and won’t) be included in the collection, which had me wrestling with the thorny problem of what exactly a digital comic is (comics scholars will know that nobody can agree on what a print comic is, so you can imagine the fun involved in trying to incorporate digital elements such as audio and video into the mix as well). It has also involved building the collection through web harvesting, tracking down webcomics for inclusion in the collection, and providing metadata (information about the collection item) for cataloguing purposes (this last task may happen to require reading lots of comics).

Alongside this, I am looking into ways that digital comics apps might be preserved, which is very proper serious work indeed. Not only are there many different versions of the same app, depending on what operating system you are using, but many apps are reliant not only on the software of the platform they are running on, but sometimes the hardware as well, with some apps integrating functions such as the camera of a tablet into their design. Simply downloading apps will provide you with lots of digital files that you won’t be able to open in a few years’ time (or possibly even a few months’ time, with the current pace of technology). This is not a problem that can be solved in the duration of a three month placement (or, frankly, given my total lack of technical knowledge, by me at all). What I can do, however, is find people who do have technical knowledge and ask them what they think. Preserving digital comics is a complicated and ongoing process, and it is a great experience to be in at the early stages of exploration.

And you can be involved in this fun experience too! If you are a digital comics creator, publisher, or reader, we would love to hear from you. We’d like to know more about the digital comics that you create, find out links to add to our Web Archive collection, and find examples of comic apps that we could collect. Please email suggestions to Jennifer.Aggleton@BL.uk. For this initial stage of the project, we will be accepting suggestions until the end of August 2017. In that time, we are particularly keen to receive web addresses for UK published webcomics, so that I can continue to build the web archive, and do the proper serious work of reading lots and lots of comics.

08 June 2017

Revitalising the UK Web Archive

Add comment

By Andrew Jackson, Web Archiving Technical Lead

It’s been over a year since we made our historical search system available, and it’s proven itself to be stable and useful. Since then, we’ve been largely focussed on changes to our crawl system, but we’ve also been planning how to take what we learned in the Big UK Domain Data for the Arts and Humanities project and use it to re-develop the UK Web Archive.

Screenshot-ukwa-homepage
UKWA homepage

Our current website has not changed much since 2013, and doesn’t describe who we are and what we do now that the UK Legal Deposit regulations are in place. It only describes the sites we have crawled by permission, and does not reflect the tens of thousands of sites and URLs that we have curated and categorised under Legal Deposit, nor the billions of web pages in the full collection. To try to address these issues, we’re currently developing a new website that will open-up and refresh our archives.

One of the biggest challenges is the search index. The 3.5 billion resources we’ve indexed for SHINE represents less than a third of our holdings, so now we need to scale our system up to cope with over ten billion documents, and a growth rate of 2-3 billion resource per year. We will continue working with the open source indexer we have developed, while updating our data processing platform (Apache Hadoop) and dedicating more hardware to the SolrCloud that holds our search indexes. If this all works as planned, we will be able to offer a complete search service that covers our entire archive, from 1995 to yesterday.

Shine-word-home
SHINE search results page

The first release of the new website is not expected to include all of the functionality offered by the SHINE prototype, just the core functionality we need to make our content and collections more available to a general audience. Quite how we bring together these two distinct views of the same underlying search index is an open question at this point it time. Later in the year, we will make the new website available as a public beta, and we’ll be looking for feedback from all our users, to help us decide how things should evolve from here.

As well as scaling up search, we’ve also been working to scale up our access service. While it doesn’t look all that different, our website playback service has been overhauled to cope with the scale of our full collection. This allows us to make our full holdings knowable, even if they aren’t openly accessible, so you get a more informative error message (and HTTP status code) if you attempt to access content that we can only make available on site at the present time. For example, if you look at our archive of google.co.uk, you can see that we have captured the Google U.K. homepage during our crawls but can’t make it openly available due to the legal framework we operate within.

The upgrades to our infrastructure will also allow us update the tools we use to analyse our holdings. In particular, we will be attending the Archives Unleashed 4.0 Datathon and looking at at the Warcbase and ArchiveSpark projects, as they provide a powerful set of open source tools and would enable us to collaborate directly with our research community. A stable data-analysis framework will also provide a platform for automated QA and report generation and make it much easier to update our datasets.

Taken together, we believe these developments will revolutionise the way readers and researchers can use the UK Web Archive. It’s going to be an interesting year.