Blog post by Tristan Roddis, Director of web development at Cogapp.

The genesis of this entry to the BL Labs awards 2017 (Artistic Award Runner up) can be traced back to an internal Cogapp hackathon back in July. Then I paired up with my colleague Jon White to create a system that was to be known as “the eyes have it”: the plan was to show the users webcam with two boxes for eyes overlaid, and they would have to move their face into position, whereupon the whole picture would morph into a portrait painting that had their eyes in the same locations.

So we set to work using OpenCV and Python to detect faces and eyes in both live video and a library of portraits from the National Portrait Gallery.

We quickly realised that this wasn’t going to work:

Green rectangles are what OpenCV think are eyes. I have too many. 

It turns out that eye detection is a bit too erratic, so we changed tack and only considered the whole face instead. I created a Python script to strip out the coordinates for faces from the portraits we had to hand, and another that would do the same for an individual frame from the webcam video sent from the browser to Python using websockets. Once we had both of these coordinates, the Python script sent the data back to the web front end, where Jon used the HTML <canvas> element to overlay the cropped portrait face exactly over the detected webcam face. As soon as we saw this in action, we realized we’d made something interesting and amusing!

And that was it for the first round of development. By the end of the day we had a rudimentary system that could successfully overlay faces on video. You can read more about that project on the Cogapp blog, or see the final raw output in this video:

A couple of months later, we heard about the British Library Labs Awards, and thought we should re-purpose this fledgling system to create something worth entering.

The first task was to swap out the source images for some from the British Library. Fortunately, the million public domain images that they published on Flickr contain 6,948 that have been tagged as “people”. So it was a simple matter to use a Flickr module for Python to download a few hundred of these and extract the face coordinates as before.

Once that was done, I roped in another colleague, Neil Hawkins, to help me add some improvements to the front-end display. In particular:

• Handling more than one face in shot
• Displaying the title of the work
• Displaying a thumbnail image of the full source image

And that was it! The final result can be seen in the video below of us testing it in and around our office. We also plugged in a laptop running the system to a large monitor in the BL conference centre so that BL Labs Symposium delegates could experience it first-hand.

If you want to know more about this, please get in touch! Tristan Roddis [email protected]

A clip of Tristan receiving the Award is below (starts at 8:42 and finishes at 14:10)