Face Tracking Integration


With a Billboard Hot 100 artist set to play at Coachella, multimedia company Comix were looking to experiment with face tracking and real-time VFX (video effects) for their production, but, being two separate applications, required a way of connecting them.

The solution was a TCP server that parsed incoming face tracking markers and forwarded this data on to real-time VFX software.

Project Requirements#

After discussions with users, we set the following requirements:

  1. Manual IP and port inputs are allowed, but no more technical than this.
  2. Informative logging for situations where connections may not work.
  3. Functionality over aesthetic.


The solution was a Python TCP server as an executable, packaged using pyinstaller. The client would be prompted to enter ports for both the TCP server and real-time VFX software upon start.

After this, the server parsed incoming face tracking data, forwarding this on to the real-time VFX software through UDP.

Supporting user and technical documentation were provided alongside the code for the final project handover.

Client Feedback#

Comix experienced zero downtime in experimenting with the executable and were able to complete their experiments.


Software Providers#

Without documentation of the message structure from the face tracking software, my initial idea was to infer a pattern from the message stream, taking me far longer than if I’d have been in contact with their developers since day 1.

Hardware Testing#

Much like your software mocks you on demo day and decides to not work, your camera may decide to give up the ghost all together before you’ve even got cracking with a project that - shock - requires your camera.