FaceTerm: Interactive Installation That Turns Visitors into ASCII Art

Continuing the spirit of TypeContour and BodyType I made for previous member's weeks, I present this member's week FaceTerm, yet another installation that combines computer graphics, neural networks, and typography.

The idea of FaceTerm is that of the classic "turn image into ASCII art". However, as we initially came up with the idea, Prof. Zach Lieberman and I both agreed that the traditional approach of luminosity mapping from pixels to ASCII characters is not too interesting, and something more sophisticated is to be done to capture the spirit of the input image. We were also inspired by emoticons or Kaomojis, where text characters are used to represent facial features, which are usually hand-crafted and pre-arranged. So how can we computationally generate "dynamic" and "personalized" Kaomojis that take the likeness the user and reflect their gestures in real time?

The luminosity mapping algorithm is very simple and you have probably seen it done many times. Take all the printable ASCII characters, and given a font, determine a "darkness value" for each and sort them accordingly. For example, with black foreground on white background, the "#" character is typically quite dark, while the ";" character is much lighter in comparison. The whitespace is undoubtably the brightest, while "M" or "W" are often among the darkest. Now, look at each pixel of an image (downscaled to whatever resolution desired) and determine its luminosity -- say 80%. You draw the character closest to that percentage in the sorted list: assuming 95 printable characters you'll be picking the 19th brightest character. You may equalize the histogram of the image or do whatever pre-processing that might help improve legibility first, or use a non-linear or weighted mapping, etc. But the point is, in any case, it essentially produces a pixelated image, with each pixel drawn with a character, whose actual shape doesn't really matter -- we only use it for its perceived brightness value.

However, if you look at hand-crafted ASCII art, they often use the characters in a more meaningful way. Say if you're trying to draw a small button on a shirt, the luminosity method might just give you an "F" for example, since the brightness value happens to be closest to that of an "F". If you move around, then the subtle change in lighting and the pixel aliasing might cause the button to turn into a "3" or "$" -- there is no consistency. Whereas a human artist will probably draw the button with an "O" (or "@") since that mirrors the circular shape of the object, and be consistent with that across frames. You would draw a ">" for a pointy bump, a "|" for a vertical edge, or use "#" to represent some mesh or grid. This is a more interesting exploration of shapes and forms in typography that we're interested in. In other words, we want to "draw" not "shade", and in "drawing" we want to simulate what a human artist might draw.

Therefore, it is natural to start by running an edge detection pass on the input image, for an edge detection captures well the outlines and salient features in an image (given adequate lighting), and is quite similar to the human drawing process, if you think about it. For that I simply used the Sobel operator. Many newer and fancier algorithms exist, but they seem to be mostly driven by the idea that Sobel and other traditional operators capture too much detail. For example, if you run Sobel on a furry dog, then you might get a million edges, one for each strand of hair; and the outline will be rather noisy: some algorithm decides that this is not so desirable and we only want a smooth outline of the dog (or at least, tell between object outline and inner details). In our situation however, it doesn't matter too much: Indeed, if I were to draw a really furry dog, I'll surely make some frantic marks on its body to indicates its furriness. Moreover, Sobel is so lightweight and dead simple that the overhead is negligible -- if it is later determined that a more sophisticated algorithm is desired and there's still some slack in the FPS to allow for it, we can always swap one in later.

Now that we have the edge detection image, we'll need to translate it into ASCII characters. Obviously the general idea would be to chop up the image into small patches, and for each patch determine the ASCII character that resembles it the most. The real question is how to calculate similarity. A few ideas popped into mind: for example, we could figure out the general gradient direction of each patch, and emprically assign the character for it: for example a "|" for a vertical gradient, a "-" for a horizontal one, a "/" for a tilted one, etc. However, this ends up only using a very small subset of ASCII characters, and the result doesn't look too impressive, and legiblility suffers at lower resolutions. But at least we have a baseline.

We could also superimpose a rendering of an ASCII character on top of a patch, and see how well the pixels align. We can do a convolution/cross-correlation and slide them against each other to account for any translation. However, what about rotation, scale, shear, or even warp? Fourier transform could probably solve some of those, but performance would likely suffer since we need to do thousands of matches per frame (e.g. 80 width x 24 height x 95 ASCII, that's quite a lot of FFT's to do!). Of course I could be totally wrong about performance because I didn't actually implement that since I went with a "lazier" method, described below.

It occurred to me that we're basically in some sort of image classification / OCR situation. We want to figure out given a pattern, which ASCII character, no matter how abstractly or spiritually, it is the most similar to, in its general shape. And this is the space which requires subhuman imagination (some imagination but not too much) where a neural network would shine. And in this particular case, a CNN (convolutional neural net) would likely suffice.

I generated a dataset of maliciously badly rotated, translated, scaled, distorted and blurred to the brink of illegible images of ASCII characters. I used various weights and italics of Source Code Pro, an open source font from Adobe, as its glyphs have quite clean and distinct shapes (compared to more heavily serifed fonts). Next, I trained a tiny custom VGG-style CNN on around 20K of these images. The CNN has ~100K parameters, and has ~92% accuracy on the testing set. It might not be super accurate if you actually want to read English from an image of English text, but for reading nonsense from an image of non-text, it should be good enough!

Looking at the test results, I realized that I might have accidentally created some sort of primitive CAPTCHA cracker.

After sorting out some minor misunderstandings between myself and the tensorflow.js library, the CNN started working quite well on the edge detection image. After doing batching, it is quite performant too. I'm not sure if a CNN is a bit of an overkill for this, but I'm a lazy one and appreciate the beauty of the end-to-end nature of this approach.

Of course, it is still not as good as if a human artist was to draw the input image as ASCII art. I realized that perhaps the key difference lies in distortion and aliasing. Say if there's a small circle exactly at coordinate (7,7). Then the human and the CNN will both put an "O". If the circle is at (7.5,7), then the CNN will perhaps resort to "()", which is still reasonable. But if the circle is at (6.65,7.3), then the CNN might just put some small characters at the four closest positions -- if you squint hard enough it still resembles a circle -- but what a human would do, is to still draw an "O" at (7,7) regardless. Another example: if there's a long slanted edge in the image, but it is not exactly 63 degrees (atan(2), or the angle of "/" character), and it wiggles along its way, the CNN will a bit awkwardly assemble the edge with a combination of "/", "-", "|", "_" and other characters, whereas the human might decide to simplify the image and simply draw the whole edge with only "/" characters. In other words, humans can understand salient features they find important, and bend reality in order to preserve them.
The current system can already compensate for that a little bit. Each patch fed into the CNN is bigger than the area the output character will actually occupy. This way the CNN get to "peek" into neighboring structure and decide its output with more context. However, to do a more holistic planning of the image like a human would, perhaps an even "smarter" neural network would be necessary. Perhaps the latest diffusion models will be able to handle this. It could be very interesting for near-future research.

Anther observation I made when playing with the system is that we humans are most interested in seeing our own likeness, and more specifically, our face. (At first I thought maybe it's just me being narcissistic but later confirmed the hypothesis when exhibiting the piece). I don't really mind if the room behind me is depicted with a random debris of ASCII characters (it is in a bit of a mess anyways), but I really want to make out my eyes, nose, mouth in the ASCII depiction. I want to see changes as I blink or make a face. The aforementioned aliasing problem as well as artifacts caused by noise and lighting makes this difficult at lower resolutions. Of course, if I keep cranking up the resolution, then the facial features do become legible, but then the role of the characters as shape components decreases, and they again degrade into substitute of pixels, which is the last thing we want since the beginning.

Therefore, I came up with a cheat, which is to treat the face separately from the rest of the image. A facial landmark detector gives much more salient information about an image than an edge detector, as long as the image is that of a face. Now that I know where the eyes, noses and mouths are, and more importantly, what states they're in, I get to represent them in the style of emoticons or Kaomojis. I used "<@>" for an open eye, "(@)" for a wide-open eye, "-e-" for a squinting eye, and "-=-" for a closed eye, and it seems have immediately granted my ASCII character likeness more character, with less characters (no pun intended).

Typically with emoticons or Kaomojis one would also give the mouth and each eyebrow a symbolic treatment. However, as the rest of the image is to-scale, I decided to go with a more realistic approach: drawing the lips and eyebrows as polylines connecting keypoints from the facial landmark detection. Note that this line rasterization happens in the ASCII art space instead of the input image space. The simplest solution would be to use Bresenham's algorithm treating each ASCII character as a pixel -- just put a "#" for line pixels and " " for empty pixels. However, that would look coarse. Therefore, I used a sub-pixel method. I treat each ASCII character as a group of 4 pixels: upper left, upper right, lower left, lower right, and rasterize the lines to a temporary buffer of 2x resolution. Then, for each character I empirically pick the glyph that best represents the 2x2 neighborhood in the temporary buffer: "/" for filled lower left pixel and upper right pixel, "b" for filled on all but the upper right pixel, and so on. Of course, some of the 2^4=16 configurations cannot be perfectly represented, or two configurations might have identical ASCII representations. But we can only blame the Romans (and other ancient peoples in part) for their alphabet.

Finally, I give the chin outline the same subpixel line rasterization treatment, and block out the original face with a big (slightly dilated and/or blurred) polygon before passing the rest of the image to the CNN -- and the result looked quite good! I had much fun playing with it; it was interesting to see how facial expressions translate to ASCII, and I feel quite convinced that the figure, despite consisting of only some characters, is me. Only one problem though, the facial landmark detection does not take anything that you didn't have the first day you were born into account: any eyeglasses, facial hair, scars, tattoos are lost.

I decided to train another CNN for detecting these. I used the "CelebA" dataset, which contains photographs of celebrities as well as a bunch of attributes: whether they have big nose, double chin, beards, eye bags, glasses, and even whether they're attractive or not (gotta appreciate the bluntness of the folks at HKU!). For our purposes, I extracted the attributes relating to facial hair and eyeglasses. There're definitely some limitations though, since celebrities are cool people and don't wear prescription glasses, and those labelled as having glasses are mostly wearing cool sunglasses. Having both eyeglasses and a moustache is even rarer. Regardless, I put together a subset of 20K images of where there are comparable number of images for each possible combination: no glass no moustache, glass no moustache, moustache no glass, moustache and glass. The last combination is augmented with some flipped images.

The trained CNN is eh, mostly working. Since the dataset doesn't contain images for "random object over the face that are not eyeglasses", anything that occlude the face will trigger it to think it's someone with glasses. And only when I take off my glasses does it suddenly become convinced that I do have moustache -- perhaps also due to the lack of training data. I decided to give up on facial hair first, since there's also lots of different styles for them, and can be a whole problem in itself to render each. I decided ditch glasses too later, since testing it more under different lighting conditions proved that it's not too reliable either -- restoring a person with glasses to perfect eyesight is arguably not that bad, but giving a normal person some glasses that flickers on and off is worse.

Now that the key technical problems are (mostly) cracked, it's time to do some lighthearted graphics programming. I wrote a set of GLSL shaders to reproduce the "retro CRT monitor" kinda effect. First a "bloom" filter is applied, which consists of separated X/Y Gaussian blur passes, and a composition with the original image to reproduce glowing text. Then a sin wave along the Y is used to modulate the brightness so that each line of pixel appear slightly separated. Then a barrel distortion warps the entire image to recreate the illusion of the "bulging" shape of these old monitors. Finally some random noise is added to the background, and the whole image is tinted. The most classic tint would be a green one, but I found that an amber one would look even better, and a blue one would look futuristic. In the final installation, I can switch between the tints with a secret keypress.

In addition to the retro look, Zach suggested that I also add keyboard typing noises that responds to the number of characters changing on the screen at a time. Of course, I can just download some mechanical keyboard sound samples -- I bet the mechanical keyboard fans must be uploading them. Or I could record some from my very own $10 keyboard. But I decided that I want to take another stab of procedurally generating sounds. I make a lot of procedurally generated graphics, but am not particularly proficient at generating sounds beyond the usual sin waves and white noises. This time, I decided to convert the unknown problem to a familiar problem -- reduction! My reasoning is that, if two PCM waveforms are exactly the same, it will sound exactly the same. If they're close enough, they'll sound close enough. Therefore, I'll just visualize the waveform of a real recorded keyboard strike, and try to generate a shape that looks exactly like it, just like how I generate mountains and terrains and such. And it worked (mostly)! The resultant waveform does produce a noise not dissimilar to what certain keyboards might produce. It doesn't capture all the nuances, but I'm still quite happy that my approach is working. I parameterized the function so that each generated sound is slightly different, and plugged it into the main software. A wave of clacking sounds as you move around is quite satisfying to hear, almost ASMR-like.

I've mostly been prototyping in JavaScript, and on a decent laptop, it runs mostly smoothly in real time with the exception of occasional kick-ins from the garbage collector. But I figured that for an installation I want it to have perfect performance and robustness. I first tried porting it to openFrameworks, but then realized that I'm so deeply stuck in raw WebGL/OpenGL land, that I'm fighting openFrameworks to get it to do what I want. The feeling was rather interesting -- I used to be happy with openFrameworks hiding the nasty details of OpenGL for me and being able to program in a more friendly interface, but in recent days after working a lot with low level GL and beginning to understand everything in and out, I become quite confused about what openFrameworks is trying to hide from me behind these private and protected classes, and kept trying to bypass it to do things the OG way. In the end, I wrote the software in C and Objective C, with an OpenGL context on a Cocoa window. The webcam feed is with AVFoundation, and the CNN model was ported to CoreML with coremltools (another big struggle with tfjs/tensorflow/keras/savedmodel versions, incompatibilities, and catch-22 situations etc., and verbally abusing ChatGPT demanding correct commands to use isn't making the experience much more bearable).

The piece was installed on an M1 mac mini with a large vertical display. It runs very smoothly and never crashes. Everyone who came to see it at member's week had some varying amount of fun, though some failed to fathom the practical use of such a thing, in which case I would mumble something about "body", "typography" and "art".