The Elements of Video Production in Mathematica

This item was originally posted on the Wolfram Research blog at http://blog.wolfram.com/2007/06/29/the-elements-of-video-production-in-mathematica/

A little hobby of mine is collecting and photographing the chemical elements. I have them all (except those that break the laws of man or physics). This is the photographic periodic table poster I sell:

My poster and related imagery can be seen in several TV shows, and most recently staff at the venerable NOVA science series emailed asking for permission to use my poster image in an upcoming show about metals. They wanted to pan and zoom over it, starting wide and then focusing down onto a few individual elements.

I said, "Fine, but I have something I think you'll like even better.... How about a video where every one of those samples is rotating in place?"

And here is that video.

What does any of this have to do with Mathematica? That video, and the more complex ones below, are directly output from Mathematica without any processing in traditional video editing tools.

Why did I use Mathematica rather than standard video or image editing tools (which I have and use routinely for other things)? Because Mathematica is the only tool I know in which it would not have been severely painful. I'm busy--I can only make a video like this if I can do it with a minimal amount of labor, because studying elements is my hobby, not my full-time job.

The raw material is made up of 42,480 individual 8-megapixel still frames created on a turntable rig (118 elements times 360 frames per element). These 42,480 shots need to be assembled into a periodic table grid with appropriate text and grid lines overlaid. Think about doing that 360 times in Adobe Photoshop (I have done it, once, for the poster). Yes, you could script it, but that would be painful. Think about doing it by first creating 118 individual element videos, then assembling them in Final Cut Pro. Yes, you could do it, but that, too, would be painful. Very painful.

But the real killer is if you want to zoom far in like the folks at NOVA wanted to, from the full table down to where an individual element fills the whole screen:

The problem with doing this move in a video editing system is that in order to have the fully zoomed version be perfectly sharp, you'd have to start with an extremely high-resolution version of the original video.

If you want to zoom to where a single element fills the height of a 1080p high-definition resolution video frame, the full table would be 18,000 x 9,000 pixels. The world has not yet evolved to the point where 162-megapixel video editing is commonplace. Or any place. When I offered to supply the video at that resolution, their response was to say that their video suite would just laugh at such a file, which is pretty much what I was expecting.

No, the only practical alternative, if you want to avoid the horrible compromise of doing the move over a single high-resolution frame, rather than a rotating video, is to assemble the frames at the target resolution one by one, picking just the images you need at the resolution you need based on the position and zoom level. When you're at wide zoom there are many images, but they are all small. When you're in tight, the images are large, but there aren't very many of them.

Doing this in Mathematica is quite straightforward, once you get over the mental block of thinking Mathematica is just a math program.

I used Import to read in JPEG files of the necessary images. (I have them stored at four different resolutions: 1000, 400, 150, and 65 pixels square; the code automatically picks the next higher resolution based on the destination size of the image.)

Then I constructed a Graphics object with the imported images placed as Raster objects. The text for each element tile is placed using Inset, with drop shadows constructed by duplicating the text multiple times shifted over slightly with transparency added.

The fully assembled frame is then exported with Export.

To do the moves (pan and zoom motions) I used a simple Bezier spline interpolation to make smooth starts and stops.

This is all rather straightforward Mathematica programming--only about a hundred lines in all, including the code to do drop shadows on text, assemble each element tile, arrange them in the standard periodic table layout, and do all the Import and Export.

Given this hundred lines of composite image assembly code, the instructions to create the specific sequence of pans and zooms NOVA wanted, according to their explicit timing specifications, is this:

The result of running this code (which takes about six hours on my laptop to generate 1,095 high-definition frames) is a directory full of JPEG (or TARGA, which is what they wanted) files, ready to assemble into QuickTime or another video format.

You could, in theory, Export the finished movie rather than the individual frames directly from Mathematica, but that's not what the NOVA people wanted, and it's actually more convenient to work with individual frame files. For example, you can generate them out of order to spot-check that it's going to fit together, or run multiple copies of Mathematica to generate different sets at the same time.

When the NOVA staff decided to switch from bismuth (Bi) to lead (Pb) as the final target, it was a one-character change, then I hit Shift-Return. Nice. While we were at it, I hit Shift-Return to generate this variation, which loops back and forth between carbon and silver, two of my favorite rotating samples:

In a world without Mathematica, this kind of thing would be done using either code in a language like C or Java (if you had a programmer and a few weeks handy) or, quite painfully, using scripting languages in some photo, video, or page layout programs. Actually, the truth is that it simply wouldn't have gotten done at all: They would have done the pan and zoom over a single, non-rotating still frame of the whole table. This is going to be maybe 30 seconds in a documentary; there's no budget for anything like what I'm describing.

In this case the project was possible only because Mathematica exists and I know how to use it, and therefore could do the project in an afternoon or two while at the same time creating a general tool that I'll be able to use for other variations (and a nifty blog post).

This is an example of the payoff you get from learning Mathematica. You probably wouldn't start using it in order to make periodic table videos, but once you know it and are comfortable with the range of things it can do, you're constantly surprised at what random project ideas suddenly seem more doable.

The most satisfying thing was when, after sending yet another draft rendering, I finally got the email asking, "Um... how exactly are you doing that?"