We got over 20 messages with essentially the same video sample in our inbox this week. They all touted the new interpolation from the DAIN experimental App or (Depth Aware Video Interpolation App)  which now analyses footage with a Neural network AI algorithm that crunches motion vectors and even what seemed impossible before “Object Occlusion” to generate higher frame rates from lower fps sources.  The technology is pretty fascinating and should be further improved by more training and samples over the coming years.

For stop motion animators, this is a complete game-changer as now you could animate with as little as 8fps and then interpolate to 30fps or 60fps with very little in the way of tearing and artifacting as long as the footage is well lit and objects clearly defined.  To make matters more interesting, it also analyses footage with shallow depth of field yielding impressive results.

Multiply Your Video Frame Rate with Interpolation or the “I” Word for Slow Motion Enthusiasts:

We visited the Interpolation topic in the past on our Fake Slow Motion article and concluded that then, the quality of interpolation while good was far from usable and you really could not compromise real high fps footage from interpolated versions except in very simple cases.

Now with DAIN technology, we have no choice but to re-visit the cases and analyze what it is capable of.  We looked at a few dozen examples and it is clear the technology has progressed forward so much that now stop motion animation, 2D Cell-based cartoon animation, and even 3D animated sequences rendered at 30p can easily be turned in higher fps increments yielding impressive and in some cases miraculous results.

We would like you to first watch the video below to understand what a depth map is and how the software in DAIN can create frames from nothing that look just like real ones.  A depth map will generate an approximated view of the world in a Lidar-Like vision representation to figure out to the best of the AI estimation where objects are in a scene according to their location close or far to the camera.

Depth-Aware Video Frame Interpolation by Wenbo Bao:

Even at 48fps from  12fps source, it is clear the technology in DAIN can yield impressive results even with heavy organic detail in the background including foliage. The software does an admirable job of estimating the relative position of objects in the video scenes.

AI使用フレーム補間アプリ　DAIN APP byTALBOの実験室 Ch.:

However, not everything is as peachy as the first example, when data is just too scarce, let’s say at 6fps, the software can completely break as not enough information can be inferred from the few frames it has to compute.

But, in the next sample, you can clearly see how 2D animation with low frame rates can be rendered in beautiful 60fps with miraculous results. We can see toon animation studios now animating at under 15fps for the majority of their projects possibly even at 8 or 10fps for many scenes and then using DAIN technology to fill in the in-between frames to deliver a final product not just for their intended market but for every TV system out there in 24, 25, 30, 48 or 60fps. This will save untold millions of work hours and dollars to animation houses.

Turning animations to 60fps using AI! by GRisk A.I. Sandbox:

Even a simple Gif animation can be turned into fluid and lifelike animation thanks to DAIN Neural Network AI technology.

Depth-Aware Video Frame Interpolation by Andromeda Kid:

In what could be the most amazing use of the technology yet, the Lego stop motion animation below was brought to a lifelike 60fps with incredible results that take the lens focal plane and objects that cross the frame into the calculation.

Boosting Stop-Motion to 60 fps using AI by LegoEddy:

But what about slow motion footage?

We hear you loud and clear and while we could see many examples where 30fps or 60fps footage could be converted using DAIN Technology, none were really pushing the envelope past 240fps to the thousands.  There is simply not enough data to create thousands for frames from a few with accurate results.

The video below of the Beirut explosion shows why interpolation for true slow motion capture, especially in very fast events cant be relied upon for real imaging studies.

Beirut explosion – neural network16X super slow motion byTringe:

https://www.youtube.com/watch?v=VMOZ__aMYKg

You can clearly see interpolation creating morph data from lower frame rates and extrapolating shapes that look more like a fluid animation than a realistic event.

Here is another example of a similar scene at 300fps interpolated:

Beirut Explosion 4K 300 FPS Super Slow Motion (with DAIN Frame Interpolation) by TimeMachine:

https://www.youtube.com/watch?v=58T8mxjJEFM

The sample looks much better and there is more detail to be seen but it still suffers from interpolation artifacts that create data that would not be present if the event was recorded with a real slow motion camera at 300fps.

There can be examples where interpolation works better like in the Saturn V rocket below at 96fps doubled from 48fps original which is not that taxing:

4K 96fps | Apollo 11 Launch 70mm – DAIN App Video Enhance AI by Neural Networks and Deep Learning:

How do I do it?

The DAIN App is available for Windows at the moment and can be downloaded and used with an Nvidia CUDA card which is needed for the neural network to run. It can take several hours for a single second to process at even 720p so if you thought you could batch convert your footage in 1080p to 60p from 24p and have it by breakfast you will need to re-examine your priorities.

The video below by GreenBox has an excellent walkthrough on how to install and use the DAIN software which is not that difficult if you have the hardware and time to render:

How to Create Slow Motion Videos with DAIN APP | AI Frame Interpolation by GreenBox:

Files that you need besides an Nvidia card and a Windows PC:

• Download Python Here: https://www.python.org/
• Download DAIN APP Here: https://grisk.itch.io/dain-app
• Make sure you support the Dain App developer with some pay at the download site!

One thing to keep in mind is that the software takes enormous resources and requires massive video RAM.  You can trick it by doing smaller tiles that your machine can chew but the calculation will take much longer than if you had the complete footage at the time of analysis.

The other is that the software can only do 2x, 4x and 8x interpolation of frames so if you wanted to have 1000fps from your 30p footage you will need to reprocess the result several times as the first step at 8x will yield only 240fps or 8x slow motion from 30p.  You will then have to reprocess that 240p footage to 4x to yield 960fps.  Expect to wait days for it to finish if it doesn’t crash.  The problem then becomes you are generating 4x more frames from extrapolated data which will be prone to more artifacts and less detail. Kind of reminds us of the videotape generation copy corruption example where the number of copies of a tape would yield exponentially worst results that more copied instances you generated which made it hard to distribute quality tapes.

You essentially needed the best data available called a Master Tape as your source and a secondary tape as your copy.   The problem then became the deterioration of the master tape the more times it was played to be copied requiring after a while a new master tape.

We found a great example of massive interpolation using a Chronos 1.4c Slow motion camera:

Chronos 1.4 Tennis Ball 14,348 FPS DAIN INTERPOLATED from 3,587 FPS by Beyond Human Vision:

So we get an 800*480 pixel video upscaled to 3k and while it was shot at 3,587fps it was interpolated at 4x with DAIN to yield a mind-bending 14,348fps.

DAIN upscaling is very good but it can’t produce miracles from such a low data input source.  You can see pixel deformations and warp artifacts if you look for them but for the untrained eye, it looks like a fluid and beautiful video.

What is the conclusion?

There is no question that DAIN is creating incredible output from near garbage input when it comes to frames. But as with our previous assessment of interpolation, there is still no magic good enough to compete with a  real slow motion camera if you want high frame rates that pass that of normal cameras.

It seems the input quality is directly connected to the best results which means that the more good data you feed the app, the better result you will get.  This means if you need an 8,000fps camera you should get a 1,000fps slow motion camera and try DAIN interpolation to see what you can do with it.  We tried to create some sample footage of our own but our hardware was not pumped up enough to allow such conversion from super slow motion footage without crashing.  Yes, we need to upgrade or lowly 12 core PC with 980 Nvidia card if we want to try this stuff. You need CUDA 5+ support and lots of VRAM.  Recommended cards at Amazon are Nvidia RTX kind as a good start.

So interpolation can work depending on scene complexity and subject, but if you want the best quality and real scientific valid data, you need a real slow motion camera, the good news is that price has been lowered on these, see our guide here, and that now you could, in theory, buy a 1080p 1000fps camera and get DAIN to get up to 8,000fps and upscale the video to 4k with something like Topaz Video Enhance to get performance comparable to a Phantom 4k camera for the untrained eye.  How good could that be really? Well, why don’t you try it yourself and share your results so we can share them? – HSC