29 August 2011

Introduction

Most videos that are shot with a handheld camera are subject to shaky movements that are larger than the built-in image stabilization - that is found in most modern digital video cameras (see en.wikipedia.org/wiki/Image_stabilization) - can handle. One option is to use an external camera stabilizer (such as "Steadicam"), but most of them are expensive and bulky to carry around, so they are usually not an option unless you are a serious video hobbyist. One exception, that we would like to mention, is the Hague MMC Mini Motion-Cam Camera Stabilizer, which is available for about $112 on eBay. It is suitable for light-weight cameras only and it is not too bulky to carry (see demo at cheesycam.com/canon-t2i-camera-stabilizer-video). It probably wouldn't work in all situations, however, such as filming through the windshield of a driving car.

In any case, if you have shaky video footage all isn't lost. There are a number software programs that can reduce the shakiness and significantly improve the video. Some of them are part of expensive video editing programs, some can be purchased as plugins for popular video editing programs, and some can be purchased as stand-alone programs. Gunnar Thalin's Deshaker, however, is available for free (as "donation-ware") as a plugin for VirtualDub, a widely used free video processing program. Many of those who have tried the Deshaker in addition to other video stabilizers claim that the free Deshaker is as good or better than the commercial programs. The Deshaker plugin is available at www.guthspot.se/video/deshaker.htm.

Gunnar's page includes a guide for the use of the Deshaker and there are some links to external guides as well. We recommend that you read through one of these guides and then read the instructions for the newest version, Deshaker 3.0, on the Deshaker home page. Version 3.0 was published on Aug 14th, 2011. We have used older versions of Deshaker in the past, and the new version 3.0 contains some significant improvements and changes to the configuration menu. One of the problems that existed in older versions was that of "overshooting" (see forum.doom9.org/archive/index.php/t-49597.html). We believe that this problem was caused by either (1) the inability of Deshaker to correctly detect scene changes, or (2) a bug that caused Deshaker to extend the smoothing across scene changes with certain Pass 2 settings.

The second problem has been fixed and the scene change detection has been substantially improved. Ideally one ought to deshake each scene separately, as different scenes with different characteristics might require different settings, and some scenes might not need any deshaking at all. This can quickly get very cumbersome and time consuming. We typically have an average of 3 scene changes per minute of video, which means that we would need to process about 30 files for a 10 minute video. The ideal situation for our videos is to find some settings that work well for all scenes and then deshake the total video file. This requires a close to perfect scene detection for an optimal result.

General usage


In addition to the detailed user instructions found on the Deshaker home page and the referenced guides we would like to add a few comments. We use Sony Vegas Pro 8 to assemble and edit our videos and frame serve the result to VirtualDub, using the Debugmode Frame Server, which is a free plugin available at www.debugmode.com/frameserver. Alternatively, you can output an AVI file from your editing program and take that as input to VirtualDub. Using the frame server saves some time. If you want to use titles on top of your shaky video, you need to deshake the video before adding the titles, or you will find that the titles bounce around in the final deshaken video. To save time you may use one of these alternatives: (1) Apply the titles to a uniform background or a still picture (e.g. the first frame of the following scene) before starting the scene video, or (2) apply the titles in VirtualDub during Pass 2 of the deshaking process. The title files should be TGA files with a transparent background and you use the Blend feature in VirtualDub to make the titles appear where you want them.

On the Deshaker page it is mentioned that you can save some time during Pass 1, if you use "Run video analysis pass" instead of "Preview filtered". This is probably the case, if your computer is able to run the analysis pass at a higher frame rate than the playing rate of the video. Our videos are typically 848 x 480 pixels at 30 fps, and our computers cannot run Pass 1 as fast as 30 fps. We have found that "Preview filtered" is about 10% faster than "Run video analysis pass". Set Audio to "No audio".

If you use the edge fill-in feature during Pass 2 and you are using version 1.9.9 or newer of VirtualDub you will not lose any part of the video (with older versions you would lose the last 30 frames when using a 30-frame fill-in unless you appended a clip in the end) and the audio is supposed to be in sync. With Version 2.7 we found that the audio was delayed, but this problem appears to have been fixed in Version 3.0.

The deshaking process may result in a slight softening of the video, so we have found it useful to add a bit of sharpening after the Deshaker filter. We usually apply a setting of 8 with VirtualDub's internal sharpen filter. Once you have found the ideal settings for your videos, including all filters as well as video and audio compression, you should save those settings (File>Save processing settings). That makes it very fast in the future, as you just need to open the video and load the processing settings before performing Pass 1 and 2 (remember to save the processing settings with the Deshaker configured to perform Pass 1). One thing you may want to add on a case by case basis are titles. Use the internal logo filter to add a TGA file, enable alpha blending and use premultiplied alpha to get the transparency in the TGA file; then use the Blend filter to blend in the title where you want it to appear on the video (View>Curve editor).

In general our results have been very good. The Deshaker works extremely well on footage shot from a car which is often very shaky. It also works well for general handheld filming. If you walk while filming, however, it often results in deshaken video that exhibits waves or ripples along the edges. This may be worse in our case as we use a wide angle video camera (corresponding to a 25 mm lens). Another problem that we sometimes experience is a jittering or flickering appearance of the video in areas with many vertical or horizontal lines. This may be due to blurriness in the original footage and it might be alleviated by using a faster shutter speed. If any of you know how to overcome these problems we would like to hear about it.

Deshaking a one-scene video


The default settings for Pass 1 work pretty well for most videos, so we recommend that you start out using those, except that you uncheck "Detect scenes" for a one-scene video. You may want to change the name and location of the Deshaker logfile. If you think you may return to a particular video to run Pass 2 with different settings, you should give the logfile a unique name, so that it doesn't get overwritten by subsequent Pass 1 runs. This can save you a lot of time, since you won't need to run Pass 1 again, just specify the right logfile before running Pass 2.

For Pass 2 you have some choices to make. As the Deshaker moves, rotates and zooms your video to remove the shakiness, black borders will appear around the edges. You can apply a fixed zoom or an adaptive zoom to remove those. Fixed zoom has the disadvantage of being applied to the entire video, so you will lose part of the picture even if there aren't any black borders. For that reason we prefer to use adaptive zoom. In the "Adaptive zoom full (no borders)" setting the video is zoomed in to remove all black borders. This may be a good choice if your video has a fairly low level of shakiness. If your video has passages that are very shaky you should choose "Adaptive zoom average (some borders)" and then apply "Use previous and future frames to fill in borders". That way you lose less of your picture. The default values of 30 previous and future frames usually work well. If you still see some black borders in the deshaken video you can try to increase these values. You may also want to check "Extrapolate colors into borders" as that will usually make an unfilled border area much less noticeable than when it's black.

The filled-in border areas are not always perfectly aligned, but it is usually pretty good, and we prefer to have slightly fuzzy borders to losing a large part of the picture, but that is a matter of taste, of course. For less shaky videos you could also use edge compensation "None (large borders)" and then apply fill-in borders and extrapolate colors. Then you wouldn't lose very much of your picture.

For the remaining settings, including the smoothness parameters, the default values usually work very well. However, in videos where you move forward or backward, such as filming through the windshield of a car or when walking forward you may want to turn off the zoom smoothness (set the value for Zoom under Motion smoothness to 0). The Deshaker perceives moving forward or backward as a change in zoom which may result in a pulsating zoom in the deshaken video. If adaptive zoom is used the adaptive zoom smoothness should be kept (default 5000).

Here is a demo that shows a 1-minute video that has been deshaken using the default Pass 1 and Pass 2 settings except for the edge compensation. There is no audio:



This example should give you a good idea of how the Deshaker works. As you can see in video D about 13 seconds into the video, the fill-in sometimes gives you some strange borders, but most of the time the borders look fine. In video E we avoid the fuzzy borders, but we are losing a rather large part of the picture in passages with large corrections. It is possible to limit the amount of the corrections by (1) reducing the motion smoothness parameters, or (2) reducing the max correction limits. In both cases the deshaking will be less efficient, but the black borders will be smaller, so it will be a trade-off.

Video F shows an example of the waving/rippling effect that often occurs in video that is filmed while walking. The video has two overlaid titles, with the upper one added before deshaking and the lower one added with the logo filter in VirtualDub after the Deshaker filter. The TGA file used for the lower title was prepared in Gimp. Video G is also filmed when walking and shows the ripple effect as well as some jitter in the vertical lines of the banyan tree. This may be due to the shutter speed being too slow. The jitter is not visible in the original footage.

Deshaking a multi-scene video


In a multi-scene video you generally don't want the deshaker smoothing to extend across scene changes as each scene usually has different characteristics. If you do, you may see a scene end or start with a zoom in or out effect that did not exist in the original video. So, in general you want the Deshaker to know all the correct scene changes before starting Pass 2. If you use overlapping transition effects (such as fade-in/fade-out) the Deshaker will usually not be able to detect the scene change. In such cases you may or may not want the Deshaker to smoothen across the scene change. If you want the smoothing to be reset at the middle of the transition we have some tips below of how to do that.

If you use a fade-out to black followed by a fade-in for your scene transition the Deshaker will detect the scene change, as it considers a blank frame an indication of a scene change. This is in our opinion the biggest problem with the otherwise much improved scene detection used in Version 3.0. If you have a dark video it may easily contain frames that are detected as "blanks" by the Deshaker. This will result in a lot of spurious scene changes in the logfile. We will present a way to fix this problem.

A smaller problem with the Version 3 scene change detection is the fact that it will only detect scene changes if the first frame of the new scene is also a "skipped" frame, which means that the %OK blocks is lower than the limit specified in the Pass 1 configuration parameters. This is good in the sense that it limits the number of spurious scene changes in the logfile, but it is bad in the sense that you cannot set a %OK block limit for scene detection that is independent of your "skip limit". In other words, if you wanted to skip using frames that have less than 8% OK blocks, but you want the scene detection to accept a scene change if the scene has up to 15% OK blocks, there is no way to specify that. We will present a way to do it below. By the way, Version 3 will now apply smoothing for skipped frames (that are not new scenes) by interpolation, which is much better than previous versions where skipped frames were always considered to be new scenes.

Apart from these problems the Deshaker 3.0 scene detection works really well, so in most cases it will detect the majority of your scene changes and - most importantly - it will rarely add any spurious scene changes (except for dark videos). If you are happy with that you may skip the rest of this post. But if you are a perfectionist you probably want all the scene changes to be correct, and if you have long videos to render it will save you a lot of time to have the scene changes correct the first time you run Pass 2. The following section will provide you with a method to do that. It may look a bit complicated at first, but it is in fact very simple and quick to do once you have tried it a couple of times. To make it easier we provide a download link for all our scripts at the end of this post.

We will also present a method that allows you to modify the logfile so that you can omit smoothing the zoom or omit any smoothing at all in one or more scenes of your video.

Using awk scripts


In the following we will analyze and modify the Deshaker logfile. This can be done in a text editor or in a spreadsheet like Excel, but that is a slow and cumbersome process. We use a powerful text processing utility called awk, which comes from the UNIX world. A version that works under Windows is available at www.simtel.net/product/view/id/51371. Download the file called awk320.zip. This file is also included with our scripts which you can download via the link at the bottom of this post. Unzip it and place the contents in C:\awk (create folder awk under C:\; you could place it anywhere but we use this folder to simplify the instructions). When you configure the Deshaker filter change the location of and the name of the logfile as shown on the left:


That will make it easy to process the logfile with awk. To use awk you open a Command Prompt window and change the directory to awk, as shown above on the right. You are now ready to use awk to analyze and modify your Deshaker logfiles.

Analyze and modify the Deshaker logfile


The default settings for Pass 1 work pretty well for most videos, so we recommend you start out using those. Note that the "Skip frame if < x% of all blocks are OK" is set at 8%, "Detect scenes" is checked and the "Threshold" is set at 20. When you have run Pass 1 you can look in the logfile to check whether the Deshaker has found all your scene changes and whether it has added any spurious scene changes. It is, however, much easier to find the scene changes using awk by executing this command:

    awk "/n_scene/ {print NR}" D.log     (listscns.bat)

This will output the frame numbers of the scene changes to the command window. If you want to output to a file you execute:

    awk "/n_scene/ {print NR}" D.log >scenes.txt     (getscns.bat)

and your scene change frame numbers will be found in file scenes.txt in your C:\awk folder.

To check whether the Deshaker has correctly identified your scene changes you need to know the frame numbers of the actual scene changes. You can find those in VirtualDub by going through the video, but that is a pretty cumbersome process. It is usually easier to find them in your video editing program. If your video is DV video, many editors can find the DV video stop/start time markers which usually indicate scene changes. If your video is composed of multiple files with a separate scene in each file (like ours), you may step through the video file by file and note the frame numbers. See the section on Sony Vegas below for an easy way to find the scene changes if you are using that program.

Once you know the correct scene change frame numbers you can edit the scenes.txt file and then update the Deshaker log file with this command:

   awk -f fixDlog.awk scenes.txt D.log >Dfixed.log     (fixDlog.bat)

The "-f fixDlog.awk" part means that awk looks for the program script in file fixDlog.awk which must contain this script:
BEGIN {i=1;j=1}
{
 if(NR==FNR){a[i]=$1;i++}
 else {
  if(a[j]==FNR){$0=(substr($0,1,8) "n_scene");j++}
  else {if($2=="skipped"){$0=""}
        else{$0=substr($0,1,39)} }
  if($0!="")print $0;
 }
}
This script will enter the scene changes from file scenes.txt into D.log and remove all other scene changes that may be found in the file, including any spurious scene changes such as blank frames that aren't actual scene changes. Lines that are just "skipped" will also be removed, but that is OK as a missing line is treated the same way as a skipped frame. With this method you don't have to worry about adjusting the scene detection threshold or the skipped frame threshold, as you have now made sure that the scene changes in the logfile are correct. This also means that you will be free to use any %OK blocks you want for skipping frames without having to consider if that limit is ideal for scene detection. If you have an easy way to generate the scenes.txt file you can actually skip the first part (getting the scenes from the logfile) and go right to the command that fixes the logfile. You could even run Pass 1 without scene detection. See the Sony Vegas section below for an easy way to generate a scenes file using this program.

If you want to have some scenes excepted from deshaking or zoom smoothing you can add the words nodeshake or nozoom in the scenes.txt file like this:
1 nodeshake
368
548
810 nodeshake
1234
1804 nozoom
2284
2565
The first scene is not normally listed in the scenes.txt file, but you can add a line with the number 1 if you want this scene excepted. You then execute this command:
awk -f modDlog.awk scenes.txt D.log >Dmod.log     (modDlog.bat)
with modDlog.awk containing this script:
BEGIN {i=1;j=1;f=0}
{
 if(NR==FNR){a[i]=$1; b[i]=$2; i++}
 else {
  if(a[j]==FNR){f=0;$0=(substr($0,1,8) "n_scene");
  if(b[j]=="nodeshake")f=1;
  if(b[j]=="nozoom")f=2;
  j++}
  else {if($2=="skipped"){$0=""}
        else{if(f==0)$0=substr($0,1,39)
             if(f==1)$0="";
             if(f==2)$0=(substr($0,1,32) "1.00000")}
  }
  if($0!="")print $0;}
}
This script will remove the lines for a scene labeled nodeshake, and it will change the zoom parameter to 1.00000 for all frames in a scene labeled nozoom. The latter may cause the borders to become a bit more fuzzy when using edge fill-in, as this function uses the zoom parameter to align the fill-in, but it is usually not very noticeable. If you use "Adaptive zoom full" you will avoid this issue.

In summary, you will need to perform the following steps if you want to get the best possible result from deshaking a multi-scene video:
1. Run Deshaker Pass 1 to output logfile D.log in folder C:\awk.
2. Establish a list of the actual scene changes in file scenes.txt. There are two ways to do that:
    a) Find the scene changes in the logfile with command getscns.bat and edit the resulting file, scenes.txt, or
    b) Create the list of actual scene changes in file scenes.txt in your video editing program or another program like DVdate.
3. If needed, edit scenes.txt by adding nodeshake or nozoom tags to scenes of your choice.
4. Run command modDlog.bat to modify the logfile, so that it contains the correct scene changes only as well as any modifications from step 3.
5. Run Deshaker Pass 2, changing the logfile to Dmod.log. Remember to set your choice of edge compensation.

Scene detection with awk scripts


With the approach described above the accuracy of the Deshaker 3.0 scene detection becomes academic, but for the record we shall list our results for about 90 minutes of video in 8 files with a total of 280 scene changes:


MissedSpurious
Deshaker 3.0:594(using the default settings)
Awk script 1:51
Awk script 2:01
Awk script 3:02
Awk script 4:10
The videos contain one dark scene, and it is in this scene that all the spurious scene changes appear. 93 out of the 94 are detected as blank frames. This can be seen by inspecting the logfile, but it can also be ascertained by running awk script 1 on the logfile. It mimics the default limits for scene detection value (20) and for %OK blocks (8), but it will miss the blanks. With an awk script you can quickly test what results you would get if you changed the thresholds. In awk script 2 we have changed the %OK blocks to 15 which result in the capture of all the scene changes. Using this limit in Pass 1 would not be a good idea, as it would result in a lot more skipped frames. Using the awk script allows us to decouple the scene detection from the skipped frames.

If we load the logfile into an Excel spreadsheet and chart the scene detection values and the %OK blocks there is one characteristic that stands out: The scene changes are characterized by a clear spike in both values (one up and one down) whereas the absolute values vary a great deal. Here is an example:

Scene detection values below and %OK blocks at the top, offset by 300.
Based on this observation we tried awk script 3 which looks for spikes larger than a given size in both values, but which does not look at the absolute levels. The results were good, with 0 missed scene changes and just 2 spurious. Instead of using separate limits for the two spikes we tried to apply a threshold to the product of the two spikes (awk script 4). Here is a graphic representation of what it looks for:

%OK block spike versus scene detect value spike. Points with purple dots are actual scene changes.
The script looks for values above the red curve which represents a product of 800. This script just missed 1 scene change and gave no spurious scene changes. It would be interesting to check this approach for all kinds of different videos.

If any of you want to try the scripts we would like to hear of your results. Please characterize your video(s). The scripts are listed below. They are run with a command like this:
awk -f script.awk D.log >scenes.txt
You may, of course change the limits or even the formulas in the scripts. The variables $13 and $14 contain the scene detection value and the %OK blocks (multiplying by 1 converts a string value to a number).
Awk script 1:     (getscns1.awk, command file: getscns1.bat)
BEGIN{FS="\t"}
{if($13*1>20 && $14*1<8)print NR;}
Awk script 2:     (getscns2.awk, command file: getscns2.bat)
BEGIN{FS="\t"}
{if($13*1>20 && $14*1<15)print NR;}
Awk script 3:     (getscns3.awk, command file: getscns3.bat)
BEGIN{FS="\t"}
{
NS3=$13*1;OK3=$14*1;
if(NR==1){NS1=NS3;OK1=OK3};
if(NR==2){NS2=NS3;OK2=OK3};
if(NR>2){
  if((NS2-NS1>30 && NS2-NS3>30) || (OK1-OK2>40 && OK3-OK2>40))print NR-1;
  NS1=NS2;NS2=NS3;OK1=OK2;OK2=OK3;}
}
Awk script 4:     (getscns4.awk, command file: getscns4.bat)
BEGIN{FS="\t"}
{
NS3=$13*1;OK3=$14*1;
if(NR==1){NS1=NS3;OK1=OK3};
if(NR==2){NS2=NS3;OK2=OK3};
if(NR>2){
  NSMIN=NS2-NS1;if(NSMIN>NS2-NS3)NSMIN=NS2-NS3;if(NSMIN<0)NSMIN=0;
  OKMIN=OK1-OK2;if(OKMIN>OK3-OK2)OKMIN=OK3-OK2;if(OKMIN<0)OKMIN=0;
  if(NSMIN*OKMIN>800)print NR-1;
  NS1=NS2;NS2=NS3;OK1=OK2;OK2=OK3;}
}


For users of Sony Vegas


We have applied the following in Sony Vegas Pro 8, but it will most likely work in other versions as well.  We will first assume that each scene exists on the Vegas timeline as a separate video file.

In Sony Vegas there is a little known keyboard shortcut that makes the time cursor jump to the next file: Press 9 in the numeric keypad.  Set the time format to Absolute Frames. You can now note down the frame number for each scene change. If you use overlapping transitions the time cursor will jump first to the end of the current file and then back to the beginning of the next. If you want to use the mid point as the scene change you should just take the average of the two numbers.

There is an even better method that can output the video file transition points to a file:
1. Save the Vegas project as an EDL file, let's use Vedl.txt as an example. Put this file in folder C:\awk.
2. Execute awk -f Vscns.awk Vedl.txt >Vscns.txt     (Vscns.bat)
You will now have the scene change frame numbers in file Vscns.txt. You can use this file to compare with scenes.txt if you have generated this file from the Deshaker logfile, or you can just rename it to scenes.txt and use it to fix the logfile as described above. Vscns.awk should contain this script:
BEGIN{FS="; ";L=0;P=0;FPS=30;}
{
if($11=="VIDEO" && $12!="\"\""){
if($3>0)print int((FPS*($3+P+L)/2000)+0.51);
P=$3;L=$4;}
}
The FPS should be adjusted to the frame rate of your video. If the video files overlap with a transition effect the formula will output the frame number of the transition mid-point.

If the video has been imported from a camcorder in DV format some video editing programs are able to detect the date/time stamps that indicate stop/start of the recording which will usually be scene changes. Vegas is not able to detect these points, so you will need to use another program. One such program is DVdate which is free and available from paul.glagla.free.fr/dvdate_en.htm. It can output a list of scenes in several different formats and you can split a DV AVI file into individual scene files. If you use these files in Vegas, you can use the method described above to get a list of the scene change frame numbers.

Another free program that captures DV video from a camcorder into separate files for each scene is WinDV, available at windv.mourek.cz/. For HDV video a free utility called HDVSplit is available at strony.aster.pl/paviko/.


Download


All the command and script files mentioned in this post can be downloaded from www.netkvik.dk/deshaker/deshaker3.0help.zip. Read the instructions in readme.txt before you proceed.