What is HDR Deghosting?

HDR (or HDRI) is short for "high dynamic range image". Images we see on our computer screens range in brightness from 0 to 255, but there are many more intensities of light in the real world. This is why when we take photographs of magnificent scenes, like sunsets, it often turns out dull. High dynamic range images contain a higher range of light. The values HDR images can be processed by tone mapping to be viewed on regular monitors.

To capture an HDR image with a regular camera, one can take several pictures of the same scene, adjusting the shutter speed with each exposure, resulting in a series of pictures from dark to light. An HDR image can be made by combining the well-exposed pixels from the source images. However, there are frequently moving objects (such as people) in the scene, causing each picture to be inconsistent with the rest. This results in ghosting - objects appear semi-transparent. The process of removing the semi-transparency is called deghosting.

Favor Choosing Pixels from the Same Image

Since Khan's algorithm works on a pixel level instead of an object level, there are often times when objects get sliced because parts of them stayed in a consistent location while other parts moved. In other words, if part of the object consistantly overlaps with a pixel, then it is chosen, but if part of it isn't in a consistent location (think: stand still and shake your head), then it is not chosen.

To encourage staying within the same image, for each pixel, I took the previous weight of the neighbors on the same layer and multiplied the newly calculated weight by that. This way, if a pixel's neighbors had high weights then this pixel is more likely to have high weights.

Below is the basic result from 5 iterations, with multi-scaling, joint bilateral upscaling, and all-or-none weighting (choosing from a single image as opposed to blending). As you can see, the people on the right are being partially eliminated because they don't cover the floor or the wall consistently enough. Even the guy in the middle of the left group has some ghosting because he didn't stay in the same position.

Here is the new result: 5 iterations, with all of the above, while favoring staying in the same image

It seems that favoring the same image solves some problems, and creates others. The lady on the right is now almost complete, and the massive ghosts that appeared before are mostly gone. However, there are a lot more people in front of the church door, and the guy in the red shirt seems to have been duplicated multiple times. Also, there is a very clear "dent" to the man in the middle of the 3-person group. And he seems to have developed 4 legs.

The above result used a 3x3 neighborhood, and it would probably would benefit from having a larger number of neighbors, but there are 2 issues that make that harder than it sounds. One is that right now favoring the same image is embedded into each iteration of the Khan algorithm. If the neighborhood for this was different than the neighborhood used for Khan, then it will have to be processed separately, thus adding greatly to the run time of the program. Another problem, or rather a lack of incentive, is that Khan algorithms results differ very minimally whether we use a 3x3 neighborhood or a 9x9 neighborhood, but the run time increases significantly. Therefore I don't really think it'll be worth it to increase the neighborhood size to the whole algorithm.

1 comment:

Giuseppe Rota said...

I just stumbled upon this blog: keep up the good work Jing!