Since Khan's algorithm works on a pixel level instead of an object level, there are often times when objects get sliced because parts of them stayed in a consistent location while other parts moved. In other words, if part of the object consistantly overlaps with a pixel, then it is chosen, but if part of it isn't in a consistent location (think: stand still and shake your head), then it is not chosen.
To encourage staying within the same image, for each pixel, I took the previous weight of the neighbors on the same layer and multiplied the newly calculated weight by that. This way, if a pixel's neighbors had high weights then this pixel is more likely to have high weights.
Below is the basic result from 5 iterations, with multi-scaling, joint bilateral upscaling, and all-or-none weighting (choosing from a single image as opposed to blending). As you can see, the people on the right are being partially eliminated because they don't cover the floor or the wall consistently enough. Even the guy in the middle of the left group has some ghosting because he didn't stay in the same position.
Here is the new result: 5 iterations, with all of the above, while favoring staying in the same image
It seems that favoring the same image solves some problems, and creates others. The lady on the right is now almost complete, and the massive ghosts that appeared before are mostly gone. However, there are a lot more people in front of the church door, and the guy in the red shirt seems to have been duplicated multiple times. Also, there is a very clear "dent" to the man in the middle of the 3-person group. And he seems to have developed 4 legs.
The above result used a 3x3 neighborhood, and it would probably would benefit from having a larger number of neighbors, but there are 2 issues that make that harder than it sounds. One is that right now favoring the same image is embedded into each iteration of the Khan algorithm. If the neighborhood for this was different than the neighborhood used for Khan, then it will have to be processed separately, thus adding greatly to the run time of the program. Another problem, or rather a lack of incentive, is that Khan algorithms results differ very minimally whether we use a 3x3 neighborhood or a 9x9 neighborhood, but the run time increases significantly. Therefore I don't really think it'll be worth it to increase the neighborhood size to the whole algorithm.