This project is to "implement an algorithm that, given a 3-channel image, produces a color image as output".
My final approach includes gradient(edge)-based Auto-alignment, Auto-cropping, and Auto-white-balancing. The final results are above.
The approximate executing time is 4 minutes for all 5 images.
B&W 1: Gradient(edge) feature using.
B&W 2: Auto-cropping.
B&W 3: Auto-white-balancing.
Basic Implementation
My first idea was to implement an SSD(Sum of Squared Differences) as the score to estimate if the different channels are aligned. My reference channel is Green, and essentially I only needed to align the Red channel and Blue channel to the Green channel. The [-15, 15] searching window worked fine for the .jpg file, but it is not enough for a larger file (.tif). At the same time, increasing window size will drastically decrease the performance of large files. Therefore, I used Pyramid to implement my program. By resizing images smaller, we can obtain an approximate number at each layer, we only need to add them on, and finally, we can get a precise number by calculating and comparing the score of different shifts in a much smaller window, the information of the image in original size is maintained.
However, SSD did not work very well on some images, therefore, I used NCC instead of SSD, which performs much better.
Edge Feature Implementation
In addition, sometimes there will be a large area belonging to one part but very different in all 3 channels, for example, the Emir of Bukhara one. Therefore, the simple comparison between brightness is not working in this case. The solution is to implement a gradient-based method. I used the Sobel operator to deal with edge detection in this case. By cooperating with Pyramid, the result is improved.
Auto-cropping
Also, since the alignment results got some noise edge around the main part of the image that we are concerned about, I also implemented the auto-cropping mechanism. The basic idea is still finding the true border of the image according to the gradient image, and by adding the brightness and normalizing them in both horizontal and vertical directions, we can get the candidate positions that are potentially lying on the true image border.
Auto-white-balancing
Given that the synthesized image very possibly has a biased white-balancing, I implemented my algorithm to adjust the auto-white-balancing mechanism. The general solution to deal with this was finding an area that is trusted to be "white", and we need to make sure that this area should be not biased to either blue, red, or green. Therefore, we can do this searching process on our image, and try to calculate the gain of each channel. The problem is that, if the white area is not biased, while the shadow in the image is not balanced, the result can be trivial if only have the information from the white area. I added a section that deals with the black area's white balance and used a tuned parameter to combine the information both from the white area and the black area. The result is even better than Photoshop(Adobe Phothoshop 2022) Camera Raw Auto white-balancing.
A little trap
One fun fact is, we cannot utilize auto-white-balancing without proper cropping. Images with noisy edges actually interfere with the calculation process of white-balancing. Therefore, the cropping step must be operated before the white-balancing step.
Summary
Overall, the final result performs very well, in terms of the precision of alignment and the color/tone.
