SCC366 2021/2022
HOLISTIC IMAGE MATCHING
Objectives: To learn holistic image matching based on the example application of face image retrieval:
1) Loading multiple imagesin MATLAB.
2) Representing images as vectors in MATLAB.
3) Implementing the sum ofsquared difference (SSD).
4) Using SSD in image retrieval applications.
Note: The accompanying `ImageRetrievalCode.zip’ file contains example codes and datasets: The input
query image (`QueryImage.png’) is stored in the `QueryImage’ directory while the database images
(`Img001.png’ to `Img200.png’) are stored in `DBImages’ directory.
Our goal is to retrieve all database images (in `DBImages’ directory) which match the query image
(`QueryImage.png’). A given pair of images,say `A’ and `B’, are declared to be a match if their distance is
smaller than a predetermined threshold . We will use a fixed number 22,000,000 for the threshold : Our
goal can be stated as finding all database images whose distance to the query image is smaller than
=22000000.
We will adopt the sum of squared difference (SSD) as our distance measure. Since this function is defined
for a pair of vectors, each image (of size HXW) is converted to a vector (of size nX1; n=HW). In our dataset, all
images are 112X92-sized and therefore, n=10,304.
Loading multiple imagesin MATLAB:
Our first step is to load multiple images and convert them into vectors. To facilitate the subsequent
multiple computation of SSD, we will store all database images in a single matrix `DBVecImages’ where
each column vector corresponds to an image.
The MATLAB script `ConvertQueryImage.m’ in `QueryImage’ directory shows how the query image can be
converted to a vector:
2
ConvertQueryImage.m
ImageHeight = 112;
ImageWidth = 92;
%%% Image height and width are fixed at 112 and 92.
VectorSize = ImageHeight*ImageWidth;
%%% `VectorSize’ is the size of vectorised image (VectorSize 1).
QueryImageFilename = 'QueryImage.png';
%%% MATLAB variable `QueryImageFilename’ stores the filename of the query image.
QImg = imread(QueryImageFilename);
%%% loads the image 'QueryImage.png' and stores it as a two-dimensional matrix QImg.
%%% Note: the matrix `QImg’ is in unsigned integer data format (uint8)
%%% which is not suitable for numerical operations, e.g., calculating SSD.
imshow(QImg);
%%% displays the image loaded.
QImg = double(QImg);
%%% converts the uint8 matrix QImg to a double matrix for
%%% subsequent numerical operations.
imagesc(QImg);
%%% displays an image in double format
%%% Note: `imshow’ can show images in uint8 format.
%%% To display an image in double format, we use imagesc function. axis image;
axis off;
QueryVecImage = QImg(:);
%%% `B=A(:)’ converts matrix A to a vector B by concatenating all the columns A.
%%% See `ColonOperatorExample.m’ as discussed shortly.
save QueryVecImage.mat QueryVecImage;
%%% saves the vectorised query image in a MATLAB data file QueryVecImage.mat.
Applying this technique, we will store all the database images in a single matrix. Since there are 200 images in
the `DBImages’ directory, we will use MATLAB’s `dir’ instruction to list all the images in this directory and
convert each image in the `for’ loop. An example usage of `dir’ instruction will be shown shortly
(`ConvertDBImages.m’script).
To store the database images, we will construct a matrix `DBVecImages’ of size 10,304 200: Each column of
`DBVecImages’ will contain a vectorised database image. In MATLAB, a specific column or row of a matrix
can be accessed by using `:’ operator: `A(i,:)’ and `A(:,j)’ denote the i-th row and the j-th column of matrix
A, respectively. Since a column (vector) of a matrix is a vector, `A(:,j)’ is regarded as a single vector while
the transpose of `A(i,:)’ (which is a row vector) is also a vector.
Note: as shown in `ConvertQueryImage.m’ example above, `A(:)’ converts the entire matrix A into a single
vector.
3
` ColonOperatorExample.m’ code shows examples of using `:’ in a matrix. ColonOperatorExample.m
A = rand(3,4);
%%% constructs a matrix (A) of size 3 4 initialized with random numbers.
b = A(:,4);
%%% constructs a vector (b) which is the same to the 4-th column of A.
c = A(2,:);
%%% constructs a row vector (c) which is the same to the 2-nd row of A.
%%% A row vector is not a vector by itself, but its transpose is a vector.
%%% Note, in MATLAB the transpose of a matrix A is denoted as A’.
d = [1,2,3]';
%%% constructs a vector (d) initialized with values 1, 2, and 3.
A(:,2) = d;
%%% replaces the second column of matrix A with the elements of d.
%%% This is how we save a vector as a column of a matrix.
e = [5,5,5,5];
%%% constructs a row vector (e) initialized with 5.
A(3,:) = e;
%%% replaces the third row of matrix A with the elements of e.
f = A(:)
%%% constructs a vector (f) by concatenating all the columns of A.
B = A([1,2],:)
%%% constructs a 2 4 matrix (B) using the first and second rows of A.
C = A([2,3],[3,4])
%%% constructs a 2 2 matrix (C) as using the elements of matrix A
%%% which are contained in rows 2-3 and columns 3-4.
4
Now, we are ready to convert all database images to vectors and store them as column vectors of a matrix:
`ConvertDBImages.m’ in `DBImages’ directory.
ConvertDBImages.m
ImageHeight = 112;
ImageWidth = 92;
VectorSize = ImageHeight*ImageWidth;
ImageFiles = dir('*.png');
%%% extracts information of all files whose filename end with `.png’.
NumDBImages = length(ImageFiles);
%%% `length’ returns the number of elements of the data structure `ImageFiles’.
%%% In this example, `length(ImageFiles)’ is the number of images (200).
DBVecImages = zeros(VectorSize,NumDBImages);
%%% This is the matrix that will store all vectorised images:
%%% Each column of DBVecImages will store a vectorised image.
%%% `VectorSize’ is 10304 (92 112).
for i=1:NumDBImages
DBImg = imread(ImageFiles(i).name);
%%% `ImageFiles(i).name’ contains the filename of the i-th image.
DBImg = double(DBImg);
%%% `DBImg = double(DBImg)’ converts the uint8 data matrix DBImg to double format.
end
%%% to be written by you from here.
…
%%% to be written by you till here.
save DBVecImages.mat DBVecImages;
%%% saves DBVecImages in a MATLAB data file DBVecImages.mat.
Exercise 1
Complete the MATLAB script `ConvertDBImages.m’ by writing the missing lines (marked with red dots
above) and run it: It will generate `DBVecImages.mat’ containing a MATLAB matrix variable
`DBVecImages’. Each column of `DBVecImages’ correspondsto a vectorised database image.
You can verify the successful completion of this step by running `DisplayVecImages.m’ script in
`DBImages’ directory. Ifsuccessful, thisscript will display an image given below.
5
6
Even if you do not complete this exercise, you can proceed to the next exercises. In this case, exercise
3 requires the pre-calculated data files `QueryVecImage.mat’ and `DBVecImages.mat’ provided in
`VectorizedImages’ directory.
Image matching using SSD:
Our next step is to build a function SSD. For a given pair of vectors a and b, the SSD is defined as
Here, the two input vectors are of size n 1. In our face image retrieval example n is fixed at 10,304.
Exercise 2
Write a MATLAB function (‘SSD.m‘) that receives two vectors and returns the SSD of these vectors.
You can verify the successful completion of this step by running `TestSSD.m’ script. If successful, this script
will print out:
Even if you do not complete this exercise, you can proceed to the next exercises. In this case, exercise 3
requires a pre-compiled MATLAB script `SSD.mexw64’ provided in `SSDCode’ directory.
Given `SSD.m’ script (or `SSD.mexw64’ code) and MATLAB data files containing vectorised query
(`QueryVecImage.mat’) and database images (`DBVecImages.mat’), we can immediately calculate the SSDs
of the query and the database images.
Note: the query image is given as a vector while the database images are stored as columns of the matrix
`DBVecImages’. The SSD between the vectorised query image and the i-th image can be calculated by
accessing the i-th column of the `DBVecImages’ as a vector. See
`ColonOperatorExample.m’ for examples of accessing (extracting) column (or row) vectors in a matrix.
SSD=2.862500
SSD=8.806100
7
Exercise 3
Complete the MATLAB script `CalculateQuerySSD.m’ by writing the missing lines (marked with red dots below)
and run it: It will generate `QuerySSD.mat’ containing a MATLAB vector variable
`SSDVal’ which contains the distances between the input query image and the database images.
CalculateQuerySSD.m
ImageHeight = 112;
ImageWidth = 92;
load QueryVecImage.mat QueryVecImage;
%%% loads vectorised query image.
load DBVecImages.mat DBVecImages;
%%% load database images stored in DBVecImages.
[~,NumDBImages] = size(DBVecImages);
%%% For the matrix A, `[MatR,MatC] = size(A)’ returns the numbers of rows and columns,
%%% to MatR and MatC, respectively.
SSDVal = zeros(NumDBImages,1);
%%% allocates memory for a vector variable SSDVal and initializes it with zeros.
%%% to be written by you from here.
…
%%% to be written by you till here.
save QuerySSD.mat SSDVal;
%%% saves SSDVal vector that contains the calculated SSD values.
for i=1:10
display(sprintf('Image%03d: SSD value = %f',i,SSDVal(i)));
end
%%% displays the SSD values of the first 10 database images.
You can verify the successful completion of this step. If successful, this script will print out:
Image001: SSD value = 25168263.000000
Image002: SSD value = 24790550.000000
Image003: SSD value = 29749814.000000
Image004: SSD value = 26188179.000000
Image005: SSD value = 39443834.000000
Image006: SSD value = 24414565.000000
Image007: SSD value = 21459433.000000
Image008: SSD value = 22668506.000000
Image009: SSD value = 22176936.000000
Image010: SSD value = 19669056.000000
8
Exercise 4
Our final exercise is to retrieve all images in the database which have distance (from the query image) smaller
than our threshold =22,000,000 and display them.
Complete the MATLAB script `ImageRetrieval.m’ by writing the missing lines (marked with red dots below)
and run it: It will display the retrieved images.
If successful, this script will display an image given below:
Note: the retrieved images are not sorted: the left-most image is not the closest. One can re-arrange the
images in increasing SSD from the query image.
Conclusion
In this assignment, the examples and exercises were developed to help you to understand how to setup and
perform holistic image matching-based face image retrieval using the sum ofsquared difference (SSD).
ImageRetrieval.m
ImageHeight = 112;
ImageWidth = 92;
Tau = 22000000;
load QueryVecImage.mat QueryVecImage;
load DBVecImages.mat DBVecImages;
load QuerySSD.mat SSDVal;
%%% to be written by you from here.
…
%%% to be written by you till here.
figure(1);
DisplayPatches(RetrievedVecImages', NumRetrievedImages, ImageHeight, ImageWidth,
NumRetrievedImages);
%%% `NumRetrievedImages’ is the number of retrieved images which are
%%% the database images whose distance from the query image is smaller than Tau.
%%% `RetrievedVecImages’ is a matrix of size 10,304`NumRetrievedImages’:
%%% Each column in this matrix contains a retrieved image.