Subjective Image Enhancement

In image processing, some objective quality criteria are usually used to ascertain the goodness of the results (e.g. the image is good if it possesses a low amount of fuzziness indicating high contrast). The human observer, however, does not perceive these results as good because his judgment is subjective. This distinction between objectivity and subjectivity is the first major problem in the human-machine-interaction. Another difficulty is the fact that different people judge the image quality differently. This inter-individual difference is also primarily due to the aforesaid human subjectivity.

Following, an overall enhancement system will be described briefly. The approach is based on the combination of differently enhanced images obtained by using different algorithms each satisfying the observer's demand only partly. The fusion result should meet the subjective expectations of every individual observer.

An Overall System for Image Enhancement
The proposed enhancement system [1-5] consists of two stages: an offline stage in which an aggregation matrix will be generated (Figure 1) which contains the relevancy of different algorithms for corresponding observers, and an online stage where new image data will be enhanced and fused for a certain observer (Figure 4).

Figure 1. The offline stage of the overall system. Using test images, the observers judge the results of the selected algorithms. The result of this stage will be an aggregation matrix ¡ containing the relevance of each algorithm for each observer (adapted from Tizhoosh, H.R. (2000): Fuzzy Image Enhancement: An Overview. In: Kerre, E., Nachtegael, M. (Eds.): Fuzzy Techniques in Image Processing, Springer, Studies in Fuzziness and Soft Computing, pp. 137-171, 2000, ISBN: 3-7908-1304-4).

Offline Stage
The offline stage consists of five phases: image enhancement by means of different algorithms (or by just one algorithm with different parameters), extraction of the objective quality criteria, learning the fuzzy measure (subjective quality evaluation), aggregation (regarding to different images and different observers), and finally, a fuzzy inference (final quality measure for each image). The result of the offline stage will be an aggregation matrix containing the relevance of all involving algorithms for each observer. The system phases can be briefly described as follows:
Phase 1 (enhancement): Different algorithms Ak (or one algorithm with different parameters) enhance all test images Xi and deliver their results X'i,Ak. The selection of these algorithms is dependent on the image quality that we are interested in, e.g. contrast, smoothness, edginess etc. At least two algorithms, or two different parameter sets for the same algorithm, should be selected.
Phase 2 (extraction): Depending on the specific requirements of the application, suitable quality measures h(X'i,Ak) are extracted, e.g. contrast, sharpness or homogeneity measures. These criteria can serve as objective quality measures and will be aggregated with subjective measures in the forth phase via fuzzy integral.
Phase 3 (Learning): The observer judges the quality of all enhanced images. The images are presented to the observer in random order. Moreover, the observer is not provided with any information about the algorithms used in the first phase.In order to map the subjective assessments into numerical framework, the ITU recommendation BT 500 can be used. The quality of the images generated by the k-th algorithm as excellent (=1), good (=2), fair (=3), poor (=4) and bad (=5). For all M judgments pi,b of the b-th observer, the MOS (mean opinion score) will be calculated:

Phase 4 (Aggregation of measures/judgments): Considering the objective measures and subjective judgments, one recognizes two conflicts. First, the observer judges the results of the same algorithm from image to image differently. Second, considering the divergence between objective and subjective assessments, the relevance of different algorithms is not always obvious. To solve these problems two new measures the degree of compromise m* and the degree of compatibility g are introduced (Figure 2).

Figure 2. Aggregating of subjective and objective measures, the degree of compatibility and the degree of compromise are calculated.

Phase 5 (Inference): The elements of vectors G (degree of compatibility) and F (degree of compromise) are fuzzified with three membership functions (Figure 3). The output of the inference system is an aggregation matrix quantifying the image quality and is represented by five non-symmetric membership functions. The if-then rules are formulated heuristically as follows:
IF gi is HIGH and m* is MEDIUM,
then relevance Y¡(b, k) is HIGH.
The aggregation matrix ¡ indicates how relevant the k-th algorithm is for the b-th observer.

Figure 3. Membership functions for the inputs (left) and output (right) of the inference.

Online Stage
In the second stage (Figure 4) the system uses only the information stored in the aggregation matrix and an index indicating the current expert looking at the images. The fusion of the sub-results can be achieved in the simplest way as follows:

Figure 4. The online stage of the overall system. The observer-depended values from aggregation matrix are used to generate the appropriate image by fusion of sub-results (adapted from: Tizhoosh, H.R. (2000): Fuzzy Image Enhancement: An Overview. In: Kerre, E., Nachtegael, M. (Eds.): Fuzzy Techniques in Image Processing, Springer, Studies in Fuzziness and Soft Computing, pp. 137-171, 2000, ISBN: 3-7908-1304-4).

In order to verify the system performance, 80 images were used (40 images for learning in the offline stage and 40 images for testing in the online stage). Eleven observers had to judge the quality of 400 images resulting in 4400 MOS values. The results are presented in following table. The MOS values are averaged for all observers.

Table. The results of tests for sharpness as interesting image quality (11 observers, 4 algorithms, 80 images).

Four different algorithms were used to modify the image sharpness/smoothness:

The fusion of sub-results for each observer is accomplished by considering the spatial information:

In Figure 5 outputs of the overall-system for two different observers are illustrated. The relevancy of the algorithms A1 to A4 using the corresponding kernels in the equation (13) is given as follows:

First observer (left image in Figure 5): [0.26 0.29 0.16 0.29]
Second observer (right image in Figure 5): [0.10 0.27 0.35 0.28]

Figure 5. The results of edge enhancement using the overall system for two different observers.

References