Automatic Correction of Internal Units in Generative Neural Networks

Automatic correction of internal units is a retraining-free framework to improve the quality and reliability of the deep generative neural networks which are models trained to generate various images. This method is a crucial element of the generative model for mission-critical applications.

Article | Fall 2021

Prof. Jaesik Choi’s research team developed a framework for the automatic correction of internal units in generative neural networks. The project was led by researchers of the KAIST AI graduate school Ali Tousi and Haedong Jeong (advisor: Jaesik Choi). This research was published under the title “Automatic Correction of Internal Units in Generative Neural Networks” at the Conference on Computer Vision and Pattern Recognition (CVPR) 2021, a top conference in computer vision.

Deep generative neural networks (DGNNs) are models which are trained to generate various images. DGNNs have become increasingly powerful in terms of producing photo-realistic images, which are often hard to distinguish from real samples. However, DGNNs still suffer from producing outputs that contain unrealistic regions called artifacts, which make them unsuitable for being employed in mission-critical applications. As a result, examining the root cause of such phenomena and possible solutions to enhance the overall quality is crucial.

To enhance the reliability of DGNNs, the research team focuses on the internal units in the networks, which cause the low visual fidelity of outputs. Although the retraining of the entire network can be an intuitive solution to remove the artifacts, such a method is not only expensive in terms of computational costs, but also hard to guarantee a proper repair without hurting plausible regions of the original generations.

The research team has proposed an automatic correction method which can be applied to the various structures of DGNNs. The proposed method comprises two stages: (1) identifying the defective units which cause the artifacts and (2) ablating the detected units in the consecutive internal layers. As demonstrated in Figure 1, manually annotated generations (Normal and Artifact) and randomly sampled real images are used to train the classifier. The research team extracts the artifact mask by applying an explainable AI technique and calculates the defective score for each internal feature map unit of DGNN. The research team corrects the artifact generations by removing the detected internal units based on the defective score in the consecutive internal layers.

Figure 1. Identification of the artifact units for each layer (top). The artifact mask (red region) is used to calculate the defective score for each internal unit. The generation flow for two correction methods (bottom). The crosses mean removed the internal units in DGNN.

The research team believes that the proposed method can be an effective tool to enhance the reliability of DGNNs for real-world applications. More information can be found in the following links.

[Links]
(1) Paper link: https://openaccess.thecvf.com/content/CVPR2021/html/Tousi_Automatic_Correction_of_Internal_Units_in_Generative_Neural_Networks_CVPR_2021_paper.html
(2) Project page: http://sailab.kaist.ac.kr/automatic-correction/