The first step in establishing a retrofit strategy for an existing building is to identify the type of thermal defects in the building envelope. Infrared thermography is mainly used to detect thermal defects. However, the diagnosis results are subjectively influenced by the auditor’s experience. This study proposes a method for classifying thermal defects into material-related thermal bridges, geometrical thermal bridges, air leakages, and other thermal defects via thermal and visible images. To verify the performance of the proposed method, a field experiment was performed on a building in which thermal defects occurred. The results of the field experiment showed that the F-scores of the proposed method were 0.9707 for air leakage, 0.9000 for a material-related thermal bridge, 0.9775 for a geometrical thermal bridge, and 0.9228 for other defects. The results of this study show the potential for automatically classifying various types of defects that occur in building envelopes.