This study propose a method for automatically detecting thermal anomalies in the building envelope. The proposed framework consists of metadata extraction, visible to thermal image registration, wall segmentation, and thermal anomaly detection. The performance of the proposed framework was verified through a case study. The case study results show that only the wall area of a building can be segmented from a visible image through supervised learning. In addition, it shows that clustering is possible between similar temperature distributions through unsupervised learning, and the temperature criteria for detecting thermal anomalies can be automatically set based on the temperature between clusters.