Although several past studies proposed deep learning methods to extract pre-disaster building footprints for post-disaster management using remote sensing techniques, building damage classification after a disaster is rarely discussed. Building damage classification using deep learning is a challenging task because imagery data of damaged buildings are limited publicly and damage levels are hard to judge from images. Most building damage classification papers with deep learning methods only judge buildings as collapse or not. Moreover, most deep learning models are designed for non-damaged building segmentation, which might not be suitable for damaged building segmentation. In order to solve the problems, including lack of imagery data, detailed damage levels, and suitable deep learning model for damaged building segmentation, this research aims to propose and evaluate a novel deep learning model to classify building damage into four levels quickly with in-house labelled damaged building information. This research proposes the Squeeze-and-Excitation dual High-Resolution Network (HRNet) model to be trained with open access xBD dataset and the 2010 Haiti Earthquake images labelled at the University of New South Wales. HRNet is adopted twice in the model because two steps need to be completed, building localization and then damage classification. Four comparative experiments are implemented with seven metrics, including combination F1, localization F1, localization precision, localization recall, damage F1, damage precision, and damage recall. The first experiment is to choose where the best place is to add SE channel attention block in the model. The Squeeze-and-Excitation (SE) block is added at four different places in the dual HRNet backbone to have a comparison with the model without SE. According to analyses of the results, the SE-PRE model has the best performance. The second experiment shows that a larger input size of images increases the processing time but performs better. The third and fourth experiments reveal that the Sigmoid function is better than Hard-Sigmoid in SE, and the transfer learning with pre-trained ImageNet weights may not be suitable for the proposed model. The theoretical contribution is proposing a novel deep learning model for building damage classification. Another contribution of this research is creating a Haiti Earthquake image dataset with four building damage levels. The practical contribution is providing a safe and speedy post-disaster building damage classification method with minimal fieldwork to support rescue teams.