Structural damage detection (SDD) is a challenging task in the field of structural health monitoring (SHM). As an exploring attempt to the SDD problem, a hybrid self-adaptive Firefly-Nelder-Mead (SA-FNM) algorithm is proposed for the SDD problem in this study. First of all, the basic principle of firefly algorithm (FA) is introduced. The Nelder-Mead (NM) algorithm is incorporated into FA for improving the local searching ability. A new strategy for exchanging the information in the firefly group is introduced into the SA-FNM for reducing the computation cost. A random walk strategy for the best firefly and a self-adaptive control strategy of three key parameters, such as light absorption, randomization parameter and critical distance, are proposed for preferably balancing the exploitation and exploration ability of the SA-FNM. The computing performance of the SA-FNM is evaluated and compared with the basic FA by three benchmark functions. Secondly, the SDD problem is mathematically converted into a constrained optimization problem, which is then hopefully solved by the SA-FNM algorithm. A multi-step method is proposed for finding the minimum fitness with a big probability. In order to assess the accuracy and the feasibility of the proposed method, a two-storey rigid frame structure without considering the finite element model (FEM) error and a steel beam with considering the model error are taken examples for numerical simulations. Finally, a series of experimental studies on damage detection of a steel beam with four damage patterns are performed in laboratory. The illustrated results show that the proposed method can accurately identify the structural damage. Some valuable conclusions are made and related issues are discussed as well.