Recycled aggregate concrete (RAC) is considered a good way of sustainable development, but compared with natural coarse aggregates (NCA), the performance of recycled coarse aggregates (RCA) is often worse. This study aimed at the basalt fiber (BF) enhances the frost resistance of RAC. Therefore, a rapid freeze-thaw cycles test was carried out on basalt fiber reinforced recycled aggregate concrete (BFRRC), with the variation of three different replacement ratios of RCA (i.e., 0, 50 and 100%) and four different contents of BF (i.e., 0, 2, 4 and 6 kg/m3). Then, the damage appearance, mass losses, and relative dynamic elastic modulus (RDEM) of specimens were analyzed. Results were showed that the appearance damage characteristics of RAC are different from NAC, as the replacement ratio of RCA was increased, the damage appearance of the specimens was exacerbated, and surface spalling transformed into corner spalling and holes were generated in the surfaces of the specimens. Compared with mass loss, REDM can better reflect the frost resistance of BFRRC. BF could significantly improve the damage appearance and RDEM loss of the specimens, incorporating 4 kg/m3 of BF can significantly improve the frost resistance of RAC. In addition, the mechanism of freeze-thaw damage was revealed by using scanning electron microscopy (SEM) and damage theory, a freeze-thaw damage model of BFRRC was established by defining the damage degree on the basis of the RDEM. The results of this work could provide a reference for the further research and engineering application of BFRRC.