Geopolymers are an important alternative material supporting recycling, sustainability, and waste management. Durability properties are among the most critical parameters to be investigated; in this study, the durability of manufactured geopolymer samples under the attack of 10% magnesium sulfate and 10% sodium sulfate solution was investigated. 180 cycles of freezing and thawing were also tested. The experimentally obtained results investigate the durability of geopolymer mortar prepared with fly ash (class F) and alkali activator. Three different quarry dust wastes replaced the river sand aggregate: limestone, marble, and basalt powder as fine filler aggregate in three different replacement ratios of 25%, 50%, and 75% to produce ten series of geopolymer composites. The geopolymer samples' visual appearance, weight changes, UPV, and strength properties were studied for up to 12 months at different time intervals of exposure to sulfate solutions to investigate sulfate resistance. In addition, Scanning Electron Microscopy (SEM), EDS, and XRD were used to study the microstructure of the samples. It was beneficial to include quarry waste as a filler aggregate in durability and mechanical properties. The compact matrix was demonstrated by microstructural analysis of the manufactured specimens. The geopolymer mortars immersed in sodium sulfate showed less strength reduction and deterioration than magnesium sulfate, indicating that magnesium sulfate is more aggressive than sodium sulfate. Therefore, it is concluded that using waste dust interrogation with partial replacement of river sand with fly ash-based geopolymers has satisfactory results in terms of durability properties of freeze-thaw and sulfate resistance.