In this research, the impact of micro-silica, nano-silica, and polypropylene fibers on the fracture energy of self-compacting concrete was thoroughly examined. Enhancing the fracture energy is very important to increase the crack propagation resistance. The study focused on evaluating the self-compacting properties of the concrete through various tests, including J-ring, V-funnel, slump flow, and T50 tests. Additionally, the mechanical properties of the concrete, such as compressive and tensile strengths, modulus of elasticity, and fracture parameters were investigated on hardened specimens after 28 days. The results demonstrated that the incorporation of micro-silica and nano-silica not only decreased the rheological aspects of self-compacting concrete but also significantly enhanced its mechanical properties, particularly the compressive strength. On the other hand, the inclusion of polypropylene fibers had a positive impact on fracture parameters, tensile strength, and flexural strength of the specimens. Utilizing the response surface method, the relationship between micro-silica, nano-silica, and fibers was established. The optimal combination for achieving the highest compressive strength was found to be 5% micro-silica, 0.75% nano-silica, and 0.1% fibers. Furthermore, for obtaining the best mixture with superior tensile strength, flexural strength, modulus of elasticity, and fracture energy, the ideal proportion was determined as 5% micro-silica, 0.75% nano-silica, and 0.15% fibers. Compared to the control mixture, the aforementioned parameters showed significant improvements of 26.3%, 30.3%, 34.3%, and 34.3%, respectively. In order to accurately model the tensile cracking of concrete, the authors used softening curves derived from an inverse algorithm proposed by them. This method allowed for a precise and detailed analysis of the concrete under tensile stress. This study explores the effects of micro-silica, nano-silica, and polypropylene fibers on self-compacting concrete and shows their influences on the fracture energy and various mechanical properties of the concrete. The results offer valuable insights for optimizing the concrete mix to achieve desired strength and performance characteristics.