Widespread damages from extreme winds have attracted lots of attentions of the resilience assessment of power distribution systems. With many related environmental parameters as well as numerous power infrastructure components, such as poles and wires, the increased challenge of power asset management before, during and after extreme events have to be addressed to prevent possible cascading failures in the power distribution system. Many extreme winds from weather events, such as hurricanes, generate widespread damages in multiple areas such as the economy, social security, and infrastructure management. The livelihoods of residents in the impaired areas are devastated largely due to the paucity of vital utilities, such as electricity. To address the challenge of power grid asset management, power system clustering is needed to partition a complex power system into several stable clusters to prevent the cascading failure from happening. Traditionally, system clustering uses the Binary Decision Diagram (BDD) to derive the clustering result, which is time-consuming and inefficient. Meanwhile, the previous studies considering the weather hazards did not include any detailed weather-related meteorologic parameters which is not appropriate as the heterogeneity of the parameters could largely affect the system performance. Therefore, a fragility-based network hierarchical spectral clustering method is proposed. In the present paper, the fragility curve and surfaces for a power distribution subsystem are obtained first. The fragility of the subsystem under typical failure mechanisms is calculated as a function of wind speed and pole characteristic dimension (diameter or span length). Secondly, the proposed fragility-based hierarchical spectral clustering method (F-HSC) integrates the physics-based fragility analysis into Hierarchical Spectral Clustering (HSC) technique from graph theory to achieve the clustering result for the power distribution system under extreme weather events. From the results of vulnerability analysis, it could be seen that the system performance after clustering is better than before clustering. With the F-HSC method, the impact of the extreme weather events could be considered with topology to cluster different power distribution systems to prevent the system from experiencing power blackouts.