This study is aimed to develop a damping model to accurately predict vibration amplitude reduction for any size of structure. It is developed in the framework of multi-scale analysis, where different sources of energy dissipation at captured at material-scales (e.g.,scale of representative volume element). In particular, we illustrate details for concrete structures, where one needs different failure mechanisms like plasticity, damage and viscosity to represent different sources of dissipation are reproduce the typical hysteresis loops of concrete with both residual deformation and change of initial elastic response. The final step in proposed approach is to account for structure heterogeneities by allowing for variability of elasticity limit, which produces the same exponential (rather than linear decay) of vibration amplitudes, just as in the case of Rayleigh damping. However, contrary to Rayleigh damping calibration that can be done only on a single structure (and for a chosen frequency), the proposed approach can be adapted to any structure size and full interval of frequencies of interest. The price to pay is in terms of nonlinear analysis, which is here rendered very efficient by hybrid-stress formulation to uncouple different damage mechanisms and by using linear evolution equations for internal variables representing such mechanisms. The details illustrated for 1D and 3D concrete model can be easily adapted to other materials, such as steel, soils etc.