A new hybrid damping technique for vibration reduction in flexible structures, wherein a combination of layers of hard passive damping alloys and active (smart) magnetostrictive material is used to reduce vibrations, is proposed. While most conventional vibration control treatments are based exclusively on either passive or active based systems, this technique aims to combine the advantages of these systems and simultaneously, to overcome the inherent disadvantages in the individual systems. Two types of combined damping systems are idealized and studied here, viz., the Noninteractive system and the Interactive system. Frequency domain studies are carried out to investigate their performance. Finite element simulations using previously developed smart beam elements are carried out on typical metallic and laminated composite cantilever beams treated with hybrid damping. The influence of various parameters like excitation levels, frequency (mode) and control gain on the damping performance is investigated. It is shown that the proposed system could be used effectively to dampen the structural vibration over a wide frequency range. The interaction between the active and passive damping layers is brought out by a comparative study of the combined systems. Illustrative comparisons with 'only passive' and 'only active' damping schemes are also made. The influence and the mode dependence of control gain in a hybrid system is clearly illustrated. This study also demonstrates the significance and the exploitation of strain dependency of passive damping on the overall damping of the hybrid system. Further, the influence of the depthwise location of damping layers in laminated structures is also investigated.