The first written record of the castle dates from 1404, when it was the property of the Gurk diocese. The property was managed by the knights of Bizeljsko. In 1441 the building is mentioned as geslos Visel and in 1475 as castrum Visl. In 1532 the castle became the fief of the Tattenbach family, who fully purchased it in 1608. After that, the castle often changed owners. The last were the Windischgratz family. The courtyard with its arcades is surrounded by four residential wings that are fortified with two large round towers. The medieval structure was renovated during the Renaissance period, when the chapel of Sv. Hieronimo (St Jerome), featuring a lavish Baroque altar, was built next to the entrance to the castle in 1623. The vault of the chapel and three rooms on the upper floor are decorated with stuccowork dating from the third decade of the 18th century.

The castle was sufficiently fortified and its inhabitants sufficiently armed to resist all Turkish raids. During the uprising of Slovene and Croatian peasants in 1573, the building was spared, as the peasants received weapons and money from the residents of the castle. The last owners of the property, the counts of Windischgratz, moved all valuable possessions, such as weapons, paintings and the archive, from the castle to Moravia in the 19th century.