There has been a castle of a sort since the 12th century.