The historic castle loomed over the town below