Although the Norman castle no longer dominates this market town in Monmouthshire, the ruins are nonetheless a constant reminder of this formidable fortification with a bloody history. A date of around 1087 has been suggested for the foundation of Abergavenny Castle, but the only remains of Hamelin de Ballon's castle is the motte.

Gerald of Wales provides some detail about the early defences of the castle, but most of what is written about Abergavenny is based on similar structures from the Norman period. For instance, there would have been a timber tower erected on top of the motte, and a wooden palisade around the perimeter. By the end of the 12th century, it is possible that the tower was replaced by a circular, stone keep. However, accounting records from the 13th century suggest that the outer defences, the town wall, and most of the ancillary buildings within the bailey were still largely of timber construction.

Rebuilding of Abergavenny Castle and the town wall in stone is dated to the early decades of the 14th century by Lord John de Hastings. It would seem that the castle was well maintained until the 16th century when, from a verse by Thomas Churchyard, Abergavenny Castle is described as being subject to 'rack and ruin'. Following the Civil War, the castle was left to decay and, over time, much of it has disappeared.

Today, there is an impressive remnant of curtain wall from the early stone castle, standing to almost full height and retaining much of its facing stone. Beside this, the fragmented remains of the late 14th century gatehouse entrance, through which a view of the ruinous west towers can be seen. Although only the shells survive, they do remain standing to varying degrees, at best to four levels high. In 1818, a hunting lodge for the Marquess of Abergavenny was constructed on top of the motte, and this has been incorporated in the present museum building.

Standing amongst the crumbling masonry, perhaps it is not too difficult to imagine some of the horrible scenes that occurred in and around Abergavenny Castle. This was a strategically important fortress, and the stark remains of massive walls are indeed testament to the battles fought at Abergavenny. Not least among these, was the infamous massacre of the Welsh nobles by William de Braose in 1175, a Norman lord of Abergavenny renowned for his cruelty. Naturally, there was retaliation for the murders and this set the pattern for life at Abergavenny Castle, with ownership continually changing hands between the Marcher lords and the native Welsh. The final curtain came down in 1645 when, following the capture of Monmouth, the remaining habitable buildings of Abergavenny Castle were burned and the garrison withdrawn.