In the early 12th century, the full wealth and power of the Bishops of St Davids was clearly demonstrated by the properties they built or acquired. Three impressive palaces were erected to accommodate their grand lifestyle, all fortified, but Llawhaden was the only one that truly resembled a castle.

Originally, little more than an earthwork enclosure surrounded by a moat, the palace was captured and destroyed by Lord Rhys in 1192. Within a couple of years Llawhaden Castle had been recovered by the Bishop, and re-building in stone began almost immediately. Following the line of the old earthworks, a curtain wall was erected to replace the wooden stockade, with semi-circular towers being placed at regular intervals along its length. During the latter part of the 13th century Llawhaden Castle was converted and extended into a prestigious palace for Bishop Thomas Bek, and his work formed the foundation for later additions.

The trend for transforming these modest strongholds into lavish, fortified mansions continued into the 14th century, and most of today's visible remains was built for Bishop David Martyn. When completed, the castle-cum-palace provided comfortable private apartments for the Bishop, numerous guest rooms, entertaining halls, a bakehouse, and living accommodation for a garrison. A massive gatehouse, constructed in the late 14th century, was the last major building project at Llawhaden Caslte, and this has survived to its full height, although much decayed internally.

With the Dissolution of the Monasteries came the changing fortunes of the bishops. Their palaces were deserted, and Llawhaden Castle has stood in neglected silence since the 16th century. Despite its extensive use as a local quarry over the centuries, some walls remain to a good height and a substantial earthwork, 70ft (21.2m) wide by some 25ft (7.6m) deep, clearly delineates the original moat. Despite the fact that this strange arrangement of ruins are devoid of any detailed work, the sheer size of the site conjures up pictures of wealth and splendour almost unimaginable some seven hundred years ago.