The Romans had long ago recognised the importance of this location in Cumbria, building a fort (Brocavum) there in the 2nd century. Situated in a lush valley beside the River Eamont, Brougham Castle was built over the ruins of the Roman fort in the 13th century. Re-using much of the old stone, and utilising some of the original earthworks as medieval defences, the transformation from a Roman to a medieval fortification was almost seamless.

It was early in the 13th century when Robert de Vieuxpont acquired the site, building his stone keep and domestic buildings within a timber palisade. By the turn of the century ownership had passed, through marriage, to the Clifford family, and it was Robert Clifford who began strengthening Brougham Castle against Scottish raids. A huge stone curtain wall replaced the wooden surround, a sturdy gatehouse complex was constructed along the northern boundary, and a new range was placed inside the curtain at the south eastern edge of the site. Robert's grandson extended the buildings along the south curtain during the 1380s but, before the end of the decade, Brougham Castle had become a victim of the Anglo-Scottish wars.

Throughout the Wars of the Roses Brougham Castle played a key role in the struggle for dominance, the Cliffords holding out against a Yorkist invasion until John Clifford was killed in 1461. Brougham Castle then temporarily under the control of the Nevilles, until John's son could reclaim the family estates. Life returned to a state of relative peace and prosperity for the Cliffords, and in 1525 John's grandson, Henry, was created first Earl of Cumberland. Although Brougham Castle had become somewhat neglected by the time of the Civil War, it did enjoy a brief revival when Lady Anne Clifford took an interest in the family home. Visiting Brougham Castle on a regular basis for over 20 years, she restored the fabric of the building, added a new bakehouse and brewhouse, and established large fruit and vegetable gardens. Following her death in 1676, Brougham Castle was virtually abandoned. By the mid 18th century, all its treasures having been sold off, Brougham Castle had become a ruinous shell.

Today's visitors still enter Brougham Castle through the solidly constructed red-brick gatehouse, this leading directly to the courtyard and the keep. As the only part of the original castle to survive, the keep is the oldest building on the site. When first built, it contained three levels (as it does today in its ruinous state), but Robert Clifford added a further storey with battlements at the beginning of the 14th century, which has long since disappeared. Little else exists to any degree of completion, but several walls in the south range of buildings remain standing to a good height, as does the south west tower. Unlike the other 'Clifford castle' to survive at Brough, Brougham Castle has a really picturesque setting, and is a wonderfully peaceful place. It is also possible to determine the outline of the Roman fort on the south side of the castle, making for a fascinating exploration of nearly two thousand years of history.