The roughly oval plan of Pevensey Castle dates from Roman times, and evidence of this period can still be seen in the remains of the rectangular gatehouse and a small postern in the north west wall. Remarkably, the walls of Pevensey Castle have remained in a relatively good state of preservation, providing a good indication of the layout and structure of the castle buildings. Although some of the early earthwork defences were subsequently replaced by structural fortifications, the old Roman ditches and mounds around the site are still visible.

Much of the Roman fort remaining on the castle site is due largely to the work of Robert of Mortain (half brother to William the Conqueror), who was granted Pevensey Castle shortly after the Norman Conquest. De Mortain used the existing fort, which had laid derelict for over 600 years, as the base for building his castle, carrying out only minor repairs to the walls forming the outer bailey, and building a new inner bailey at the eastern end.

A new gateway replaced the original main entrance to the southwest, and the east gateway was repaired. Other alterations made were mainly additions and improvement to existing structures within the original fort. An irregular, rectangular-shaped enclosure was created using part of the Roman wall and two bastions on the southeastern side. Shortly after the inner bailey was created, the rectangular stone keep was erected, incorporating part of the east curtain wall and a Roman bastion. Some time later, three more bastions facing the inner bailey were added to the keep.

The gatehouse of the inner bailey was much altered throughout Pevensey Castle's history, although only parts of the towers and the passageway have survived. Along the curtain wall of the inner bailey are three round-fronted towers dating from the 13th century, despite alterations made by troops who garrisoned Pevensey Castle at a later date. Most sections of the curtain wall have remained intact, and within them the remains of fireplaces can be seen.

This extensive site, situated on the gentle Sussex coast, is a fascinating insight into two very distinct periods of building that were brought together to create a strong, medieval fortress. Following a long and turbulent history, Pevensey Castle was left uninhabited by the 16th century, and fell into a ruinous state, despite a brief period where it was reinstated for defence purposes with the threat of the Spanish Armada. From that time, Pevensey Castle passed through a succession of owners until finally it came into the possession of the Duke of Devonshire who, in 1925, presented it to the State.