Highcliffe Castle is a Grade I listed building which was built between 1831 and 1835 by Lord Stuart de Rothesay. It has been described as "the most important remaining example of the Romantic and Picturesque style of architecture."

The Castle was built on the site previously occupied by High Cliff, a Georgian mansion designed for the 3rd Earl of Bute (a founder of Kew Gardens), with grounds laid out by Capability Brown. The Earl's fourth son, General Sir Charles Stuart who sold the estate apart from Bure Homage, a small house on its outskirts, inherited High Cliff. All that remains of High Cliff today are the two entrance lodges, presently being used as a restaurant and some of the garden walls and features on the present estate.