John de Broughton built the house and surrounded it by a moat in around 1300. William of Wykeham, Bishop of Winchester and Chancellor of England, bought the house in 1377 and it has been in continuous ownership by the same family ever since. The licence to crenellate was granted in 1406 allowing the house to be called a castle.

The unusual compound title of Saye & Sele dates from the original creation of the Barony in 1447. It is partly personal, from the connection with the family’s French ancestors in Normandy, the Lords Say, and partly territorial,, through ownership of land at Seal (now on the outskirts of Sevenoaks) in Kent.  The family name Fiennes (pronounced Fines) is from a village in Artois in northern France. Giles Fiennes had emigrated to England in the 13th century, following in the train of his cousin Queen Eleanor of Castile when she came to England to marry the future Edward I.*

The 16th Lord Saye & Sele changed the family surname in 1849 to Twisleton-Wykeham-Fiennes. The present-day, 21st Lord Saye & Sele reversed the process in the mid 1960, so the current family name is simply Fiennes.

*From the Broughton Castle hand book