“No, don’t look till you’re at the top!” says Patrick Seely. There, laid out before us is a stunning microcosm of what makes the Isle of Wight beautiful. We can see the coast of Ventnor right through to the white cliffs of Freshwater; the Needles, but also the meandering coil of the Yar; and linking it all are the rolling hills, which from this vantage point seem higher and more majestic than they do from the road.

What we’re actually looking at is the map of Patrick’s family history. The land the Seely family once owned, and do no longer. The phrase “better to have loved and lost” springs to mind.

For it was love which brought the Seelys to the Isle of Wight – love of this place, this view. “At least that’s the family story,” says Patrick. They are relative newcomers to the Island, in that their family only goes back 150-odd years. But they have certainly made their mark. Patrick and his wife Susannah now farm the famous Dunsbury lambs. The Seely estate once ran from Freshwater Bay golf course, all the way through Brook Down, which is what we are standing on, through Mottistone Down to the top of Strawberry Lane.

“My family history is the perfect example of Victorian entrepreneurship: of coming from a certain point, making a lot of money, and within three generations losing it all. That’s what happened.”

Patrick’s great-great-grandfather, Charles Seely, inherited his father’s milling business in Lincoln. As a young man, a severe illness caused Charles to be sent to the Isle of Wight to recover, staying in Ventnor. One day he walked from there to the Needles, and so profoundly was he struck by what he saw that, so the family story goes, he said: “When I’m older I shall own this land.”

Charles became a canny businessman. In the 1840s he bought a farm in Nottingham, which happened to sit on top of the Nottingham/Derby coal field. So just as the age of steam was hitting its stride, Charles had found coal, and the boy from the Midlands began to fulfil his dream. He bought the Brook estate, followed by the Mottistone estate in the 1850s; later, with Gatcombe, Ashengrove and Bowcombe, Charles Seely had effectively cornered the market in good shooting land.

Now one of the richest men in Britain, he went into Parliament, where he caused something of a stir. He welcomed the Italian revolutionary Giuseppe Garibaldi to his home in Lincolnshire and to Brook House. There were two consequences. Charles’s wife Anne fell in love with Garibaldi: letters, which came to public attention when they went on sale in 2008, spanned 10 years. And Queen Victoria, whose path frequently crossed that of Charles as a senior Island figure, refused to speak to Charles.

“Then at the time of Victoria’s Jubilee 1897, the responsibility of giving the Queen a present on behalf of the Isle of Wight fell to Charles,” explains Patrick Seely. “She is supposed to have said to him: “It’s been a long time, Mr Seely.” He replied: “As we grow older we get wiser, Ma’am.””

Patrick is also amused by another aspect of Charles’s business sense. Slurry, the waste from his coal mines, was made into pig iron, but long after the days of wooden ships the Royal Navy was still tied into a contract to accept the weighty ‘pigs’ for ballast. They just lined Portsmouth naval yard.

It was all grist to the coffers of this former milling family, and the landscape below us reflects this story of growing wealth. Brook Hill House, known locally as JB Priestley’s home after the war, was built when doctors advised Charles’s son, Sir Charles Seely, that living a little higher up would benefit his health. It is designed by Ashton Webb, architects of Dartford Naval College, and is now divided into apartments.

Elsewhere can be found other physical legacies of the family’s wealth translated into largesse for the good of the community. “There is a tremendous sense of public service running through the family’s history,” says Patrick. Sir Charles, an MP until 1895 and a keen sportsman, is also remembered for promoting the free library service – the libraries in Newport and Cowes both bore the Seely name, and a reading room was created in Brook. Sir Charles’s son Jack gave the land on which stands the Royal Solent Yacht Club to his fellow yachtsmen, and secured the ‘royal’ epithet so as not to be outdone by the rival Lymington club.

The spirit of the entrepreneur was reflected in this next generation. Of Sir Charles’s three sons, who were all brought up at Brook House, Jack is described by Patrick, his grandson, as having had nine lives and used every one of them. Jack Seely was thrilled to the core with danger. He was a dedicated lifeboat man, becoming the last Coxswain of Brook lifeboat, and diced with death many times.

General Jack, as he became, is remembered too for his acts of heroism in the Boer War. While he was away fighting with the Hampshire Yeomanry he was elected to Parliament as a Conservative, but the shambles with which the war had been conducted caused him to cross the house to become a Liberal, under Asquith. He was followed across the floor by Winston Churchill, with whom he had fought in the war.

Jack was part of that great Liberal government, preceding Kitchener as War Minister. However, just before the First World War the Curragh Mutiny took place in Ireland, and as senior person responsible, Jack resigned. In the same year, his wife died, giving birth to her seventh child. Jack withdrew from public life, but come the war he was given command over the Canadian cavalry and “got into all sorts of scrapes”, according to his awed grandson. His adventures are recorded in Galloper Jack, a lively history written by Patrick’s cousin, Brough Scott (the face of horse racing on television).

The force of Jack’s charisma is shown when, at the time of the Great Depression, he went about persuading people to take a reduction on their war loans in order that the Government’s finances could be repaired. For that, and for other public service, he was made Lord Mottistone.

While Jack was away with the Canadian Cavalry his father, Sir Charles, had died. The Nottingham coal estates went to his brother while Jack was left the estates on the Island. “There wasn’t enough income coming from the land,” says Patrick. “He went through his money rather quickly.” Jack sold the Brook estate and moved to Mottistone – hence the choice of title. His son John was a talented architect, known to the Bloomsbury set, and it was he who took what had been a rather dowdy farmhouse, its grounds partly subsumed by a landslide, and, under the guidance of no lesser figure than Sir Edward Lutyens, transformed it into the elegant house it is today, in the style of an Elizabethan manor.

To generate some income Jack took to writing novels, the titles reflecting the tales of derring-do which had been so much part of his own life: Adventure, Forever England, My Horse Warrior, and Launch, which is about Brook Lifeboat.

Widower Jack married again, to the widow of his Principal Private Secretary while he was in parliament, Sir George Nicholson. His new wife brought her seven-year-old, John into the Seely family. Sir John Nicholson Bt grew up to become the youngest chairman of the Blue Funnel shipping line, and also ran all the supply lines to the Far East for Mountbatten’s campaign in Burma.

“So all of this goes full circle. My Grandfather was Lord Lieutenant in the 1930s, Mountbatten became Lord Lieutenant, followed by John Nicholson, and my father – his half-brother – succeeded him.” Patrick’s father, David, Lord Mottistone, was also the last Governor of the Isle of Wight.

As Patrick and I look down on the land below us, the signficance of the family’s size becomes apparent. “My great grandfather was proud to announce he had 60 living descendents. But, always feeding bigger and bigger families, they managed to spend it all.”

So the 10,000 acres once owned on the Island was sold piecemeal. The Mottistone Estates were given to the National Trust in the early 1960s, while Brook House went to a property developer, houses being built in its former grounds. “It’s rather sad to see. The house had a lovely view to the sea before all the gardens were no longer tended.”

Clearly that is not the end of the story. Patrick Seely went to business school in France, and mixing with people from all sorts of cultures he found many of them had no sense of their roots. “So when Susannah and I married I said I thought it would be really good if we went back home and created a sense of identity for our family.”

They managed to buy the old rectory in Brook, and when the farm came on the market they bought it. Children, and lambs, continue the Seely family presence on the land for the next generation. “It really is the most beautiful place in the world,” says Patrick.