The Queen would "excitedly" take Harry's phone calls when the Sussexes first moved to the US - but she became "perplexed" by her grandson's "complaints" over time, the Queen's staff claim.
Writing for the Daily Mail, journalist Richard Kay claims to have spoken to the Queen's staff about the matter.
He said there was no doubt Harry and the Queen had "a close bond" adding: "The Queen's staff have told me how, in the early days of Harry's exile in America, the Queen would excitedly take his phone calls.
"Over time this changed and she later became perplexed by Harry's complaints."
The news comes as Harry and Meghan are expected to fly back to their children believed to still be at their home in California today after an extended stay in the UK.
Harry was visibly emotional during yesterday's final goodbye to his beloved grandmother.
The Sussexes arrived at Westminster Abbey separately, with Meghan travelling with Sophie, Countess of Wessex, Kate, Princess of Wales, Prince George and Princess Charlotte - while Harry walked behind the coffin with his brother and father.
They reunited to take their seats, sitting side-by-side for the hour-long religious event.
The pair were spotted by royal well-wishers and funeral guests offering touching displays of support to one another on the difficult day.
Harry, dressed in a suit rather than his military uniform as he is no longer a working royal, held hands with his wife as they comforted each other while leaving the hall.
Sharing an image of the sweet moment on social media, one Twitter user posted: "I love how Prince Harry always holds Meghan close."
Later on in the day, the Duke was seen walking solemnly on the grass outside St George's Chapel ahead of the committal service as he spoke to Princess Anne and her husband Vice Admiral Sir Tim Laurence.
After the service he looked defeated, puffing out his cheeks as he and Meghan got in a car to leave.
The family then witnessed Her Majesty being buried alongside the Duke of Edinburgh a service that was conducted by the Dean of Windsor.
It was one of the only parts that was not be televised of the state funeral yesterday after billions of people around the world watched the state funeral.
Paying tribute to his grandmother earlier last week, Harry said how much he "cherished" the time he'd spent with the Queen.
"You are already sorely missed, not just by us, but by the world over," he said.
