Jeremy Hunt has admitted that Boris Johnson's Brexit deal has caused damaging trade barriers with the EU.
The Chancellor, who yesterday unveiled an Autumn Statement which experts warn will inflict years of pain on millions of Brits, said it will "take time" to remove these barriers.
Meanwhile he stressed that immigration will be "very important" for the economy.
His comments came after the Office for Budget Responsibility (OBR) said Brexit caused a "significant adverse impact" to trade volumes and business relationships between UK and EU firms.
Asked if rejoining the single market would boost growth, the Chancellor told BBC Radio 4's Today programme: "I think having unfettered trade with our neighbours and countries all over the world is very beneficial to growth.
"I have great confidence that over the years ahead we will find outside the single market we are able to remove the vast majority of the trade barriers that exist between us and the EU. It will take time."
Pressed on the single market, he said: "I don't think it's the right way to boost growth because it would be against what people were voting for when they supported Brexit which was to have control of our borders and membership of the single market requires free movement of people.
"So I think we can find other ways that will more than compensate for those advantages."
Mr Hunt insisted that immigration is required to boost growth.
"There needs to be a long-term plan if we're going to bring down migration in a way that doesn't harm the economy," he said.
"We are recognising that we will need migration for the years ahead - that will be very important for the economy, yes."
The Chancellor insisted Home Secretary Suella Braverman backs his plans, saying: "Her priority is to reduce illegal migration and deal with the small boats issue."
The Brexit deal brokered by Mr Johnson while prime minister saw the UK leave the EU's single market and create barriers to doing business with Britain's largest trading partner.
On Friday, the Institute for Fiscal Studies said "very clearly Brexit was an economic own goal" that has harmed economic growth.
The economic think tank's director Paul Johnson said it has been "very bad news indeed and continues to be bad news, particularly the way that we've done it, the hard type of Brexit we've had, distancing ourselves from the single market".
On Thursday, the OBR said: "Our trade forecast reflects our assumption that Brexit will result in the UK's trade intensity being 15% lower in the long run than if the UK had remained in the EU.
"The latest evidence suggests that Brexit has had a significant adverse impact on UK trade, via reducing both overall trade volumes and the number of trading relationships between UK and EU firms."
The OBR also forecasts payments of £18.9 billion to Brussels under the terms of the Brexit divorce deal between 2022-23 and 2027-28.
The SNP's Europe spokesperson Dr Philippa Whitford lashed out at the Tories, accusing them of "putting rotten ideology" ahead of "good ordinary people".
She fumed: “It beggars belief, when you look at the dire UK economic forecasts and the Chancellor’s own admission that single market membership would boost the economy, that the Tories have done nothing to rectify this problem of their making. Instead they have doubled down on their deluded Brexit obsession along with Labour.
“We’re not even in some situation where the causes of the UK’s economic woes are some great mystery or that there are no solutions in sight, rather one massive one is staring the Tories in the face - a reversal of the extreme Brexit which has hammered the economy, hit trade and businesses and ripped away economic and social opportunities.
“That is ever the story with this Conservative party though, all too willing to put rotten ideology ahead of the good of ordinary people and the economy.
