Opposition Leader Peter Dutton admits it will be an uphill battle to retain the Victorian seat of Aston following the resignation of former minister Alan Tudge.
Mr Tudge announced he was stepping away from federal politics, citing health and family reasons after reporting death threats against his teenage daughters.
The resignation has triggered a by-election in his outer Melbourne seat of Aston, which the Liberals hold by a 2.8 per cent margin.
Mr Dutton said the upcoming by-election would be a difficult one for the coalition.
"It is a tough seat for us to hold, there's no question about that, and there are lots of families there, though, who are really feeling the pain of their mortgages," he told the Nine Network's Today program on Friday.
"It'll be a hard fought contest but we'll get a date (for the by-election) from the speaker, no doubt, in the coming weeks."
While Aston was considered a safe Liberal seat, Mr Tudge suffered a more than seven per cent swing against him at last year's federal election.
Mr Dutton said work was already under way on selecting a candidate to run for the seat but admitted the loss of Mr Tudge in the electorate would see a drop in the party vote.
"By-elections are always difficult and there are different issues, lots of local issues," he said.
"Alan is a popular member and that always brings a vote with it which won't be present in the by-election but we'll preselect a great candidate and that process has already started."
Mr Tudge has been in parliament since 2010, holding coalition ministries in education and human services.
He stepped down from cabinet after being accused of emotional and physical abuse by his former staffer Rachelle Miller, with whom he had an affair.
Mr Tudge denied the claims of abuse but opted not to return to cabinet once an investigation found he did not break cabinet rules.
In an emotional speech to parliament, Mr Tudge said his decision to resign was cemented following the death of his father earlier this year, as well as death threats made against his daughters
"My daughters who are 18 and 16 know nothing other than their dad being away," he said.
"They've had to put up with things that no teenager should have to - including death threats, the most recent of which was last week."
Deputy Prime Minister Richard Marles said the impact politics had on families was often great.
"You've got to feel for Alan and his family in terms of what they've experienced over the last few months," he told the Today program.
"We all choose to go into politics but our families don't and often they bear a lot of the burden here."
