Gary Neville's comments comparing striking workers in the UK to Qatari migrant labourers was not "legitimate or appropriate", Downing Street said on Monday.
The ex-England footballer was criticised for comparing Qatar's treatment of workers with the British government's actions.
Speaking on ITV ahead of the World Cup final, the pundit accused ministers of "demonising" rail workers, ambulance drivers and "terrifyingly" nurses who have all voted for industrial action over pay and conditions.
He said poor labour conditions should not be accepted in Qatar or “any region”.
When asked about the comments on Monday, the Prime Minister's official spokesman said "it's clearly not a legitimate or appropriate response”.
Media regulator Ofcom is considering whether to investigate after receiving complaints about Mr Neville's remarks.
Senior Tory MPs have also criticised the ex Manchester United right back.
Former minister Simon Clarke said: "It is beyond ridiculous that Gary Neville is given free rein by ITV Sport to overtly politicise a major sporting event."
However Mr Neville hit back, describing the Conservative party on Twitter as "the biggest set of charlatans to ever be in power".
An ITV spokesman said: "Gary Neville was expressing his own personal views in the context of a discussion about treatment of workers in Qatar within a live broadcast. His views are his own and were not endorsed by ITV."
