OTTAWA -- Liberals are demanding that Prime Minister Stephen Harper send a clear signal on racism by expelling an MP from the Conservative caucus over comments about temporary foreign workers that many have deemed offensive.

New Brunswick MP John Williamson has faced days of criticism for his public description of "whities" staying home while "brown people" are imported to fill jobs under the temporary foreign workers program.

The political website iPolitics reported Williamson's comments last weekend during a panel at the annual Manning Networking Conference in Ottawa.

Williamson, Harper's former chief spokesman, apologized on Twitter but has not spoken of the incident in the House of Commons. Nor has the prime minister commented on his MP's remarks, despite several invitations to do so.

That sparked an incendiary exchange during Wednesday's question period -- and later Liberal demands for Williamson's removal from Conservative ranks.

John McCallum, the Liberal multiculturalism critic, questioned why there have been days of silence from Harper when even some fellow Conservative MPs have criticized Williamson's words.

A Liberal member of New Brunswick's provincial legislature has also introduced a motion calling for public censure of the remarks.

McCallum raised the matter of a briefing note, revealed this week by The Canadian Press, that cautions Conservatives about publicizing the cost of immigrant settlement services out of concern it could upset the party's base.

"The only explanation for the prime minister's silence is that he has given in to these racist sentiments within his own party," McCallum charged as the House of Commons erupted. "How else can the prime minister justify his silence?"

Pierre Poilievre, the Conservative employment minister, responded by demanding that McCallum himself apologize immediately for comments that were "unacceptable."

In an interview, McCallum said it is time for Williamson to be cut loose.

"The appropriate thing at this point is for Mr. Williamson to be removed from the caucus," he said.

"In the year 2015, for a member of Parliament -- and a former director of communications to the PM, no less -- should use the words 'whities' and 'brown people' is just unacceptable."

The Williamson furor adds to a highly charged political battle involving new Conservative anti-terrorism legislation and government comments targeting the wearing of the niqab by Muslim women.

McCallum said it is the prime minister's responsibility to lower the temperature, and firing Williamson would help.

"It's not my job to do Mr. Harper favours, but I think the effect of it may be to show that he finds this behaviour unacceptable," said McCallum.

"Because right now he has said nothing that I'm aware of, and he's almost giving the impression that it's not a big deal for him."

A spokesman in the Prime Minister's Office suggested the matter is closed.

"MP Williamson immediately acknowledged his comments were inappropriate and apologized unreservedly for them," Stephen Lecce said in an email.