Tony Abbott has been warned by his own MPs that his decision to dump proposed changes to the Racial Discrimination Act will see Coalition supporters flock to Clive Palmer.
And several MPs have also cautioned the Prime Minister against dealing with certain Islamic groups.
Mr Abbott dumped his long-held promise to repeal section 18c of the Racial Discrimination Act after a backlash from ethnic communities earlier this month.
He said he needed the support of the Australian Muslim community for proposed changes to anti-terrorism laws.
The decision to abandon the government's election pledge was raised in the party room on Tuesday by several MPs including Liberal senator Cory Bernardi and Nationals MP George Christensen.
Mr Christensen told the Prime Minister that for many Coalition voters, the decision to abandon making it no longer illegal to insult or humiliate someone on the grounds of race was the last straw. He said many were now planning to vote for the Palmer United Party instead of the government.
Senator Bernardi told his colleagues he was disappointed the government had ditched its pledge altogether instead of coming to a compromise. He then went on to warn the Prime Minister against dealing with individuals such as the Sydney Muslim community leader Keysar Trad and groups like the Islamic Council of Victoria, parts of which have advocated for some aspects of sharia law in Australia.
Senator Benardi's caution were echoed by Liberal MP Luke Simpkins who told MPs the government shouldn't "give oxygen" to people whose plans are "inconsistent with the values of our country".
Senator Bernardi also formally told his colleagues he would be supporting crossbench senator Bob Day's compromise bill in the Senate, as previously revealed by Fairfax Media.
Another MP complained that the decision to abandon the repeal of section 18c was made via the media and without backbench consultation. However, the Attorney-General George Brandis said backbench dissent on the issue had been "very influential" in the government dropping the idea.
