At Sunday's annual congress of AFIC, the umbrella organisation representing the key state Muslim bodies, six out of seven state councils present backed a vote of no confidence in Mr Patel.
Mr Patel, who migrated from Fiji in 1992, has been replaced by Haset Sali, a Sunshine Coast commercial lawyer who once likened the actions of former mufti of Australia Taj Din al-Hilali to those of Hitler.
At the congress, held in the nation's biggest Muslim school, Malek Fahd in southwest Sydney, it is understood Mr Patel stormed out, urging all the state councils to follow him. The Christmas Island and Tasmanian councils joined him, but the rest stayed.
"He knew his numbers were failing. He walked out and said he was going to AFIC's head office, asking others to come with him," Mr Sali said yesterday.