The Cartoons That Shook the World, by Danish-born professor Jytte Klausen, set to be published in November, will examine the protest campaign against the caricatures first published in the Danish newspaper Jyllands-Posten in 2005.
She told Fox News she was disappointed that Yale University Press had "rolled back our own principles" and questioned why it has consulted anonymous experts before making its decision.
"I regard the experts' advice to the university as alarmist and misplaced," she said.
"The images were published widely before protests gathered steam, orchestrated by a network of Islamist preachers and motivated by anti-Western sentiment."
Sherry Jones, an author whose novel the Jewel of Medina was shelved by Random House because of fears of violent reprisals, said: "I decided to take a stand for free speech and publish my books in spite of threats and violence because I wanted to make a positive difference in the world,"
"Yale University Press's decision, like that of the executives at Random House, does the opposite," she said. "Self-censorship changes our world for the worse."
The original cartoons, published in September 2005, where the result of a competition set by the newspaper.
One showed Mohammed with a ticking bomb in his turban and like others was regarded as blasphemous by many Muslims whose faith bans the depiction of the Prophet.