The editorial titled "Silence as Women Are Attacked" appeared on in the New York edition of the daily, reminding how Modi tweets "frequently and considers himself a talented orator".
"Yet he loses his voice when it comes to speaking out about the dangers faced by women and minorities who are frequent targets of the nationalist and communal forces that are part of the base of his Bharatiya Janata Party (BJP)," said the Times.
It recalled how on Friday, Modi said the rape cases had brought shame on the country and that "our daughters will definitely get justice".
"But his remarks ring hollow because he waited so long to talk about the cases and spoke in broad generalities -- describing the crimes as 'incidents being discussed since past two days'."
The daily accused the Prime Minister of taking a similar approach in the past when addressing cases in which vigilante groups affiliated with his political movement attacked and killed Muslims and Dalits, falsely accused of killing cows.
There was a widespread public outrage as people across the country protested against rape and murder of an 8-year-old girl in January in Jammu and Kashmir and rape of a teenager in Uttar Pradesh.
The Prime Minister, however, has not directly spoken about the crimes and other cases involving supporters of the BJP.
Modi has also not talked about a rape accusation against a ruling BJP lawmaker in Uttar Pradesh.
"Modi's silence is as perplexing as it is distressing," the Times said, adding he seemed to have failed to learn the lesson of the Congress that did not forcefully respond to protests in late 2012 and early 2013 after a young woman was raped and killed on a public bus in New Delhi.
"The BJP won the elections in large part because Modi promised to make the government more responsive to the needs of Indians who were left behind by a government dogged by corruption scandals and widely considered rudderless.
"Instead, he has exhibited a pattern of silence and deflection that is deeply worrying to anybody who cares about the health of the world's largest democracy."
The Times editorial said Modi could not be expected to discuss every crime committed by someone who supports him.
"But these cases are not isolated or random examples of violence. They are part of an organized and systematic campaign by nationalist forces that want to terrorise women, Muslims, Dalits and other underprivileged citizens.
"The Prime Minister has a duty to safeguard and fight for all of the people of India, not just those who are allied with him politically."