Men dressed in Iraqi federal police uniforms are reported to have tortured and killed residents of villages south of Mosul, Amnesty International says.
Evidence gathered by the group suggests up to six people from the Shura and Qayyarah sub-districts were shot dead after being detained on suspicion of having ties to so-called Islamic State.
The Iraqi federal police has issued a statement denying the accusations.
Pro-government forces launched an offensive to retake Mosul last month.
About 50,000 Iraqi security forces personnel, soldiers, police, Kurdish Peshmerga fighters, Sunni Arab tribesmen and Shia militiamen are involved in the three-week operation.
Amnesty researchers visited the locations where the extrajudicial executions are alleged to have taken place as IS militants retreated, taking with them hundreds of women, children and old men apparently for use as human shields.
According to information they obtained, about 10 men and a 16-year-old boy who escaped being forcibly transferred were tortured after handing themselves over to a small group of men wearing police uniforms in the Nus Tal area on 21 October.
Shortly afterwards, reinforcements arrived and the men were taken on foot to an open desert area between the town of Qayyarah and the Shura area.
Fighters dressed in police uniforms allegedly beat the group with cables and rifle butts, punched and kicked them, and pulled their beards - in one case setting a man’s beard alight. They were also made to lie on their stomachs and shots were fired between their legs.
Three men were then separated from the group. Amnesty said men in police uniforms then subjected them to particularly brutal beatings before shooting them dead. Their remains were found in the same area five days later.
The body of a fourth man, who was handcuffed and led away by a group of men in police uniforms after being beaten, was discovered nearby almost a week later, according to Amnesty.
Also on 21 October, another young man was found dead with two bullet wounds shortly after he left a house near the Mishraq sulphur factory, which IS fighters set alight before retreating. He was blindfolded with his torso exposed.
Amnesty said the sixth man was apparently shot dead as he ran towards forces that included men in police uniform while pulling up his clothes to show that he had no explosives.
"When the Mosul military operation began, Prime Minister Haider al-Abadi made clear that violations by Iraqi armed forces and its allies would not be tolerated. Now is the time for him to prove just that," said Lynn Maalouf, deputy director for research at Amnesty’s Beirut office.
"The Iraqi authorities must immediately investigate these alarming reports of extrajudicial executions and torture. They must remove from active duty all individuals who are suspected of committing war crimes and other serious human rights violations, pending the outcome of judicial investigations."