The Shia-led Hashd al-Shaabi forces said human remains had been found at the site “with prison uniforms”.
So-called Islamic State (IS) is alleged to have killed hundreds of inmates, most of them Shia Muslims, when it seized Mosul in 2014.
Badoush prison was recaptured earlier this week.
A spokesman for the Hashd al-Shaabi paramilitary forces, Karim Nouri, said the bodies had been “lined up in a way that indicates they were shot dead in groups”.
On Saturday, Iraqi forces said that a “large mass grave” had been found that contained the remains of hundreds of “civilian prisoners who were executed by (IS) gangs after they controlled the prison during their occupation of Mosul”.
The news follows a 2014 report by Human Rights Watch (HRW) which said that hundreds of male inmates were murdered by IS gunmen at the time.
“We are waiting for forensic teams and human rights officials to begin unearthing the grave to uncover the whole story,” Mr Nouri said on Sunday.
What happened at the prison?
After Badoush prison was seized in June 2014, some 1,500 inmates were rounded up and transported by lorry to an isolated stretch of desert about 2km (1.2 miles) away, survivors told HRW.
According to the rights group’s report, IS gunmen separated the Shia inmates from the Sunnis and Christians and then marched them to a ravine, where they were forced to kneel along its edge.
The Shia inmates were shot in the head or back with assault rifles and automatic weapons, according to the survivors, who escaped by pretending to be dead or because they were shielded by the bodies of victims who fell on top of them.
The Iraqi army’s 9th Armoured Division and allied militiamen recaptured Badoush prison, north-west of Mosul, last week.
A statement released by the Iraqi forces did not say whether they found anyone who was being detained by IS.
Battle to retake Mosul
On Sunday, the Iraqi military said it had retaken two more neighbourhoods in the western district of Mosul from IS. Iraqi security forces said they now control more than a third of west Mosul.
Iraq’s government launched an operation to retake Mosul in October, and announced that the city’s eastern side had been liberated in January.
Troops are now continuing their push into the more densely-populated west.