TUNIS, Tunisia — He is an orphan of the Islamic State group’s self-proclaimed caliphate, a Tunisian toddler who is now caught in diplomatic limbo and has been stuck in a Libyan prison for a year.

Tamim Jaboudi’s grandfather has managed to visit the child twice in the prison in Tripoli, delivering a winter jacket and as much familial warmth as he can manage in the brief meetings. But Tamim barely knows him, and by now can hardly remember his parents – a Tunisian couple who left their homeland to join the Islamic State group and were killed last year in an American airstrike, according to the grandfather, Faouzi Trabelsi.

Living among a group of around two dozen Tunisian women and their young children imprisoned in Tripoli’s Mitiga prison, Tamim is being raised by a woman who herself willingly joined the Islamic State group, according to his grandfather and human rights groups.

“What is this young child’s sin that he is in jail with criminals?” asked his grandfather, Faouzi Trabelsi, who has traveled twice to Libya to see the boy and twice returned home emptyhanded. “If he grows up there, what kind of attitude will he have toward his homeland?”

European governments and experts have documented at least 600 foreign children of fighters who live in or have returned from Islamic State territory in Syria, Iraq or Libya. But the numbers are likely far higher. In Libya, the fate of 44 Tunisian children is particularly uncertain. The North African nation descended into chaos after the 2011 civil war and has been split into competing governments with numerous militias, tribes and political factions. Militias in December captured the main stronghold of the extremist group in Libya, Sirte.

Both Tunisia and Libya say they want the return of the women and children, but for months any effort to hand them over has fallen apart with little explanation. That has raised complaints in Tunisia that the government does not want them back for security concerns. But the militia running Mitiga prison, even as it says it wants to hand over the families, has tightly controlled access to them, saying the Tunisians need permissions from the office of Tripoli’s top prosecutor. Part of the problem also appears to be that Tunisian officials are reluctant to deal directly with the militia, since it isn’t a government body.

On Wednesday, a Tunisia delegation was in Tripoli and was supposed to come to the prison, but the visit was canceled at the last minute, and the delegation returned home emptyhanded.

Tunisia is willing to take them, said Chafik Hajji, a Tunisian diplomat who handles the cases of Tunisians who have joined the extremists. “There is no wrong in being born in a conflict zone,” he said. “Once their Tunisian citizenship is confirmed, they will have an individual treatment.”

Tamim’s grandfather Trabelsi spoke with The Associated Press in his spotlessly clean living room in Tunis. Outside, the neighborhood was rough at the edges, its streets pitted with neglect. Around the corner, adolescent boys brawled as a crowd circled around to watch.

He said his daughter, Samah, married a young man from the neighborhood after a quick courtship, and then the newlyweds left for Turkey, a common jumping off point for Europeans and North Africans joining extremist groups. Tamim was born there in April 2014. The couple returned to Tunisia briefly, then went to neighboring Libya, where they remained for two years, he said.

The Islamic State group paid particular attention to recruiting families, boasting it would build a society to endure for generations.

“In the long term, there is the new generation of ISIS,” said Mohammed Iqbel, whose Association of Tunisians Trapped Abroad advocates for the families of those who have left. “And if we don’t save them, they will be a new generation of terrorism.”