Troops from the Amisom peacekeeping force in a camp for internally displaced people in Mogadishu. Photograph: Clar Ni Chonghaile for the Guardian
Soldiers from the UN-backed Amisom force are edging towards the Islamist militants' stronghold of Kismayo
Colonel Kayanja Muhanga is describing his troops' latest victory when suddenly there is a rattle of machine-gun fire somewhere beyond the fortified base set among thorn trees and cacti.
"That's about three kilometres away," the Ugandan commander says. "Mop-up operations. We know where they are."
"They" are al-Shabaab, the Islamist militia allied with al-Qaida who are in retreat. Having surrendered the capital, Mogadishu, last August, they were recently pushed out of Afgoye, a town 30 miles away, by a force of African peacekeepers aided by Somali troops.
Ethiopian forces have also driven them out of the southern city of Baidoa, and Kenyan troops, now part of the Amisom peacekeeping force, are edging towards their stronghold in the port of Kismayo. Amisom commanders, Somali government officials, and residents of Mogadishu say al-Shabaab, which means "the youth", is on its last legs. Its forces are scattered and weak, deprived of income and losing fighters.
There are increasing reports of rebels switching sides – young men such as Khalid, a 24-year-old who surrendered to Muhanga's forces and is helping Amisom winkle out militants hiding among the population in Afgoye. Khalid joined al-Shabaab four years ago. Speaking through a translator at Muhanga's base just outside Afgoye, he said he left because he saw the rebels were punishing civilians. "I found these people were deceivers," he said, cracking his knuckles. During the interview, his phone rang. Khalid said it was his former commander. He put him on speakerphone and they talked for a long time. Afterwards, the translator said the commander threatened to kill Khalid with his own hands if he ever caught him.
Khalid seemed unconcerned. "The most important thing is that [the militants] are not supported by the people … Shabaab don't have any strategic points. They are in the bush," he said. Increasing the pressure, the US last week offered rewards of up to $7m (£4.5m) for information on seven al-Shabaab leaders, the first time the militant group has been targeted by the Rewards for Justice programme. Khalid said it might lead to some "useful information".
The 17,000-strong Amisom force has notched up gains where others have failed, such as the US and the United Nations in the 1990s, and more recently Ethiopia, which invaded in 2006 but left three years later having failed to defeat al-Shabaab. Amisom is backed by the UN and funded by the international community. Its commanders acknowledge that without this support, it could not function. The troops are from Uganda, Burundi, Kenya and soon Djibouti and Sierra Leone. Some Amisom officials believe this has helped win support among Somalis, who are notoriously hostile to foreign intervention. "This is a very unique partnership between the United Nations and the African Union [AU]," said Augustine Mahiga, the UN secretary-general's special representative to Somalia, in his Mogadishu office.
"What the AU has been doing is peace enforcement. Mogadishu is free, Baidoa is free. It doesn't mean it is the end of al-Shabaab but there are areas of stability and in these areas, we need to keep the peace."
It is certainly not the endgame just yet. In Baidoa last Thursday, one person was killed in a grenade attack on the foreign exchange bureau. The day before, a decapitated body was found nearby. Hundreds of children have also reportedly been snatched by al-Shabaab.
The deputy district commissioner Sandeere Mohamed Iftiim said children aged 14 and 15 had been taken from Baidoa but their families could not talk about it while al-Shabaab were in the town. "If they talked about it, they could be tortured or killed," he said, adding that he and other government officials were trying to encourage rebels, and the children forced to join them, to surrender.
Some of those calls are being heeded. Another official in Baidoa said around 36 "defectors" had formed a counter-terrorism unit. This group, the official said, had facilitated the arrest of 110 al-Shabaab fighters, including 40 members of the group's intelligence arm.
Many analysts say Amisom's next big military challenge will be capturing Kismayo. But Brigadier-General Paul Lokech, commander of the Ugandan contingent, said driving the militants out of the Shabelle region around Mogadishu was even more crucial. "If you liberate Mogadishu and Shabelle, that is where the bulk of al-Shabaab is. That is their centre of gravity," he said.
Lieutenant Colonel Paddy Ankunda, an Amisom spokesman, believes an estimated 250 foreign fighters – from Britain, America, Yemen, Pakistan and elsewhere – in al-Shabaab's ranks will flee if and when Kismayo falls. There have been some signs that militants are heading north, to the semi-autonomous region of Puntland and beyond. There are also fears al-Shabaab could seek to build stronger links with al-Qaida in the Arabian Peninsula in Yemen. Al-Shabaab still carries out suicide bombings and other deadly attacks in Mogadishu. There is no guarantee that further losses in Somalia would reduce the threat from the militants in neighbouring countries such as Uganda, where nearly 80 people were killed in an explosion while watching the World Cup final in 2010, or in Kenya where al-Shabaab and its allies have claimed several grenade attacks. The Islamist militants are not the only threat to peace in Somalia, which is regularly described as one of the world's most failed states: there are also freelance militias, former warlords and unscrupulous politicians. Military officials stress that they can only do so much. Somalia also needs a political solution and there are few who believe the discredited members of the UN-backed transitional federal government can or will deliver that. A new parliament and president are due to be in place by 20 August. But there are reports of bribery and intimidation of the traditional elders who are supposed to choose members of a national constituent assembly, which will then pick the new parliament.
Then there are the guns-for-hire and former warlords, who could re-emerge in the vacuum left by al-Shabaab. There are already reports that freelance militias are harassing displaced people in Mogadishu. Mogadishu's mayor, Mohamud Ahmed Nur, believes Somalia is at the beginning of a new era, one that is fraught with challenges, but not necessarily from al-Shabaab. "This is the beginning of the end of al-Shabaab," he said. "They lost experienced leaders and they lost weapons and manpower. And at the same time, the Somali people turned against them." His concerns centre on the fragile political process. "[Somalia] needs a very strong government with vision. If we continue this way, we are …. I don't know," he says, lifting his hands helplessly. Nur says al-Shabaab will be gone in six months. And yet, in a poignant reminder that the group could live on in some form even after a rout on the battlefield, he says he knows he could be killed any day. "It says in the Qur'an, you don't know where you will die or when you will die. So I will not worry about death or al-Shabaab." via Guardian News