In another incident, two soldiers were reportedly killed in an ambush near Ataq, the provincial capital.
The governor of Shabwa, Ali Hassan al-Ahmadi, said: "Abdullah Mehdar was killed last night by security forces, which had besieged the house he hid in."
Security officials said Yemeni forces had surrounded the house, in a mountainous region, and exchanged fire with some 20 militants inside.The remaining militants escaped. The Spanish news agency, EFE, said one member of the security forces had been killed in the operation.
But a local tribal leader told Associated Press news agency Mehdar and the arrested men were not "active members" of al-Qaeda.
"They were young men who went astray, but I don't think they were really members of al-Qaeda," Sheik Atiq Baadha said.
He said local leaders could have handed over the men if they had been approached, and warned that sympathy for al-Qaeda could increase if government forces continued with their current tactics.
On Tuesday, Yemen's foreign minister renewed a call for dialogue with al-Qaeda militants, provided they downed weapons and renounced violence.
But, he added, if they refused, government forces would continue to pursue them.
The spotlight was turned on Yemen after the Yemen-based group Al-Qaeda in the Arabian Peninsula said it had carried out a failed bomb attack on a US-bound airliner on 25 December.
Earlier this week, President Barack Obama said he had "no intention" of sending US troops to Yemen or Somalia to combat militant groups in those countries.Analysts say al-Qaeda militants have been moving to Yemen after coming under pressure in Afghanistan and Pakistan, and following a crackdown in Saudi Arabia.
Last week, Yemeni officials said another local al-Qaeda leader and two other militants had been arrested after being injured in a raid 25 miles (40km) north of Sanaa.