U.S. President Barack Obama’s decision to release five Taliban commanders from Guantanamo Bay in exchange for the only U.S. soldier held in captivity by the Taliban sparked controversy and debate this week.

Despite claims by the Afghan government, which was not involved in the swap, that the prisoners would be brought to Afghanistan according to an agreement made with Washington, the U.S. Ambassador to Afghanistan has denied any such agreement.

The decision to make the prisoner trade was made by the White House without the knowledge of the U.S. Congress or the Afghan government, which has caused leaders in both countries to scramble responses in the days since. Some have questioned the dangers of letting such high-ranking Taliban leaders go free, others have suggested the swap was related to broader reconciliation efforts.

Of those who have framed the prisoner exchange in the context of the peace process, Afghan officials have said that the prisoners are expected to be handed over to the Afghan government in line with an agreement between Kabul and Washington. But on Tuesday, U.S. Ambassador to Afghanistan James Cunningham maintained that the releases were not part of the peace process and were instead strictly bound by the conditions of the prisoner exchange and nothing more.

“We didn’t have an agreement to bring them to Afghanistan, it is always been the intention that they would go to Doha and that has been known to your government for quite sometime,” Ambassador Cunningham said.

U.S. officials have said the five Taliban commanders that were released in exchange for Bowe Bergdahl, who was held captive in Pakistan for five years, would be held under close surveillance by the Qatari government in order to prevent them from posing any threats.

“The discussions to achieve this were carried out with the help of the government in Qatar, our discussions were actually with them and we are very grateful for their support,” Cunningham added.

But President Hamid Karzai was reportedly enraged when he found out about the prisoner exchange, because the deal was made behind his back. Karzai has tried desperately to open up lines of communication and negotiation with the Taliban over the past year to no avail.

In Washington, a number of Republican leaders have also mounted criticisms of the trade, which they said violated U.S. law because it was made without the consent of Congress. They have also suggested the trade was a bad idea in the first place given the prominence of the five leaders.

“Well, I think, while all of us are incredibly relieved that he’s been released and for him and his family, there are concerns about national security,” Republican representative Mac Thorberry said. “One is does this indicate to terrorists that we will negotiate with them and reward more kidnapping? Secondly, what on Earth’s gonna happen with these people that we released, five very dangerous Taliban leaders, supposedly they are not supposed to leave Qatar for the next year, but there’s a lot of questions about that, and thirdly the president violated a provision of law in not giving Congress advanced notice.”

But the White House has defended the legal grounds for Obama’s decision-making process. “As Commander-in-Chief, the president had the responsibility to take the action he did, to ensure that as Chairman Dempsey said, our only remaining prisoner of the conflicts in Iraq and Afghanistan was safely recovered,” White House spokesman Jay Carney said.

Meanwhile, a number of Afghan experts have said that the controversy over the prisoner swap is just one example of how cold relations between Washington and Kabul have ultimately hurt Afghanistan.

“The lack of direct contact between the leaders of the two countries provides a chance for various countries and groups to take advantage of it,” Kabul MP Shukria Barakzai said. “When we discuss the issue of US forces in Afghanistan, when we see the release of Taliban commanders and when we discuss aid, we know that nonexistence of relations led us to this and Afghanistan becomes the loser of the U.S. war on terrorism.”

The five Taliban detainees that were released are as follows: Mullah Khairullah Khair Khawah, who served as Minister of Interior and Governor of Herat under the Taliban government; Mullah Fazel Mazloom, the Taliban Army Staff Chief of Staff; Mawlawi Abdul Haq Waseeq, the Deputy head of intelligence under the Taliban; Mawlawi Noorullah Noori, former Governor of Balkh and later Kandahar Army Chief for the Taliban; and Mawlawi Muhammad Nabi Omeri, a senior Taliban leader in southern Afghanistan.

The five militants are said to be close companions of Mullah Muhammad Omer, the reclusive Taliban leader.