On the morning of December 28 1979, I happened to be shopping in the main bazaar of the southern Afghan city of Kandahar. It was a beautiful, cold, crisp Afghan winter morning with a bright sun and an azure sky, and the bazaar was a cacophony of noise, chatter and traffic. Earlier I had heard unclear accounts on the BBC World Service of a coup the night before in Kabul that deposed President Hafizullah Amin, and suddenly, around midmorning, there arose a deep-throated rumble that seemed to be emerging from the bowels of the earth. Puffs of dust rose from cracks in the asphalt and people began to shout that an earthquake was starting.

Within minutes the entire road was shaking and the rumble had become a roar that drowned out the shouting. In the distance we spotted a long line of Soviet tanks, armoured personnel carriers and lorries, clanking their way down the main road to the shops. The Soviets had crossed the border at Kushka in Soviet Turkmenistan and driven over 600 kilometres through the city of Herat to arrive in Kandahar – along the very road that Moscow had built in the 1960s.

Every adult Afghan remembers where he was and what he was doing when he saw his first Soviet soldier in December 1979. For the Afghans – and for those who have stood by them through the years of war, occupation, civil war and war on terror that followed – it is a moment burnt into memory, and it is recalled with precision, like September 11 or, for Americans of a certain generation, the death of John F Kennedy.

I remember it all vividly: the bazaar in Kandahar abruptly fell silent – the only time I have ever seen it so. Everything happened in silence, in slow motion, a movie advancing frame by frame. Afghans frozen in awe held onto their shopping bags, rooted to where they stood. Women in shock lifted their burqas from their faces to get a better view.

The first tanks drove nonchalantly into the bazaar and stopped; their hatches opened and young Soviet soldiers, barely older than teenagers, jumped out carrying snub-nosed machine guns. Dusty, tired and stiff, they ambled to the nearest tea stall and ordered cups of piping hot green tea. We watched them in absolute silence; the Afghan soldiers and police in the bazaar offered no resistance. Before long there were hundreds of Soviet soldiers flooding the bazaar, and large crowds of Afghans gathered to stare at them as if they were visitors from the moon – and they might as well have been, as far as the Afghans were concerned. The country had been invaded by bands of hippies, but no Afghan had seen an armed foreigner since the British arrived more than a century before. Decades later, when the American attack on Afghanistan commenced, there were so few US soldiers on the ground that most Afghans did not see a single American.

Since the first American bombs fell on Afghanistan in October 2001, a cottage industry of doomsayers has arisen among academics and journalists, warning that the US will fail in the so-called "graveyard of empires" just as the Soviets did.

As Barack Obama takes office – and reiterates his intention to send 30,000 more troops into Afghanistan, such prophecies have returned anew, insisting that Afghanistan is a cesspool of ungovernable tribes, unscaleable terrain and unwinnable wars.

But to compare the American and Soviet invasions is misguided, as Gregory Feifer's brilliant and timely new history of the Soviet war, The Great Gamble, makes clear: the Soviets had no support inside or outside the country when they blundered in with their tanks to prop up an unpopular Afghan communist government that took power by coup.

The Soviet army, badly underfunded and overwhelmed by adverse conditions in Afghanistan that drained troop morale, relied on massive firepower to kill and maim Afghans rather than winning their hearts and minds. So long as the Afghan mujahideen maintained their sanctuaries in Pakistan, where they were rearmed and funded by the CIA – and where they could recruit from among the vast pool of Afghan refugees – the Soviets could not defeat them.

Feifer, a Moscow correspondent for National Public Radio, dug up old Kremlin documents and met the officers who planned the war and the soldiers who fought it to construct an intimate portrait of how it was fought and ultimately lost.

He does not compare the Soviet war to the American one, but it is impossible to read his account without thinking about how the Afghans have reacted to their new occupiers. For both the Soviets and the Americans tried to remake Afghanistan in their own image, and both failed to do so – but the Americans, at least, still have a chance to succeed.

While the UN Security Council condemned the Soviet occupation of Afghanistan annually, the American attack had universal international backing. There was overwhelming support among Afghans for the overthrow of the Taliban regime, because it came with the promise that US and the international community would rebuild the country and better the lives of ordinary Afghans – a promise that kept most Afghans on the side of the US for several years after the invasion. The fall of the Taliban was followed by the creation of a new interim government in Afghanistan led by Hamid Karzai, which – for a few years, at least – enjoyed the support of a broad majority of Afghans; presidential and parliamentary elections followed, legitimising the political system now in place.

Nevertheless, eight years after the American invasion the reinvigorated Taliban insurgency has now reached the gates of Kabul and neither US objectives nor Afghan hopes have been fulfilled; those who argued in 2001 that Afghanistan could never be subdued today press their case with renewed confidence. But these arguments demonstrate a poor understanding of Afghan history: it is true that the Afghan state was never colonised in the 19th century, but ultimately it did become a kind of informal protectorate of the British in India.

For nearly two centuries, until the Soviet invasion, the Afghan kings ensured that they maintained a modicum of ethnic and tribal peace within their realm and that Afghanistan did not pose a threat to neighbouring states. It is true that they were never able to build a modern, centralised and powerful state, but they did create what I would call a "minimalist state" that ensured a degree of stability, if not development.

With more modest political and social ambitions in Afghanistan and a broad regional alliance of neighbouring countries who pledge not to interfere in Afghanistan – an aspect of the conflict the Bush administration ignored even before turning their attention away from Afghanistan toward Iraq – the American experience in Afghanistan need not end like the Soviet one.

The era of Soviet influence in Afghanistan began in the 1950s, when both the US and USSR waged a Cold-War competition for influence in the country. Both superpowers built modern roads, dams and power plants, but in the 1960s the Americans opted out, preferring to bolster neighbouring Pakistan as a regional linchpin to prevent Soviet expansion southward to the warm waters of the Gulf. Soviet influence continued, particularly in training and arming the Afghan army. Officers were taken to the Soviet Union, where they trained and learned Russian – and became supporters of the communist cause.

In April 1978 a coup from within the army toppled the government of President Mohammed Daud, killing him and his family. Five years earlier Daud had ousted his cousin, the former King Zahir Shah, and ended the monarchy to establish a republic that gradually became a dictatorship. The 1978 coup was led by officers loyal to the small, communist People's Democratic Party of Afghanistan (PDPA), which was divided into two factions: Khalq (The Masses), a largely Pashtun group with its roots in rural Afghanistan and Parcham (The Flag), made up of a wider array of more urbane and better educated individuals that included some Tajiks and Uzbeks. The Khalqi leader Nur Mohammed Taraki, a journalist and writer, became president, but a bitter internecine war followed.

Taraki quickly ousted the Parchamis from any positions of power, imprisoning, killing or exiling them – setting in motion an endemic power struggle that was to devour the party from within. Meanwhile young and enthusiastic but untutored PDPA cadres were sent into the countryside to carry out rapid communist reforms, in the process committing cardinal social sins in a deeply conservative and Islamic land. Tribal chiefs and landlords – the rural elite – objected to the expropriation of their property and houses in the guise of "land reform" and provided the initial resistance to communist rule; tribal and clan ties between the elites and their peasant workers fostered widespread support for the resistance.

Within months Taraki was begging the Soviets to send in forces to save his regime, arguing that the Afghan army of 90,000 men was rapidly disintegrating. In a telephone conversation in March 1979 recounted by Feifer, Taraki begged the Soviet prime minister Alexei Kosygin for help: "I ask you to extend assistance," he said. "I suggest you place Afghan markings on your tanks and aircraft and no one will know the difference."

Old, dithering and unable to inspire confidence, Taraki was soon challenged by his deputy Hafizullah Amin, who a year after the coup jailed Taraki and then had him smothered to death with a pillow. The bitter infighting within the PDPA and the growth of the armed resistance posed a fatal threat to the party's hold on power. The Soviets, who had provided advisers, massive aid and military equipment to the PDPA, saw their latest satellite state slipping away or worse still, falling into the hands of the Americans and their Pakistani allies, who were backing the Afghan mujahideen.

After much haphazard deliberation in the Kremlin by the ageing fogeys of the Politburo, which Feifer ably narrates, the Soviets decided to invade Afghanistan on December 12, 1979. Two weeks later, the Soviets killed Amin and replaced him with Babrak Karmal, the leader of the Parcham faction. Karmal remained in power for almost seven years, but never recovered from the fact that he had been hoisted into power on the back of Soviet tanks. The mujahideen multiplied under America's lavish patronage, and Afghanistan became the front line in the most confrontational new phase of the Cold War in Asia since Vietnam, as a belligerent Ronald Reagan demanded nothing less than victory. By 1988, the Soviets had had enough and Mikhail Gorbachev, facing enormous economic and social problems within the USSR, moved to withdraw all Soviet troops by 1989, bringing an ignominious end to a brutal and costly campaign.

A few weeks after September 11, the US secretary of defence Donald Rumsfeld sent a group of generals to Moscow to talk to retired senior former Soviet officers about their experiences in Afghanistan. The Russians told the Americans not to even try to conquer and occupy the country or rule it; the Afghans were impossible to govern and the terrain was too hostile for modern armies. Their unanimous advice was: "Get al Qa'eda and then get out."

As Washington prepared for war, pundits and commentators pointed to the disastrous Soviet defeat in Afghanistan as a warning; they penned long thinkpieces explaining that the Afghans had held every invader since Alexander the Great at bay. Ultimately Rumsfeld and the CIA chief George Tenet gambled and sent some 600 Special Forces troops and CIA agents, who with sacks full of money and the use of US air power mobilised the anti-Taliban resistance called the Northern Alliance and won a quick victory, routing but not defeating the Taliban and al Qa'eda.

Today the Afghan people are still paying the high cost of Rumsfeld's refusal to put boots on the ground early enough to actually defeat the Taliban and stabilise major population centres. It was a mistake Rumsfeld would repeat in Iraq.

Subsequent accounts from American forces in Afghanistan suggest they pressed Washington with demands for more troops to block the retreat of al Qa'eda and Taliban leaders into Pakistan. But Rumsfeld, already preoccupied with planning the war in Iraq, was determined not to utilise the forces he was holding in reserve for the assault on Baghdad; instead the Americans in Afghanistan were told to co-operate with unreliable warlords, who were paid heavily to block escape routes. After taking money from the Americans to prevent the escape of al Qa'eda, they happily accepted more cash from al Qa'eda to ensure safe passage.

The Bush administration did fail – but they failed in a very different manner than the Soviets. Already intent on confronting Saddam Hussein, the Americans had no strategy for the day after the fall of the Taliban, for the moment when Afghanistan needed to be stabilised and rebuilt. Between 2001 and 2004 Washington did not have any comprehensive strategy for rebuilding the country or providing adequate security to its citizens beyond attempting to capture al Qa'eda fighters. The renascent Taliban in their Pakistani safe havens were ignored by America for too long, while the US lavished $12 billion on Pervez Musharraf, who pursued a dual strategy: going after al Qa'eda, but clandestinely supporting the Taliban.

The American failures in Afghanistan were not foreordained by Afghanistan's unyielding terrain or fractious tribal politics: they were failures of decision-making and commitment in an attempt to achieve ambitious goals with minimal resources. George W Bush, who disdained "nation-building" as he ran for president in 2000, had no plans to do it in Afghanistan.

The country's battered infrastructure is still badly in need of repair, without which the Afghan economy cannot function. The economic development that Afghans eagerly awaited has not materialised. Poorly co-ordinated international aid payments were squandered on advisers and high overheads, and large chunks of the promised aid never reached the Afghans. The incompetence and self-dealing of the aid process was matched by corruption permeating Afghan government, fuelled further by the booming drugs trade.

The government's writ barely extended beyond Kabul, while the countryside was controlled by rapacious warlords empowered by the Americans. Rather than rebuilding state institutions like the army, civil service and justice system, building up capacity within government ministries and reconstructing the basic infrastructure required for economic development, the Americans chose the path of least resistance –providing inadequate funds and handing over security in the countryside to the warlords.

The Soviets did have a nation-building strategy: they wanted to turn a conservative, largely rural, underdeveloped multi-ethnic country into a modern communist state. They looked to the rapid spread of communism in Eastern Europe as a model, ignoring the fact that by 1979 communism was failing in those states and, indeed, in the Soviet Union. So the Soviets built factories in Kabul to create an Afghan working class that would carry forward the communist revolution, forcibly carried out land reforms that would give plots to the peasants, pushed for mass education and equal rights for women, downplayed Islam and brought thousands of Afghans to the Soviet Union to turn them into Marxist cadres.

But the Soviets still failed, because they had invaded and then occupied Afghanistan without international or regional support, and without any real base of support inside the country.

The Afghan communists were a handful of disparate, bickering, even murderous ideologues whose only aim was to emulate the Motherland of Socialism – the Soviet Union – rather than formulate a gradual, slow modernisation of their country that would be non-communist in tone and not antagonise the religious, feudal and tribal codes that really mattered to the people. In fact every Soviet reform antagonised vast sections of the population. Land reform mobilised the landlords and khans against the regime, while the peasants went hungry as they had neither the tools nor the cash to invest in their new plots.

The mullahs rejected coeducation and burnt down schools while declaiming the lack of reverence the Afghan communists showed toward Islam; factories lacked markets or raw materials. By crudely emulating the Soviet Union as a model of perfection, the Afghan communists tied a noose around their own necks.

At the same time the Soviet army carried out a hugely destructive scorched earth policy to defeat the mujahideen. By the time the Soviets withdrew in 1989, 1.5 million Afghans were dead and five million had become refugees in Pakistan and Iran. The country lay in ruins and its citizens faced a humanitarian disaster; even the minimalist state that had existed before 1978 had ceased to exist, paving the way for the even more brutal destruction wrought by the civil war that brought the Taliban to power.

The vast majority of Afghans – perhaps as many as 90 per cent – welcomed the end of the horrors perpetrated by the Taliban regime. Afghan expectations were enormous: after 23 years of war and loss they hoped their lives would change for the better, their country would be rebuilt, stability and security would be ensured, the role of the gun and the warlords would be reduced and their children would go to school and have health care.

But for Washington, Afghanistan became a holding operation while the real war moved to Iraq. The Americans did provide sufficient troops to maintain security – but when more troops and aid finally arrived in 2004, the US military continued to make serious tactical mistakes in its battle against the Taliban insurgency. Even today, however, many Afghans continue to support the American presence – clinging to the hope that as long as the Americans stay, there is a chance their country will be rebuilt.

The Soviet army faced an altogether more difficult environment. Deprived of any kind of legitimacy or moral sanction by the Afghans, they faced a losing battle from the start. Some 620,000 Soviet soldiers served in Afghanistan during the 10-year occupation, and 13,833 were killed, according to official numbers that are almost certainly a fraction of the real tally. Nearly half a million Soviet soldiers were forced from the battlefield because of wounds or sickness, and more than 10,000 became invalids.

Compared to the American forces of today, the Soviets ran a Third World army. Again and again Feifer draws out the horrors of being a Soviet soldier, the hazing, the buggery and rape, the lack of modern equipment or vehicles, the shortage of warm clothes, insufficient food and water and the consequent stealing and petty crime to which most soldiers resorted so they could survive. When there was food, there was no fuel for cooking it. There was no heating in the harsh winter months and no air conditioning in the blistering summer. Basic diseases were rampant. Malaria, dysentery and hepatitis decimated the troops far more efficiently than mujahideen bullets. There were never enough chlorine pills to decontaminate water supplies, and the looted sleeping bags of dead mujahideen became prized possessions among the Soviet forces. Drug taking became rampant to the point where some Afghan guerrilla commanders made sure that Soviet outposts were supplied with free cannabis and heroin so that the garrison was incapacitated. Moonshine vodka was brewed from anything that was available including fuel oil and plants. Demoralisation was rampant and desertions multiplied.

The Soviets entered Afghanistan employing the tactics they had been trained to use against a Nato army invading from Europe, and it took at least five years for them to embrace the tenets of counterinsurgency warfare. They fought the mujahideen with overwhelming firepower – artillery, aerial bombardment, ad the wanton destruction of houses and villages to deprive the mujahideen of cover. I saw peaceful villages razed to the ground, with their farmland salted or mined to prevent anything ever growing there again.

Instead of halting the resistance such atrocities delivered more recruits to the mujahideen, while the Afghan army shrank from 90,000 men to just over 30,000 in the first year after the Soviet invasion, due to massive desertions, insurrections against their Soviet officers, appalling conditions and poor logistics.

The mark of a Third World army is that its discipline and command structure rapidly deteriorates in a crisis, and that is precisely what happened to the Soviets in Afghanistan. The highly disciplined US commandos who led the Northern Alliance forces into battle against the Taliban in 2001 faced no such problems, though later they too suffered from a lack of critical equipment as the war in Iraq sucked resources out of the country.

American and Nato forces have also employed excess firepower, in order to make up for the lack of manpower, and errant aerial bombings have killed thousands of civilians and elicited harsh criticism from Karzai. But the counterinsurgency strategies introduced in Iraq and quickly imported into the Afghan theatre have yielded some positive results, and the Provincial Reconstruction Teams deployed in all of the country's 34 provinces – small groups of US or Nato soldiers aided by civilians who carried out small reconstruction projects as well as train Afghan army and police – have had modest successes.

But the Americans still face the same problem the Soviets found insurmountable: the safe havens available to militants across the border in Pakistan. In the 1980s the Pakistanis reaped the benefits of becoming a front-line state in the Cold War, harvesting billions of dollars in aid while acting as the conduit for Western money and arms secretly funnelled to the mujahideen through Islamabad. Pakistan's powerful Inter-Services Intelligence (ISI) virtually ran the war, distributing Western largesse to the Afghans, planning battles and campaigns and recruiting and training fighters. Through the 1990s Pakistan supported a series of Pashtun warlords, eventually putting their considerable weight behind the Taliban, whose leaders – along with thousands of their followers and the remnants of al Qa'eda – fled to Pakistan and have remained there ever since.

Barack Obama has pledged to withdraw US troops from Iraq while stepping up the American commitment in Afghanistan. He has yet to fully flesh out the policy he will pursue, but seems to understand that what is required is a "comprehensive surge" that goes beyond new troops and new tactics to increase and coordinate development and reconstruction, provide security to the Afghan people and embark on a diplomatic initiative to bring Afghanistan's multiple meddling neighbours together to stabilise the country and end the sanctuary the Taliban still enjoy in Pakistan. Obama has appointed a special envoy to the region, the seasoned senior diplomat Richard Holbrooke, and has begun to get tough with the Karzai government over corruption and the drugs trade.

The Bush administration lacked an overarching strategy for Afghanistan and its neighbours, and Obama does not want to repeat that mistake. He has already announced orders to close the prison at Guantanamo Bay and he will soon make a historic speech in a Muslim capital, where he is likely to repudiate Bush's "global war on terror" and announce a policy of talking to militant groups, including the Taliban, while continuing the pursuit of global jihadists. Bush left regional diplomacy largely in the hands of the Pentagon, while Obama will restore the role of the State Department. Several senior administration officials have acknowledged that they cannot "shoot their way to victory" in Afghanistan.

In the interim, however, the situation in Pakistan has worsened substantially, with the rise of the Pakistani Taliban, heightened post-Mumbai tensions with India, an insurgency in Balochistan and grave rifts between the elected government and the powerful army. Ironically it may be the case that Afghanistan poses less of a challenge: the Taliban, although feared, are still intensely disliked, and the country can be stabilised with a massive infusion of money and troops alongside proper strategies for reconstruction and the rebuilding of state institutions.

Ultimately Feifer's book reminds us that it is not impossible to rebuild Afghanistan as a secure and stable state. The Soviets failed because they tried to apply a model that had nothing to do with the nature and traditions of the Afghan people. The Americans have also failed, mostly because they have refused to dedicate resources sufficient to win hearts and minds and defeat Taliban forces.

It is still not too late for the Americans to reverse course: the majority of the Afghan population has no desire to return to Taliban rule. What gains the Taliban have made can be attributed to fear and intimidation – and the inability of the Kabul government to provide security and economic development. Many of the Taliban are fighting for local grievances and it will be worthwhile for the Americans and the Afghan government to engage these commanders and soldiers in dialogue with offers of amnesty. Afghanistan is indeed winnable – and it will be won not by military force alone but by giving the Afghan people what they were promised after September 11: a new life.

Ahmed Rashid is the author of Taliban and Jihad. His most recent book is Descent into Chaos: How the War Against Islamic Extremism is Being Lost in Pakistan, Afghanistan and Central Asia.He lives in Lahore, Pakistan.