It made its warning jointly with think tank British Future.
According to the BBC, figures due to be published on Thursday are expected to show net migration is more than three times higher than the government’s target.
The Home Office says it remains committed to reducing the net total to the tens of thousands.
The Institute of Directors and British Future say radical policies to prevent migrants coming to the UK will damage the economy.
The two groups are calling for a “comprehensive immigration review” to establish what policies could be put in place to achieve the tens of thousands target, and what their impact would be on the UK’s economy, culture and society.
They say the prime minister and Home Secretary Theresa May should ask an expert body, the Migration Advisory Committee, to set out plans for dealing with immigration and their likely impact.
These should then be put to the general public through town hall meetings across the country and a “citizens’ jury” to reflect public opinion, the groups say.
Sunder Katwala, director of British Future, said: “The prime minister currently has no long-term plan to meet his net migration target. It is little wonder there is such low public trust on immigration and the government’s ability to manage it.
“Responding to each damaging set of immigration figures with new ad-hoc policies is no way to manage such an important issue.
“A comprehensive immigration review would set out what’s possible and what isn’t. It could also give the public more of a say in what happens – providing impartial facts about the impacts of different policies on the economy, society and public services.”
Simon Walker, director general of the Institute of Directors, said: “Scrabbling around to find measures to hit a bizarre and unachievable migration target is no way to give British businesses the stable environment they need.
“Combined with ministers’ increasingly strong rhetoric on immigration, the UK’s reputation as an open, competitive economy is under threat.”
The prime minister vowed to get net migration down to “tens of thousands” each year “no ifs, no buts” in a speech in 2011.
But the most recent figures in May showed net migration in the UK stood at 318,000 in 2014.
Speaking in February, immigration minister James Brokenshire said the government had been “blown off course by net migration from within the EU, which has more than doubled since 2010”.
Measures introduced by the government to reduce net migration have included capping visas for skilled non-EU workers and introducing a minimum income threshold of £18,600 for anyone planning to sponsor a non-EU family member, such as a spouse, to come to the UK.
The government has targeted abuses of the student visa system by closing “bogus” colleges.
Meanwhile, other figures also due out on Thursday are likely to show that more than eight million people are living in the UK who were born overseas.
That data will be released by analysts at the Migration Observatory at the University of Oxford.
BBC home editor Mark Easton says the sight of desperate refugees trying to get into Europe and a few trying to smuggle themselves into Britain is thought to have helped make immigration the top concern for voters.
Ministers are under pressure to show they have control of the UK border, he added.