India`s grievance with Pakistan was that about 42 terrorist camps were functioning in that country, Antony told reporters here.
"Without destroying these terrorist camps, it would be difficult to have permanent cooperation in Indo-Pak relations," he said.
However, government view was to continue with dialogue, he said, adding, Defence Secretaries of India and Pakistan would hold discussions on May 30 and 31.
Emphasising on the need for dialogue, Antony said India wants to have good relations with Pakistan.
"Dialogue is an important factor. Without dialogue nothing can be achieved," he said.
Antony said government`s stand was clear that it wants to maintain a cordial relationship with all its neighbours.
"We can change our friends, but we cannot change our neighbours," he said, adding "We know the reality that we have to live along with our neighbours. Even though we have problems with our neighbours, we are trying to re-establish our friendship."
Pointing out that India has boundary dispute with China, he said efforts were being made to solve the problem.
The Indo-Chinese defence cooperation which was stopped recently, would be started again and in June a defence group from India would be visiting China, he said.
India cannot agree on all stands being taken by the US, Antony said, noting that large quantity of arms supplied by America to Pakistan in the name of fighting terror in Afghanistan was being used against India.
India had already brought this to the notice of the US, he said.
To a query on Adarsh Society scam, Antony said, "I have ordered a CBI probe without anybody asking. Without getting the CBI report we cannot accuse anyone," he said.