Tersi's decision to quit came four days after the Italian government reversed its earlier decision not to send back to India, Massimiliano Latore and Salvatore Girone, who faced murder charges for shooting dead two Indian fishermen off Kerala coast February last year.
Italy had first reneged on its assurance to Indian Supreme Court of sending back the two marines but later backtracked after both the Indian government and the apex court stepped up the heat amid New Delhi's warning of downgrading ties with Rome.
Terzi announced his resignation in Parliament, Italian news agency ANSA reported.
"I resign in contention with the decision to send the marines back to India. The misgivings I expressed had no effect on the decision taken," the 66-year-old told parliament, indicating a rift in the government over the marines' issue.
"I am resigning because for 40 years I have maintained, and still maintain, that the reputation of the country, the armed forces and Italian diplomacy, should be safeguarded," he said.
"My reservations had no impact and the decision was not mine," Terzi said before parliament in a statement of resignation. "My voice went unheard".
"I am also standing down in solidarity with our two marines and their families," Terzi added.
Taking a different line, Italian defense minister Giampaolo Di Paola vowed not to resign and to stand by the two marines.
"I know what Massimiliano Latorre and Salvatore Girone, looking me in the eye, said on the night of March 21: 'Don't abandon us'. I won't abandon the ship," he told parliament.
"It would be easy to resign, but it wouldn't be right and I won't do it," Di Paola added.
Outgoing Italian Premier Mario Monti said he was "stunned" by Terzi's decision and also did not "share" the ex-foreign minister's stance.
Monti said he would appear before both houses of parliament on Wednesday to discuss the case, answering calls from ex-premier Silvio Berlusconi's People of Freedom (PdL) party, which has long been critical of the premier's handling of the affair.
The Italian marines had flown back to New Delhi on March 22 after securing assurance from India that they will not face death penalty nor will they be arrested, bringing to an end the 11-day diplomatic row between the two countries.
They were accompanied by Italian Deputy Foreign Minister Steffan de Mistura in a military plane.
The dramatic U-turn by the Italian government, enabled the marines to meet the deadline set by the Supreme Court when it gave them permission to go for a month to vote in the elections here.
Defending their initial decision not to send back their marines even at the cost of diplomatic breach of assurance, Mistura had said in New Delhi that death penalty was unacceptable and became an issue for the Italian government when the Supreme Court talked about setting up of a special court to try the soldiers.
As the diplomatic crisis escalated, India forbade Italian ambassador Daniele Mancini from leaving the country and airports were put on alert.
External Affairs Minister Salman Khurshid had said that no deal has been worked out with Italy for bringing back the marines but the government had "clarified" to authorities in Rome that the case does not entitle death penalty.
The two marines were serving as security guards on 'Enrica Lexie', an Italian oil tanker, when the incident occurred last year.