55-year-old Rudd was sworn in as Prime Minister by Governor-General Quentin Bryce at 9.30 am (local time) at a short official ceremony at Government House in Canberra.
Anthony Albanese was sworn in as Deputy Prime Minister and Chris Bowen as Treasurer.
This is the third time in the last three years that a new Prime Minister has taken over in Australia with Rudd reclaiming the Labor Party leadership just three months ahead of the country's general elections.
Yesterday, Rudd had ousted Gillard who lost the leadership battle after Rudd secured 57 votes in a caucus ballot, compared to 45 for Gillard.
She now returned to the backbench.
Rudd said he resumed his old job with "humility, with honour, and with an important sense of energy and purpose".
Addressing Parliament today as Prime Minister, Rudd said politics can be brutal and asked MPs to remember that "in this Parliament, in this place we are all human beings".
"Try, just try to be a little kinder and gentler to each other," he said.
Rudd used his first official comments as Prime Minister to acknowledge the contributions of Gillard and former treasurer Wayne Swan, while talking of the difficulties of political life.
"I've got an obligation to the nation, we are talking about who leads the nation. I am not going to let that speculation run endlessly. This is the moment for Caucus to decide. I accept that outcome," Gillard said earlier.
The leadership change comes ahead of the September 14 general elections, which surveys suggest Labor is set to lose.
It is unclear whether Rudd will stick to Gillard's schedule of the September election or go for an earlier one.
The earliest date Rudd can call an election for is August 3.