Tillerson, who was travelling to Africa, had to return in midway citing "the demands of work and the need to be in Washington for in-person meetings."
For technical purposes, Tillerson said he would remain the Secretary of State till March 31 and till then he would delegate his responsibilities to the Deputy Secretary of State John Sullivan.
The 65-year-old former Exxon Mobil chief executive and the president had strained relations and the two had differences over a host of issues including the US policy on North Korea, Russia and Iran.
In his remarks at the Foggy Bottom headquarters of the State Department, Tillerson did not take the name of the president. He read out from a prepared statement and looked dejected at the turn of events.
Tillerson has been in this post for a little over a year. The period is seen which experienced substantial decline in the power of the State Department and its annual budget was cut in an unprecedented way.
Preparing to leave the State Department, Tillerson said he was committed to an orderly and smooth transition. He planned to leave Foggy Bottom at the end of the day, even though his commission as the Secretary of State expires March 31 midnight.
Tillerson said he would return to private life as a private citizen.
He said Trump called him hours after he tweeted about him being fired. He also spoke with the White House Chief of Staff John Kelly on a smooth and orderly transition.
"What is most important is to ensure an orderly and smooth transition during a time that the country continues to face significant policy and national security challenges," Tillerson said.
"Between now and then, I will address a few administrative matters related to my departure and work towards a smooth and orderly transition for Secretary of State-designate, Mike Pompeo," said the outgoing Secretary of State.