"I don't think you can forgive him," Harmison told talkSPORT.
"When you're known as a cheat -- and he is, I'm not going to sugar-coat it, the three guys cheated -- that's on your CV.
"You're marked and you take it to your grave.
"Whatever Steve Smith does, he'll always be remembered for what happened in South Africa."
Harmison added that is "something he's got to live with".
"I can't see anybody's opinion changing on Smith, Bancroft, or Warner. Because of what they've done, they've tarnished the game," the former England pacer said.
Smith was last year handed a one-year suspension by Cricket for his involvement in the Sandpaper Gate fiasco, which also saw Cameron Bancroft and David Warner getting banned for nine months and one year, respectively.
Making a strong comeback into the national squad, Smith has collected a total of 671 runs in five outings so far, nearly the double of what the second-best has in eight innings (Ben Stokes, 354 runs).
The star Aussie batsman recently took over the No. 1 ranking from Virat Kohli and has been applauded across the cricket fraternity for his heroics in the ongoing Ashes series.
(With IANS Inputs)