"I was aware what the guys were saying -- something about my passport -- but I blanked them," he told The Daily Mail in an interview conducted in New Zealand where England are touring.
"It was only later that Rooty (Joe Root the England captain) said the guys got ejected.
"It was the first time I'd seen someone get ejected from a ground, because there were some abusive fans when we played Pakistan at Trent Bridge (heckling Ben Stokes).
Archer, who says an elderly spectator at a county game with Kent had once queried how was he playing for Sussex, said racist incidents occurred far less in cricket than football.
"The world's changing," he said.
"It's becoming more multicultural. A lot of people have accepted it for what it is.
"Look at the England cricket team -- there's huge diversity.
"It's the same with any football club in the world.
"I think people have to accept it. Times have changed, it's not 2007 any more."
Archer says he sees himself as a role model to young British West Indians who have aspirations to play cricket for England.
"Yeah, to let them know it's possible," said Archer.
"It doesn't really matter where you're born.
"If you know that cricket's what you want to do, you never know where you'll end up.
"I didn't know my dreams would come true and I'd end up playing cricket for England.
"If it happens for me, it can happen for anyone."
"Not many England fans knew who I was anyway, so if they saw me in the street they probably thought I was a footballer, or something," he said.
"I guess that was the beauty of it, being able to go under the radar."