The 30-year-old has been touted as a potential 2019 opponent for IBF, WBA and WBO belt-holder Joshua, though WBC champion Deontay Wilder is his priority for an April 13 fight at Wembley.
Whyte beat former WBO champion Joseph Parker last July to enhance his claim for a title shot and he insisted it is inevitable he will get another chance to take down Joshua, who won their previous encounter with a seventh-round knockout in December 2015.
"I can't bank on what he's doing, I've got to focus on my own career," Whyte told Sky Sports ahead of his fight with Dereck Chisora at the O2 Arena in London on Saturday.
"It would be great if we could get the rematch on, but it's only a matter of time.
"There's only six or seven top guys around and I have consistently proved I am of that level and I am only getting better and better. It's only a matter of time."
— Matchroom Boxing (@MatchroomBoxing) December 17, 2018
Joshua described Whyte as an "enemy" and expressed a desire to see Chisora "whup" him when they meet for the second time this weekend.
Whyte responded: "They're both Finchley boys and they're mates, but I bet he said it with a stupid smile on his face.
"It's cool though. Listen, I've been fighting all my life, climbing this hill my whole life, and I'm not looking for sympathy.
"I don't care, people can say what they want. I think when I fought him [Joshua] there were 50 guys on a panel and I think everyone picked me to get knocked out in the first round.
"And I almost knocked him out in the second. It is what it is. He can say what he wants. I'm easy, I'm nice and I train hard. I am ready.
"Nothing great comes easy. If it comes easy everybody would be doing it."
Photo Getty Images. Caption: Dillian Whyte promoting his fight with Dereck Chisora