American Williams, 38, stopped Rus serving out the match at 6-5 before winning 7-6 (8-6) 3-6 7-6 (7-0) in almost three hours.
Rus, ranked 72nd in the world, became edgy as a memorable win loomed and it was costly.
"It was a real physical match," said 23-time Grand Slam champion Williams.
"I played pretty well, I was being aggressive and hit some good shots, but she just kept fighting."
The two-time champion will play Greece's 13th seed Maria Sakkari in the third round.