Barty has enjoyed a breakthrough year in 2017, claiming her first WTA Tour title and climbing into the world's top 50 for the first time.
However, prior to Wednesday, the Australian had lost all six of the matches she had played against top-10 opponents.
That winless streak was broken in Cincinnati as Barty overcame a second-set wobble to beat Williams 6-3 2-6 6-2.
"To play Venus on such a big stage is a privilege for me and to play as well as I did, I'm really excited," said Barty.
"She's a genuine champion of the game and paved the way for us young players coming through. It's an honour for me to share the court with her, and [I'm] happy to be moving on."
Barty, who completed her win over Williams in style with a forehand winner, will face Elina Vesnina or Caroline Wozniacki in round three.
.@AshBar96 stuns Venus Williams 6-3, 2-6, 6-2!
— WTA (@WTA) August 16, 2017
Photo Getty Images. Caption: Ashleigh Barty on the WTA Tour