Osaka had 41 unforced errors, her backhand in particular letting her down as she was unable to build on a comfortable first set for the second straight match.
"I'm literally shocked that I won this match," Osaka, seeded 13th, told an adoring crowd in Louis Armstrong Stadium.
"There was a certain point where I felt like I put my entire heart on the court. I just kept trying every point."
After an early exchange of breaks in the second set, Osaka failed to convert a pair of chances and Mertens broke again for a 5-3 lead.
Osaka gifted the Belgian two break points with a double fault, saving one with a forehand winner before sailing a backhand wide.
Mertens finally took the set when Osaka dumped a backhand into the net.
Mertens kept the pressure on in the third and was rewarded with a break for a 3-2 lead, aided by a string of backhand errors from Osaka.
Osaka was clearly frustrated, berating herself and exchanging some testy words with her team.
"I didn't feel like I played amazing; it was really getting me down," she told the crowd, offering an apology for her behavior.
She was down a break, trailing 4-3 when she took a medical timeout for treatment on her left calf and ankle.
Osaka returned with renewed vigor, breaking Mertens at love for 4-4 and saving a break point in the next game for a 5-4 lead.
Mertens didn't go quietly, saving a match point on the way to 5-5 and saving two more in the final game before Osaka secured the win.
US Open 2026
The US Open is the fourth and final Grand Slam of the tennis season, running from August 30th to September 13th. It is a hard-court tournament held at Flushing Meadows in New York City. Follow the links below for more information!
All you need to know | Men's singles draw | Women's singles draw | Latest US Open news