In footage from a documentary about Birmingham, Brady is recorded questioning then manager Rooney's "work ethic".
The former England captain's time in charge lasted only months before he was dismissed by Birmingham's American owners.
Rooney told BBC Sport: "I think Tom came in once, which was the day before a game where the days are a little bit lighter anyway, and I don't think he really understood football that well.
"But what he does understand is, he's a hard worker, we know that."
Rooney added: "Football is not NFL – NFL works for three months a year. Players do need rest as well, so I think he's very unfair, the way he's come out and portrayed that.
"Listen, I respect Tom Brady massively.
"He's one of the greatest (if not the greatest) athletes of all time, and Birmingham do look like they're getting it right now, which is good, and I think what they have done has got the players out that they needed to get out."