Michael Phelps is entering the last year of his thirties, and he is feeling it.

The 23-time Olympic gold medalist, who turned 39 on Sunday, June 30, says that he can feel himself getting older. “Every single year I feel it,” he tells PEOPLE while discussing his longstanding partnership with OMEGA. “My body gets more and more sore when I get out of bed every single day.”

While Phelps is doing “nothing special” for his birthday — “Just waiting for my 40th and then I’ll be forward towards my 50th,” he quips — the retired swimmer is planning to partake in his favorite pastime. “This week I’m going to play golf. I’m going to probably at least play 18 [holes]…. Maybe 27…. Maybe 36,” he says.

His 39th birthday is not the only milestone Phelps is celebrating during the month of June. The swimmer celebrated his eighth “legal anniversary” with wife Nicole Johnson earlier this month.

After getting engaged in 2015, the two later wed in 2016, holding three separate ceremonies. They have since welcomed four sons: Boomer, Beckett, Maverick and Nico.

“So it’s kind of funny, but we have two anniversaries,” Phelps says of him and Johnson, 38.

“We have a legal anniversary where,” he says, referring to the fact that they first tied the knot in a private backyard ceremony so Johnson could have a legal name change, which would make international travel for the 2016 Rio Olympics with Boomer easier. “We had our first child before we got married. So to travel to Rio overseas, we decided that to make it easier. We would just have a little small celebration. Somebody married us in our backyard.”

“So I guess that’s one of our anniversaries and our other anniversary is in October,” Phelps adds, referring to their destination wedding celebration in Cabo San Lucas. “That’s the one that we probably celebrate the most.”

Eight years into marriage, and four kids later, Phelps says the best part of being married to the former Miss California USA is “just each day looking into her eyes.” He tells PEOPLE, “Every single day is so different and we do have four amazing little kids, and I know there’s nobody I’d rather spend the rest of my life with.”

Phelps and Johnson welcomed Nico in January, and are commemorating their family in a special way. “My wife gave me a really cool limited edition CK 1492 for one Father’s Day,” the swimmer says, referring to the OMEGA watch. “We have to engrave the other two kids’ names on the back — we only have two kids on the back now — but that’s such a special timepiece for me.”

Looking forward toward his forties, and navigating them alongside his wife, Phelps says, “I’m not saying every single day is easy, but I think that’s why we’re together because we challenge each other in ways to be better and grow together. So that’s something that I adore.”