You're totally right! Only the user skill matters. Thet pet skill is something I've been working on / noodling over recently, with gardens as well.

The idea is to give pets these RPG skills that you can level up that are on the profile, so you can have pets dedicated to specific things. I haven't found the right way to make that in the game yet..