What is an overlay? I have heard people talking about pet overlays.

An overlay is the image you use in place of the default pet image. You can code them onto a pet's page, or upload one through the custom image page. If you upload a custom one, you'll see that image in place of the default around the site for whichever pet you've assigned it to.