I've tried making some based off a paint shop editor but it comes out really pixeled and not smooth like the clothes here...

Am I doing something wrong?

hi! It depends on a lot of different factors!

How big is the canvas? Are you using a soft or sharp brush? Do you use pen pressure?

It's also possible your paint shop editor simply doesn't resize too well. I know there are programs better suited for soft resizing than others ;c

If you want, feel free to mail me with a screenshot and I'll see if I can help!