There is an achievement for that. Is giving something to the Red Rreign something different than throwing it into the fire?

Plus: will he burn Fireside staff or do we need to have something non Fireside in our inventory in order to have it burnt?

It will burn an item that you give him, that he doesn't like. Not just a random one in your inventory.

As far as the giving item one, people suspect it's a glitched one, and it isn't working right now.

It is working now, I got the Achievement.