Tried both , almost all guides have spell damage but seems to me physical damage is better since Hammers are the attack skill we spam and smites are autocast from idols ,did i miss something
I think you missed the part that smite does fire damage. So scaling physical damage won’t scale smite at all. You are just scaling the hammers themselves when you do that. You scale one or the other.
Also, proc’d skills that can go on your skill bar don’t inherit anything from the proccing skill unless explicitly specified (like the Spellblade’s Firebrand does).