Because you can get max bonus passive points and idol slots without doing all the sidequests, a lot of sidequests are pointless as a result. I think it would help if there were more sidequests with a unique item as a reward or something else that is useful besides the usual rewards.
I know you can do this because of the sidequest that gives you one of two uniques and the vendor sidequest that gives you a ton of gold.