There are more quests which grant idol slots and passive slots than you can obtain in total. So you do not need to complete every idol/passive reward quest in order to get all of them. However, you will still need to obtain these from quests, so if you skip too much, you may need to revisit quests to collect your rewards.
This also helps make things smoother for multiplayer, where you may join someone further along, and end up missing a few quests, so you can still potentially obtain all of your idols/passives joining someone further along.