The UI will simply not let me re-roll an affix unless I have at least one shard of that type. This is because you need to be able to press the “up arrow” button on the unwanted affix to get the Reroll button to appear, however there is no “up arrow” button when the player has no shards of that type available.
Pretty sure this is not by design? There’s no reason for it.