What went wrong?:
Storm Totem’s cast speed is greatly reduced against certain enemies when using its Unmatched Storms node. On killing a Voidfire Maggot or Winternid, for instance, Storm Totem’s next Storm Bolt cast is greatly delayed, see video. The first batch of enemies cause a normal, fast attack speed, while the second batch after traveling slows the casts down considerably.
This effect seems to only occur with the Unmatched Storms node allocated.