Before patch 0.83, I had no frame drop in this area, but I started to have crazy frame drop after the patch. It seems to happen only when you have minions.
My necro and my primalists with pets had problem, but my Lich and sentinel who had no minions seem to have no problem at all.
Two things:
Immortal Summit is one of the potentially problem maps when it comes to performance - devs have it on their list.
Minion performance is a general problem with LE and has been around for a while… so its always going to be affected much more than other builds/playstyles when there are potential performance issues with a particular patch or map.