Clicking on party members avatars activates abilities

I play in party with my friends. Oftentimes I want to portal to other player, so I right-click their avatar (in top-left side of screen) to open the contextual menu, where I click to open the portal to them.
However, this also activates my right-click ability, which is Transplant in my case, and unexpectedly teleports me to some weird place. This is annoying, and sometimes even dangerous.
Similar problems happen with other GUIs too, with both left and right clicks, but the party member avatar is the most annoying for me.