Soulfire - retry boss charges key but doesn't work

trying to do campaign skips via dungeons. I used a key and a charm to go directly to the soulfire boss, he killed me and I chose the option to Retry. It gave me the warning that I would immediately respawn and I would be charged a key. I was charged a key but I respawned in The Outcast Camp. I figured I did something wrong and didn’t think much of it. I teleported to the dungeon and again used a key and charm and again died at the boss. I selected Retry, again and I was charged a key and it respawned me in The Outcast Camp, again.

Thanks!