If you want to return back in Outcast Camp your character is spawned in this area:
Why?!
And so you have to walk all the way and it’s very very slow and boring, 'cause we cannot use any movement spells.
And…I don’t know why but sometimes portal its spawned in random zone, like here:
Random Portal
So I suggest to change the spawn zone for your character near your chest and the blacksmith and the portal should be also near.