Shouldn’t it? 4 companions using 4 skill specializations as compared to 1 skill specialization? The idea would normally be that whatever 3 skills you use in place of those other 3 would make up the damage difference (hopefully adding yet more damage besides), at least that would be my expectation. What were you filling those slots with when using the squirrels?