When using +skill level items they sometimes mix the allocated points with permanent points.
Example: Put on +2 relic, allocate 2 points to skill A. Remove Relic, points are removed from skill B which were put on there from level ups. Skill A has still 2 points.
Did this quite a few times before I noticed, need to level multiple skills from 15 to fix things up.