It certainly may seem as though it will not budge, but you should be able to remove it without too much effort. Be mindful of the force you are using to ensure you do not damage the spacer during this process.
You should be able to twist it and pull up at the same time to get it loose. You could lightly apply a lubricant to it. That should allow you to remove it by hand in most cases.
If you are still unable to remove it or to get it to move, you could wrap it in a rag, and gently tap the edge upwards with peice of wood and hammer, moving around the outer diameter, lightly tapping.