Use spike to reduce the risk of a story or increase the reliability of an estimate
Spike is a term that comes from the Extreme Programming (XP) practice. The goal is reducing the risk of a technical problem or increase the reliability of a user story's estimate. The duration and objective(s) of a spike should be agreed between the Product Owner (PO) and development team before the start.
When we should do it: during Sprint or between them?
During the Sprint. The development team can propose it. It is a story to be prioritized and scheduled by the PO like any other story.
Who should do it: whole Development Team or just a part of it?
One or, possibly, two developers should be enough.
How serious should be a problem to decide make a Spike, instead solve
this problem inside current tasks?
In my experience, if the risk is high or the estimate is very high based on worst case scenario, I have found a spike helps.
How much time Team should spend for Spike?
I have typically time-boxed it at 4 hours. Could go up to 8 hours in rare cases.
How we should determine time in previous question, if problem is not
"estimatable"?
Just timebox it and either build a prototype/proof-of-concept or make a recommendation based on the findings during that time.
What Team should do, if they don't investigate problem in reserved
time?
While you may not have a full solution, you now know more about it than earlier. Re-estimate the story based on this new knowledge and move on.