TL;DR
Spikes are estimated in story points. Tasks related to the spike should be estimated in time units. The time-box for a spike is calculated based on the associated tasks defined on the Sprint Backlog.
What Spikes Are For
[H]ow do you timebox spikes using story points? [They are a] mix of effort and task complexity.
You left out "uncertainty." Story points are also a measure of uncertainty. A user story where no one even knows where to start will approach infinite uncertainty, whereas a well-written story with good boundaries will limit the uncertainty.
For example, "research the number of angels that can dance on the head of a pin" will have a high level of uncertainty, unless you happen to already have a pin that angels are known to use as a nightclub. In contrast, something like:
As a programmer,
I want to explore alternative ways to render the home page
so that it loads in under 1,200 milliseconds.
has some uncertainty about the solution, but it also has well-articulated goals, clearly-defined boundaries of "good enough," and is likely to suggest uncertainty-reducing approaches that can at least be sized for effort.
Unlike a normal user story, a spike isn't always expected to generate a potentially-shippable increment. A spike is a success if it adds knowledge to the team and reduces uncertainty about a related story. For example, the spike above might result in learning that a team can't reduce load times without sacrificing content or re-designing the architecture. That's a win if it helps the team estimate other UI or UX stories. Even if the spike is incomplete at the end of the time-box, it reduces uncertainty about the level of effort required to meet related goals.
Stories vs. Tasks
Stories are estimated in story points. Tasks should be estimated in time units, since they are essentially an allocation of the enclosing time-box. There are no hard rules about this, but in my Scrum practice:
Stories on the Product Backlog are estimated in Story Points. While there are exceptions, most spikes belong on the Product Backlog where they are visible to the Product Owner and the organization. Research has an associated cost that must be borne by the project, and it is the Product Owner's responsibility to prioritize research via the Product Backlog.
Tasks on the Sprint Backlog are estimated in time units. Part of Sprint Planning is decomposing user stories into granular tasks on the Sprint Backlog. I generally recommend sizing tasks between a half-day and two days; anything smaller creates too much process overhead, and anything larger violates the underlying "done/not-done" principle.
Time-Boxing Spikes
Spikes, by definition, have a maximum time-box size of one sprint. That's your upper bound. At the end of a sprint, the spike is done or not-done, just like any other story.
Spikes also have an implicit minimum, which is the smallest size of a task on the Sprint Backlog. This represents a spike's minimum cost to the project, but can also be used to carve out a time-box within a sprint for the spike.
A spike has a story-point estimate. Each of the exploratory tasks for a spike should have a time estimate on the Sprint Backlog. You handle spike slippage the same way you handle user story slippage: fail early. While tasks don't require the same rigid time-boxing as the sprint as a whole, tasks that exceed their Sprint Backlog estimates should be communicated in the daily stand-up, and the whole team (including the Product Owner) should decide what to do about spikes that won't deliver value or that can't be completed within their time-box.
A Concrete Example
As an example, a two-point story might have five tasks with a time estimate of one day each. The story therefore has a time-box of five days, and each task has a time-box of one day. If a task exceeds its one-day time-box, it is reported in the daily stand-up where the team can coordinate and decide what to do.
Perhaps the team chooses to allocate more resources to the task, shifting time or people from some other story that turned out to be easier than expected. Then again, maybe the team decides that enough information has been gathered to meet the uncertainty-reducing goal of the spike. On the other other hand, the team might also decide that the potential gains from the spike aren't worth putting other stories at risk of being not-done at the end of the sprint, and therefore kill the spike early.
As you can see, this is really the same decision tree that the team has with regular stories. The primary difference is that a spike is usually designed to feed into the estimates of a related story in a future sprint, rather than deliver something concrete. As such, even a "failed" spike is usually a success in that it reduces uncertainty or highlights complexity of a related story.
Self-Organizing Teams
As a final thought, remember that successful Scrum teams are self-organizing. As a Scrum master, your job is to remind people to respect the time-boxes that they allocate. However, within the limits of the framework, it is perfectly acceptable for the team to allocate or shift internal resources within a Sprint.
In plain English, that means that the team is responsible for meeting the Sprint Goal. If they miss goals because they fail to respect time-boxing, this will be clearly reflected in the burn-down chart and in the Product Backlog, and should be addressed in the Sprint Retrospective.
Time-boxing is a tool; it's not an end in itself. The framework requires that the time-boxes for iterations and defined meetings be respected. However, within a Sprint, time-boxing is intended to be a decision-making tool, not a line in the sand which no one can cross.
Bad teams ignore time-boxes. Good teams respect the limitations imposed by the time-boxes. Great teams know when they can stretch a self-imposed time-box without compromising the Sprint Goal.
