I work with a simple rule: if it's new and you've never done it before, you probably have no idea how long it will take, and estimates will be impossible.
However, you will be able to estimate just how little you know about it. Here's the scale I use:
5 - this is the first time anyone's done it in the world, or we've never heard of it being done.
4 - someone outside our company has done it before.
3 - someone in the company has done it before.
2 - someone in the team has done it before.
1 - almost everyone knows how to do this.
The more that's known, the more accurate the estimates will be.
I tend to divide the backlog into capabilities; for instance, the capability to trade copper, or comment on a blog post, or take a photo with the phone.
Imagine that you're Kyocera, back in the days before the cameraphone. You're the first people to create a cameraphone. So what do you focus on first? Do you worry about being able to make calls and receive calls, or do you try to see if there's a way of getting a tiny camera into the space?
Of course you do the second, because if you have too much trouble you know you have to rethink your approach, and possibly even abandon the idea. Once you've listed your capabilities, pick the ones which are newest and therefore most differentiating, and do those first. Don't even worry about estimating stuff which is a 5 on the complexity scale! That's a pure spike, and it just means that your project has a high level of innovation associated with it - congratulations!
For anything which is a 1 to 4, David Anderson suggests these buffers for uncertainty:
100% certainty (my 1): 15%
90% certainty (my 2ish): 25-30%
80% certainty (my 2ish): 50%
50-70% certainty (my 3): 100%
<50% certainty (my 4): 200%
my 5: I already told you, don't bother estimating this. You have no idea.
So if you want to have a go at estimating in real days, or you want to map your story points to time, or you already have some idea based on cycle time, then you can use these buffers to determine how much extra to add to the estimate. If you want to add buffers together, then add up all the squares and take the root of that (not all uncertainty will end up being bad).
Now you can do release planning if you want to. But honestly, if you're doing the new stuff first then you'll make the biggest discoveries up-front, and it will be readily apparent if your project is on track.
A couple of hints which will help:
If your business don't trust you, then that's the biggest risk. Forget what I said and do something easy and simple so they can see you deliver; then you'll have their trust and you can move to the risky bits.
Look out for new stakeholders who might stop you going live, new features, new bits of UI, etc.. Your goal is to get feedback on those bits ASAP. If your team can hard-code data and get feedback from that, do it. Feedback from stakeholders and users is the main reason Scrum works.
Sometimes the feedback will be "no", and that's OK because you're getting it early in the project while you still have time to change direction. Make sure your stakeholders understand that they're not seeing the finished product.
Forget Agile's rule that "it must be a whole shippable vertical slice". If you can get feedback more quickly than that, do it. That way if you throw it away you're throwing away as little investment as possible. When you get positive feedback, that's the time to turn it into a full slice. Connecting stuff to the database is not risky, for instance; devs do it all the time.
If you're going to outsource anything, make it those commoditised bits that are already pretty well understood. Don't outsource your differentiators as it will lengthen the feedback cycle and you'll lose the ability to change direction.
When you run spikes, if the developers aren't particularly disciplined then they'll produce pretty poor code. That's fine for the spike, but make them throw it away afterwards. Really good devs who know TDD and tend to produce maintainable code or are good at refactoring can stabilize the spike. Git is a great version system for managing the branches and merging you'll need.
For further reading, try Chris Matts and Gojko Adzic on Feature Injection (the precursor and similar to Gojko's Impact Mapping), Dan North's posts on the Perils of Estimation and Deliberate Discovery, and my post on Cynefin for Devs (Cynefin is pronounced "kuh-nev-in" and is another way of working out the difference between predictable stuff and things you might need to spike out).
Hope this helps and gives you some ideas to try out!