When doing Scrum, I use story points based on the total number of points completed in a sprint. When doing Kanban, I am not sure how to work out capacity.
What is the best way to do this?
You have several options. None of them are exclusive to Kanban though. They all work perfectly fine with Scrum too.
Remember, Kanban isn’t a project management methodology. It’s a continuous improvement methodology. Change things as needed, but do so empirically. Run small experiments and measure the outcome. I’d start by continuing to use story points while simultaneously collecting the other measurements. This allows you to compare the two and be confident in your new metrics.
In Kanban, you generally aren't measuring "team capacity" because kanbans track queues, not people. The closest you are likely to come to a direct capacity measurement in Kanban in the WIP limit, which sets a capacity limit on each queue within the system. While similar, this isn't interchangeable with the notion of team capacity as used by time boxing systems like Scrum.
That said, you can use Kanban-native measurements to approximate iterative capacity measurements. However, doing so is probably an anti-pattern, as Kanban optimizes for flow rather than fixed-capacity scheduling.
In systems like Scrum, where you're working in iterations, measuring capacity makes sense. However, when you're working with flow systems like Kanban, you generally measure how long things are enqueued or how long they take to process. Typical Kanban measurements include:
While all of these metrics have some bearing on how much "capacity" a team has and provide the ability forecast the run rate of a project, they are not direct measurements of capacity. Instead, they are generally used to forecast the time to complete similarly-sized batch items, or to estimate the total runtime to empty a backlog queue of a given size.
You can certainly extrapolate capacity from Kanban-type metrics by measuring average cycles within a sliding time window. This is similar to velocity, but it's important to understand that Kanban is a flow system, not a time boxing system.