The Budget Holder is the individual who is ultimately accountable for ensuring that the budget is followed. Budget holders are usually the company's managers and operational directors, who are tasked by the company's owners/shareholders or board of directors with ensuring that the budget is followed. From your question, this can technically also be a project manager.
From experience, you can be granted access to an account and you can also receive funds for the budget into an account of your choice depending on the organization but whichever is meant to serve a goal which is to ensure that all financial transactions involving the company/project comply with existing financial rules and regulations governing them. The Budget Holder also decides how much of the company’s budget is allocated to each department or division.