Let me tackle the two parts of your question slightly separately.
How To Prioritize
The Scrum Guide says that "The Product Owner is responsible for the Product Backlog, including its content, availability, and ordering." At the end of the day, the Product Owner has final say over which of the criteria you listed is the best way to prioritize the backlog. However, the whole team as well as stakeholders can be involved in helping the Product Owner make judgement on the complexity, ROI, risk, etc., of a PBI. In other words, engineers, designers, business analysts, business stakeholders, QA Analysts, and others can provide valuable input to help the PO make a call on how to prioritize.
Generally, I've found a few methods that help Product Owners to come to grips with prioritization:
- MoSCoW Prioritization -- categorize high-level features/epics by Must Have, Should Have, Could Have, and Won't Have for an upcoming release. Working with the development team to understand timeline and risk, the PO can determine what features must make it into the next release of the product, and then two separate levels of "Nice-to-Haves." Won't Have also helps specifically rule out less important features. I STRONGLY suggest using real corkboards with sticky notes or index cards for this exercise.
- User Story Mapping -- Explained pretty well here, if you have specific user stories written out you can prioritize them by functionality, then draw your MMF/MVP line based on what features are absolutely necessary and which aren't. This is pretty easy to do on a whiteboard or large piece of paper with sticky notes.
- The "Hundred-Dollar" Method -- The Product Owner and key stakeholders get 100 "Dollars" or "Points" to allocate to the various user stories or features on a corkboard or whiteboard. Participants can put as many of their points toward any specific feature they want. (Engineers may put a lot of points toward refactoring a legacy product, while marketing may put a lot toward a flashy new feature. The CEO may put all of his or her points toward what he or she thinks will have the biggest ROI.) The PO can then use this information to help them prioritize their backlog.
When To Prioritize
For high-level feature prioritization, I recommend having a Release Planning session minimally once per quarter. If multiple releases are planned for a quarter, each one should have its own Release Planning session. This is where a PO can use MoSCoW prioritization or other methods to help determine what will be in an upcoming release at a high level.
During development, the priority of the Product Backlog can be changed at any time by the Product Owner, taking into account the necessity of grooming/refining items before they're put into a Sprint. In other words, teams should never commit to items that they don't consider fully groomed, even if they are at the top of the backlog.
At a minimum, teams should review the near-term prioritization of items with the Product Owner at every Sprint Review and Grooming meeting to ensure that they are refining the right items for the next Sprint and working on the current highest-priority features.
I hope this helps -- feel free to let me know if there's anything I didn't cover here!