In JIRA Agile the column limits don't actually stop the columns from filling up because each column (in general) represents a workflow state (like In Progress or Done). The limits just alert you when there are too many issues in a certain state. This is important in Kanban, which emphasises limiting work in progress (the Atlassian intro to Kanban explains this in more detail).
It sounds like what you actually need is an additional status to be added to your workflow. For example, if you find the 'Passed QA' column is filling up with work that's actually been deployed, or moved to UAT then you could add a workflow step for that so that issues go from (e.g) Open > In Progress > Internal Testing > Passed QA > Deployed to UAT > Deployed to production. You probably have Resolved and Closed as statuses already so you could possibly use those (if they aren't already used to mean something different) or place another status in between.
Another way to handle this (as Prashant alluded to) would be to add a label or component to the issues that you want to hide (like 'ready to deploy' or similar) and then adding a quick filter. A JQL query like 'labels not in ('ready to deploy') would remove those issues that didn't have that label.
But I'd concentrate on the workflow I think. It sounds like you may have identified a process problem, not a software usage problem.