As Marv mentions in his comments, this is a very broad question, but I will try to float a few ideas that might be helpful (given the breadth of the question, I'm certain they will not be comprehensive).
First and foremost, test as you go. Do not wait until a test period if you can in any way help it. This is really true of almost any project. The test period gets crunched and not enough time is there for testing.
Challenge your developers and testers to identify lower-level tests to validate a lot of your diverse functionality - especially at the integration test level. For example, if I want to be able to lighten or darken a region, I should be able to create a number of automated test cases to validate that the right calculations are done and changes made to the color of the pixels. I could use something like Fitnesse to create a large set of tests that thoroughly test this functionality below the UI layer.
Of course, this work has to be validated on the UI too, but if you've verified that all the calculations work under the hood, you just need to test that the data makes its way into the UI properly, requiring far fewer tests. Again, challenge your team to find automated ways to test this. For example, a team I worked with recently used image matching to make sure that given a certain starting image and putting in certain parameters, you would get the expected resulting image.
This sort of work does take time, but it also adds a lot of stability. If you put in time throughout the project to invest in these automated regression tests, a substantial amount of what you now test manually will run in a weekend in an automated test and your manual testing can focus on the extremely complex and dynamic scenarios with plenty of time to spend on it.