First of all it's a good sign that the team gave you feedback about it. It means they care.
Second. As a Scrum Master your job is to make sure things required in the Scrum Guide happen:
The purpose of the Sprint Retrospective is to:
Inspect how the last Sprint went with regards to people, relationships, process, and tools;
Identify and order the major items that went well and potential improvements; and,
Create a plan for implementing improvements to the way the Scrum Team does its work.
Was that happening on first few retrospectives?
If I were you, I would make sure the team understands purpose of this meeting and ask what format they'd like to try in order to make those points fulfilled.
Dedicate a retrospective to figure out the way your team wants to do it.
As you said - you shouldn't force the team to use a specific format. Use your knowledge to guide the team and help them to come with one.
You've already showed possible variations so they have some idea around it.
Fun formats are a way to go time to time, in my opinion. The frequency needs to be suited for the team. After couple of raw discussions you'll sense that introducing some fun would be beneficial. Nevertheless, mind that people don't like changes too often. Once you pick a format for the meeting, stick to it and give it a try at least 3 times.
For the first time with new format people learn how it works. They should have an opportunity to explore it further to make most of it.
Once the team tried various of plans you can ask them (preferably before the meeting) what kind of format they'd like to go with.
I used to do 2-3 times Start-Stop-Continue activity on the retrospective and intersperse it with some new format. Lean Coffee is another, quite universal approach you can use.