It's all challenging, but in different ways. Once you go beyond a typical 4 or 5 piece band, to a more ensemble style, everything has to be carefully written and put together, regardless of the instruments involved. Otherwise it's a mess.
Not that 4 or 5 can't make a mess, if they're awful, but it's generally less of a concern, if they're reasonably good players/writers.
A trio, on the other hand, opens up more possibilities for each individual to explore their instrument and find creative ways to fill up their space. Do that well, and a trio can really kill it. Do it badly, and it can come off as thin and wimpy.
I do often favor the trio format, myself.