I can help with both AX8, and Helix (as well as Axe-Fx II, the flagship Fractal product). I have extensive experience (home, studio, gigs, I also host a local meetup for people with modelers, and so on) with all of the above.
Your post seems to call out usability as the main point, so I'll start there.
The Helix would "win" here if you were limited to the on-unit editing. The AX8 would "win" if you were limited to the computer (PC, Mac etc) editor. Both have parameters you can easily change on the fly directly on the unit - presets, scenes (which the Helix recently got in firmware 2 - Fractal has had these forever), tone shaping, the "blocks" (i.e amp, cab, FX devices) and so on. The Helix looks a lot nicer and IMO is a tad more intuitive when working on the hardware. When you flip to the PC editor (where I would recommend you do most of your heavy lifting for either unit, and leave "on the unit" editing for quick changes at shows if needed), Axe-Edit is truly hard to beat. It is SUPER easy to use - it starts off simple, but has pages/tabs of items that you can go as deep as you'd like with. My typical use case is roughing in a preset at home, usually with some of my band's recorded stuff. Then, at rehearsal I tidy it up fully. For that, I always take my Surface Pro (or any laptop, really) to do the edits. Not only so I can edit with the desktop application, but so I can quickly back everything up and have a copy of my final preset, just in case something catastrophic would happen. Then, at gigs, if anything at all is adjusted, it's minor EQ changes, which are super simple to do on both the AX8 and the Helix. The scribble scripts are "nice" looking, but I very rarely look down at my AX8 or Helix, and even when I do, I never have time to try to read the screens. Your use may be radically different, where the scribble scripts are important.
From the tone aspect, Fractal has the Helix beat, and it's not even close. There is just absolutely no comparison. Helix IS improving with every update, just as Fractal is. Fractal's update cadence is faster. And Fractal has quite the headstart on the Helix, too. Part of the issue is that the Helix has about one third of the DSP power that the AX8 has, so compromises will always be there. For bass guitar, the two are fairly close to one another in that they both are missing a lot of key elements. None of our really popular effects are available, for instance. Line6 or Fractal need to step up in this area, but neither seem willing to presently (it's likely a chicken/egg kind of problem. Not many bassists are into these. Hard to spend time and money developing for a market that may not buy your product. Most bass players get scared off because of this hole, though).
With that said, there is no problem dialing in wonderful bass tones on either unit. I prefer the AX8 over the Helix for bass, simply because I can dig deeper in everything and really get exactly what I want, not just be "close enough".
Where they really start to break apart is on guitar. The AX8 is just so much more refined that it's painfully obvious when you have the two side by side. Sure, I can (and have) gigged a Helix and felt pretty good about what I heard... but as soon as I directly A/B it (with my playing, my guitar, my amp) to the AX8, I strongly feel the AX8 has the Helix beat in every regard, and not by a small amount.
For what its also worth... YouTube, FaceBook and other popular video platforms SUCK for trying to get these things A/B'd properly. They compress the ever living heck out of the audio, as well as transcode it down a lot. SoundCloud is OK, but there is still obvious processing on the audio file that changes it, so it's not great for picking up subtle things.
A point of clarity as well - The Axe-Fx II and AX8 are similar, but not identical. They share the same processing algorithms, but the AX8 has slighter slower DSPs. The Axe-Fx II has two TigerSHARCs, whereas the AX8 has two ADSP-21469s. With that comes some limitations - can't run as many Amp and FX blocks in a single preset. But, sonically, they are identical.
My last hesitation would be Line6's long-term plans for Helix. How soon will it become deprecated, and then unsupported entirely? Line6 has a history of doing that. Fractal, on the other hand, bends over backwards to support their hardware. A shining example is that recent firmware updates would have caused them to no longer support one of their older rack models (one that you haven't been able to buy new for >2 years now). Their development team stepped up and worked for a couple of weeks to make it supported though, and still to this day, it is supported. They literally developed their way around a hardware limitation, just so people on the older hardware can enjoy the new firmware (new firmwares include algorithm updates as well as new amps, cabs and FX on occasion, so it's a big deal to get new updates). It's well known that eventually, that rack unit will just not be able to run the new firmware, but seeing how much effort Fractal put in to keep it going was very eye opening for me. I can't imagine Line6 going through that effort. It may be that Line6 has learned their lesson and changed their mentality about that - no one can say one way or the other yet.
Really, the best thing you can do is to have both for a couple of weeks at home and at your rehearsals. Put them to the test exactly how you'll use them. Spend some hours and learn the interfaces for both and get decent at tweaking them. Then see if one stands out for you.
