I agree with most of your post, and also agree that many HC J pickups sound very nice in their own way. However, pickup manufacturers have been trying to nail a single coil J tone in an HC since the Hi A's came out 30 years ago, and to my knowledge and experience, none have succeeded to date.
Well Hi-A pickups were single coils, only they had a single coil for each string. Bill Bartolini was not trying to simulate a Jazz pickup. Their original motto was "For the Clearest Sound Around." So he was trying to do something different. I bought my first Hi-A pickup in 1976. I actually spoke on the phone with Bill. Things have changed since then.

They also made real single coils, but they also were intended to sound different.
You can't say none has succeeded, they have, but on the guitar, where there are a lot more of a market for pickups. A lot of advances have taken place for hum canceling single coils for Strats and Teles. First Chris Kinman came up with his noise free single coil technology, which he patented, and then those were pretty much copied by DiMarzio, with their Virtual Vintage line, and the new Duncan Classic Stack Plus STK-S4. These are all stacks, but use a different technology for the dummy coil, and they really sound exactly like single coils.
You can hear the DiMarzio Virtual Vintage pickups on the guitar in this track I recorded:
[Invalid or Expired Link Removed]
Now why these companies don't use this idea for bass pickups, I don't know.
Another recent advancement is the Suhr/Ilitch Electronics Silent Single Coil System. That works very well too.
But... there has never been as many products out for bass players, and this continues today. There's no reason it wouldn't work.
Another recent trend is the rediscovery of the sidewinder type of humbucker. Bill Lawrence had made these for years with his L-250 Strat size pickups, and now Fralin makes a P-90 for guitar. The guitarist in my old band used them, and they sound like P-90s with no hum.
Obviously I make use of this style of pickup as well. I'm redesigning my Sidewinder Js, so we'll see what I get.
It isn't surprising to me that a quite different mechanical design between the two styles of pickups would result in an overall difference in performance that would carry through among all the different executions.
There's not a whole lot to a single coil pickup. It's a very simple device. The main reason why it's tricky to get a hum canceling Jazz pickup, that sounds just like the single coil version, is not because it can't be done, it's because it's hard to do it and make it fit in the standard Jazz pickup form factor. A lot of people say my full size Sidewinders sound like a cross between a Jazz bass and a Ric. I wasn't trying to sound like either, but they do sound like single coils.
There is a limited amount of space in a Jazz shell. If you want to use two smaller coils, as in the split designs, you might not have room to do the amount of turns of wire to get the tone where you want, so you have to switch to the next gauge thinner wire. That's going to alter the tone. making the pickup taller might work, but then it won't fit into the bass right, and the change in geometry might change the tone.
But, it will sound like a single coil, just maybe a different single coil. The frequency response will be correct.
A single coil P pickup sounds quite different from the more standard HC pickup that came into use all those years ago.
You mean the split P pickup? That has nothing at all to do with hum cancelation. That's easy to see why, and was done intentionally. The split P wasn't just a matter of making a humbucker, Leo also wanted a different tone, so he made wide squat coils. That's why it sounds different. He also changed the magnet setup with the two magnets per pole. This was to reduce the attack from the string and give a more compressed tone. He patented that too. He wanted the sustaining portion of the note to be brought up and the attack down.
I used in a previous example the difference between a Fender Strat and Jazzmaster. I'm talking about a real Jazzmaster, not the reissues. The pickups on those are made wrong. The Strat pickups are similar to a Jazz bass; tall and thin. The Jazzmaster has a very wide and squat coil, just like the P bass. Thats to make it mellower sounding, which is why it's a
Jazzmaster. So even though the Strat and Jazzmaster both have real single coil pickups, they sound totally different, just as a P bass and jazz bass do. Oddly enough the Jazz bass was made to sound brighter. Go figure! But it probably made sense with the strings and amps used at the time.
A DiMarzio Model J sounds different because it was made to sound different. It has a lot of steel in it, and ceramic magnets. The idea was to make a fat sounding pickup because that;s what people wanted back then. The P bass was far more popular than the Jazz, so they had the Model P first, and made a similar sounding pickup for the Jazz a few years later.
On the other hand, some of the vintage split Js sound more like vintage single coils like with the Nords.
Given the single pickup design of the P, I understand why that happened, given the hum of a 'single' single coil. Since hum is a zero issue (if you know what you are doing) with two single coil pickups, I never quite got the motivation for even trying to do it.
Because your are stuck using both pickups at the same time. I really dislike using both pickups at the same time, and rarely do it. Im sure I'm not alone.
That is one of the reasons I don't dig the PJ configuration. Since the P is already HC, there is no way to hum cancel the J pickup, pretty much requiring you to use an HC in the bridge J position. Not for me
Use two Ps! P/J basses never worked well in my opinion. The P over shadows the J. I had two Ps in my '74 P bass and that sounded very cool. Not like a Jazz bass, but they weren't all that popular back then. I think they sound nice, but I hate that narrow neck.