Boutique refers to a specialized product made by a small company or shop. Custom means that a customer dictates some of the features of the product. So the two are referring to different things and not mutually exclusive in any way.
Chris Stambaugh is a one man shop that will hand build pretty much any sort of bass you want him to. Boutique and custom.
Carvin is a relatively large operation with a lot of automation who don't allow changes to the basic shape and construction of their models but do allow a wide range of options such as woods, color, hardware and your choice of their electronics. Custom but not boutique.
And something like the Fodera standard series (like the new Matt Garrison standard) is a handcrafted instrument which the user does not get to make any choices. Absolutely boutique, but not custom.
And finally, non-custom shop Fenders are built in a number of different large factories and produce basses in set configurations so that the customer may have some choice, but only of options Fender chooses to construct. Not boutique or custom.
That's a breakdown based on strict definitions. But of course, the words aren't always used in the proper sense. Often time these conversations get bogged down in minutia such as whether a builder who outsources big chunks of the build (CNC work on bodies, finishing) are really a "boutique" shop or if just giving options truly makes a bass "custom" but the words themselves have pretty specific meanings and are easy to apply.