The thing you have to understand is that the design of the cabinet
must match the traits of the speaker to perform well.
This is absolutely necessary in bass speaker cabinets. It's not so necessary in cabinets for higher frequencies, especially for guitar. Guitar cabs often have an open back - the cabinet is literally there just to hold the speakers, nothing more.
Bass speaker cabinets are designed using the technical specifications of the speaker that will be put into them. These are called
Thiele-Small parameters, (often referred to as "T/S", explained
here.
Using these T/S specifications, along with the internal dimensions of a speaker box and the size+length of the cabinets port, you can calculate how well (or how poorly) a specific bass driver will perform in a box.
If you put a bass speaker in a cabinet that wasn't designed for it, you will not get good bass response. If the cabinet design isn't at least in the ballpark of what the speaker needs, the speaker will easily distort and even become completely blown at fairly low wattage and volume levels. A bass speaker
needs a properly designed cabinet much like a house needs a properly designed foundation.
No offense, just trying to explain, but saying you think it's "silly" that you can't just randomly put bass speakers in any box you care to try is very much like saying it's "silly" that you can't just thow any ingredient you might find in a kitchen into your soup - including drain cleaner.