There is a setting for USB cording output volume in the Global menu I think if I can recall. Overall I have found the B3 to be extremely loud and have had to turn down overall volume, patch volumes, and output volumes.
You can balance the input monitoring volume to the DAW output volume as well as Recording Gain from "Global" on the B3
I attached a crappy image of the device to show these settings.
I have not recorded directly via the USB output on mine, and it has been a while since I used Reaper (a great DAW btw) but you should be able to tweak your "REC GAIN" on huge B3 and hopefully view a level meter in reaper until you reach your desired level.