Getting a decent sound analog in the home studio is going to cost you a fortune. Believe me, I've tried. It can be done, but as has been mentioned, it's mighty pricey.
Digital, your best bet is in the box mixing -- meaning in the computer. Then all you need is a decent computer with lots of ram (I run a P4 1.2 ghz with 2 gb of ram), an interface (there's a bunch out there like the Presonus Firepod or Firestudio, MOTU stuff, etc.), and software (most interfaces come with some kind of software).
To record drums, guitar and vox, all at once (ensemble setting), you're going to need 4 mics on the drums minimum (kick, snare, and two overheads), at least one mic per guitar amp, and then one mic per singer. So if you have one guitar amp, with a three vocal set up, you're looking at 8 inputs required which the Firestudio can handle meaning you don't need a mixer.
If you mix in the box, then you can use software plugins which means you don't need external DSPs for effects.
This is still going to cost you a bit in equipment and cabling. Plus you have to learn the software.
What's your budget, and what type of music are you recording?
Oh yeah, and the room you have to record in will make a huge difference to the overall sound, analog or digital.
Check out
www.recordingproject.com for some good forums on recording as well as
Invalid Link Removed
Also check out this site
Invalid Link Removed for a whole pile of info on recording techniques, equipment and mic placements (uberimportant)