Yeah, a USB interface purpose-built for connecting your instrument to your computer is definitely the way to go. These devices range from very inexpensive (29.99) to very pricey depending on the features and number of instruments you want to connect at the same time. Here's a link to some at zzounds: http://www.zzounds.com/prodsearch?pa...q=&form=search
Sort the list by price, lowest to highest and you can see how inexpensive it is to get started with hardware.
On the software side, Audacity is a good way to get started with simple recording, but IMHO, the next logical step you will want to take is writing and recording your own tunes. For that, a real digital audio workstation (DAW) tool is invaluable. Reaper is a great way to get started, free to download and try out and very inexpensive to purchase. You can find that here: http://www.reaper.fm/
Good luck and have fun!