The Creative Labs Nomad Jukebox 3 is what I have been using for field recording for a year or two. I generally take it with a [Invalid or Expired Link Removed] and a pair of mics to record shows I play at, and with the associated power adapters and cables it all fits into a small laptop bag.
The pros: It will record as WAV (either DAT or CD quality) or you can record in a couple different levels of mp3. Mine has a 20G hard drive but you can get ones with more space. It's got both an analog and an optical digital line in, which is really really nice if you happen to do any studio work and want a digital copy of what went on. You can hook it to your computer via firewire or USB and drag and drop recordings, just like you're talking about. It just names the recording based on the time and date. It's the exact size and shape of a Discman, so there are a lot of options for carrying cases. The best part: It's been around for a while. This means you can get them used at a great price.
The cons: You need software to transfer files to your computer. It's not smart enough to just act as a firewire drive. Creative has it for free download or there is third party software available.