1. Please take 30 seconds to register your free account to remove most ads, post topics, make friends, earn reward points at our store, and more!  
    TalkBass.com has been uniting the low end since 1998.  Join us! :)

Question for internet experts...

Discussion in 'Off Topic [BG]' started by STFU, Jul 1, 2004.

  1. STFU


    Jun 17, 2004
    I would like to be able to explore a website's directory structure and content... sorta like what appears in a web browser when you access a directory without an html index or document... or a listing like you see when using FTP. I'm sure this is the way search engine "spiders" search the web...

    I have run into a few occasions where I suspect the content I'm looking for is on a website, but am unable to navigate to it by the html links on the webpages.

    How to do this?

  2. DigMe


    Aug 10, 2002
    Waco, TX
    IMO the simplest way to search a website for content is to do a google website search.

    For instance, if I want to find all the mentions of Phil Lesh on the CNN website I'd just type in the following:

    lesh site:cnn.com

    And I would get this result.

    It's really handy but sometimes doesn't work for certain purposes. For instance if you did a search of "stfu site:talkbass.com" you'd get nothing...not sure why it won't work out like that but it could have to do with the forum setup, although it will show results for some forums.

    brad cook
  3. grovest


    Feb 26, 2002
    Yup, I'm with ya. Unfortunately, most commercial sites turn off directory browsing.
    Don't be so sure about that. In fact I'm quite sure you're mistaken. Refer to here: http://www.googleguide.com/google_works.html

    What you're describing would in essence require the search engine to 'guess' what files a directory contained. Search engines do 'assume' some files exist and look for them automatically- for instance robots.txt

    However, the majority of the pages listed in search engines are found by collecting hyperlinks on other pages, which were in turn found from hyperlinks on other pages, etc. 99% of the remaining links are from people manually adding a hyperlink to a search engine's database.
    Google. Or email the webmaster
  4. Stephen Soto

    Stephen Soto

    Oct 12, 2003
    use mozilla if your not man.
  5. xyllion

    xyllion Commercial User

    Jan 14, 2003
    San Jose, CA, USA
    Owner, Looperlative Audio Products
    Most sites do not allow you to browse them as a directory listing. Allowing you to do so would expose files that they might prefer to keep a secret.
  6. STFU


    Jun 17, 2004
    Thanks all, especially Woody. I had assumed some things and you know what they say about assumptions...

  7. no. no i dont. what do they say about assumptions?