Question for internet experts...

[STFU]

I would like to be able to explore a website's directory structure and content... sorta like what appears in a web browser when you access a directory without an html index or document... or a listing like you see when using FTP. I'm sure this is the way search engine "spiders" search the web...

I have run into a few occasions where I suspect the content I'm looking for is on a website, but am unable to navigate to it by the html links on the webpages.

How to do this?

 

[DigMe]

IMO the simplest way to search a website for content is to do a google website search.

For instance, if I want to find all the mentions of Phil Lesh on the CNN website I'd just type in the following:

lesh site:cnn.com

And I would get this result.

It's really handy but sometimes doesn't work for certain purposes. For instance if you did a search of "stfu site:talkbass.com" you'd get nothing...not sure why it won't work out like that but it could have to do with the forum setup, although it will show results for some forums.

brad cook

 

[grovest]

I would like to be able to explore a website's directory structure and content... sorta like what appears in a web browser when you access a directory without an html index or document... or a listing like you see when using FTP.

Yup, I'm with ya. Unfortunately, most commercial sites turn off directory browsing.

I'm sure this is the way search engine "spiders" search the web...

Don't be so sure about that. In fact I'm quite sure you're mistaken. Refer to here: http://www.googleguide.com/google_works.html

What you're describing would in essence require the search engine to 'guess' what files a directory contained. Search engines do 'assume' some files exist and look for them automatically- for instance robots.txt

However, the majority of the pages listed in search engines are found by collecting hyperlinks on other pages, which were in turn found from hyperlinks on other pages, etc. 99% of the remaining links are from people manually adding a hyperlink to a search engine's database.

I have run into a few occasions where I suspect the content I'm looking for is on a website, but am unable to navigate to it by the html links on the webpages.

How to do this?

Google. Or email the webmaster

 

[Stephen Soto]

use mozilla if your not man.

 

[xyllion]

Most sites do not allow you to browse them as a directory listing. Allowing you to do so would expose files that they might prefer to keep a secret.

 

[STFU]

Thanks all, especially Woody. I had assumed some things and you know what they say about assumptions...

 

[thinking of you]

no. no i dont. what do they say about assumptions?