Writing scripts to download 1000+ articles from newspaper sites.

[machine gewehr]

I know there are some computer engineers/people who know coding etc. so I'm here to ask first.

This is my last semester and I'm taking the project/thesis whatever it is called.

At first step,what I need is to collect ~1k articles from 10 different columnist.I can do that by copy pasting but you can imagine how long that'll take.Few weeks later I'll need 10k+ articles and copy pasting will be next to impossible.

For example,

http://hurarsiv.hurriyet.com.tr/yaza...lt.aspx?ID=249

Yılmaz Ozdil is one the columnists that I'll take 100 articles.I'll need to take articles from this site which is the newspaper I read,Hurriyet.

Can you give me any heads up,where to start,any tutorials you can recommend? Anything is much welcome.I have never written a script like this,I'll be searching the web too but any help from my TB bretheren.

[L-A]

You'll should try to use RSS feeds, if not to retrieve the content, at least to index it.

Then you might want to look into apps like Calibre, that can aggregate online newspapers just what you're describing, and maybe into the Readability bookmarklet, which can identify the main content of a web page pretty well.

www.calibre-ebook.com
www.readability.com