Programmers: what language to use/learn for this project?

[CallMeIshmael]

Alright, I have this idea for a project I'd like to work on that involves a lot of interaction w/ websites. Basically, I will need to go to a bunch of different sites and perform some group of actions (inputing a username or other text info, browsing through a few pages, downloading info, etc).

I would like to be able to do this without actually going to the websites through a typical browswer, and actually write into the program the ability to connect with a site, scan it and input information.


Programming background: good with C/C++, and some of the more stat oriented languages (matlab and R type things). But, just about all of my C/C++ experience is related to more simulation/math type things, with a bit of win32 stuff, so I have no idea where to start doing this in C++.


What language would be good here? Im assuming its probably a choice between C++ or Java? I have no Java experience, though I imagine that w/ the C++ knowledge I could learn it in decent time, and Im in no immediate rush here. I did a tad of research, and it appears that stuff like the URL/URLConnection classes might be a step in the right direction, but I really have no idea. Obviously Im sure this *could* be done in C++, but if a lot of the leg work is already done in the java library (or some other language) it would make sense to go that way.


Obviously, if there is some other language that makes the task easy, Im more than open to suggestions. Though, the task would also require a fair amount of math, where speed is fairly important, so the language would have to be able to handle that as well.

[pokergrader]

Python or Java. And unless the math is incredibly complex, network latency is going to be the bottleneck, at which point any language will be fine.

Python is easier and more fun, Java is more like C/C++ (and faster than python), but either one will do the job.

[Meech]

Anything with a HTTP library would suffice. My current personal preference is Ruby.

http://stdlib.rubyonrails.org/libdoc...doc/index.html

[kerowo]

Might think about C# as well to leverage your C/C++ knowledge.

[psionic storm]

[ QUOTE ]
Obviously Im sure this *could* be done in C++, but if a lot of the leg work is already done in the java library (or some other language) it would make sense to go that way.

[/ QUOTE ]
theres tons of 3rd party c libaries on the net as well.

[_dave_]

[ QUOTE ]
[ QUOTE ]
Obviously Im sure this *could* be done in C++, but if a lot of the leg work is already done in the java library (or some other language) it would make sense to go that way.

[/ QUOTE ]
theres tons of 3rd party c libraries on the net as well.

[/ QUOTE ]

cURL is a Free example of such a library.

[CORed]

You might want to look at curl or wget. These are not programming languages per se, but are command-line utilities for doing mass retrieval from web sites, and I believe both have some scripting capabilities, or could be called from a program written in another language. They will handle the http headers, etc. and save you some programmin. No sense reinventing the wheel; your likely to end up making it square. If you do need to write it from scracth, perl, jave, ruby and python would all be reasonalbe choices. Java would probably be the easiest to pick up if you have prior experience in C and C++.

[BradleyT]

C# should make this pretty easy. At my last job I wrote one in VB.NET to scrape search engine results from MSN/Yahoo/Google back before they all made APIs available.


http://www.ddj.com/184405712 has a c# example.