Drop in replacement for Snoopy
I’ve just finished a so called drop in replacement for the snoopy library. I grabbed a RSS aggregater from the planet-php site. The aggregater was using magpierss, which in its place was using snoopy to get the RSS feeds. Unfortunately the whole snoopy library is a bit outdated (the magpierss as well). This shouldn’t be a problem, but I ran in to a few anyway.
The first problem I had was that snoopy always got a time out when fetching content from a server with keep-alive enabled. It should look at the ‘Content-Length’ header, but it doesn’t. Second was that fetching data which was send chunked, header ‘Transfer-Encoding: Chunked’, didn’t arrive well at all. The sizes are send before each chunk, which made them end up in the result. That of course, made the whole XML invalid making magpie fail.
The new snoopy library uses Curl instead of the file handlers (fopen, fgets, etc). Basically all the functionality of snoopy is already in the curl extension, so I just had to write a wrapper for it. It should be a drop in replacement, but I’ve only tested it with magpie, so there might be some bugs still.
I’m happy again. If you are sad about snoopy and want to be happy as well, you can download the lib from this site. Should there be any bugs, please let me know.
For the record… If you are not using snoopy, don’t start now and just learn how to use curl.
18 May 2007 Arnold Daniels
Thanks, that works for me !
Cool man, thank you! RomRP
Works beautifully. Thank you, Arnold! (Once I installed the curl module for php, that is).
I saw your comment regarding Snoopy on the infotech blog, and I couldn’t agree more. I started using snoopy as the base for a web spider, and it just didn’t cut it. I actually wound up switching over to the Zend HTTP packages.
http://framework.zend.com/manual/en/zend.http.html
Thanks ! I\\\’ve spent a couple of hours trying to figure out why magpierss was failing on some feeds while doing ok with others and then finally spotted the problem (the failing feeds were sent as http chunked data). Luckily found this site, replaced snoopy with your code and now it works like a charm
The HTTP extension provided by PHP does everything snoopy does and much much more. It uses Curl if compiled in with it. It’s tested extensively, why re-invent the wheel?
Marlin,
The HTTP extension is not a ‘drop in replacement’ for snoopy. You would need to do some rewriting for a project. This lib is here to replace snoopy in projects that already uses snoopy. For new projects, I do not suggest to use this lib (as stated in the last line of the article).
Yeah, I understand that you’re creating a drop-in replacement for snoopy, I guess I wasn’t explicit enough – I meant snoopy itself.
fixes for submit:
//Tell curl to use the specified request method and version
if (!empty($this->httpmethod)) curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $this->httpmethod);
if ($this->httpmethod == “POST”) curl_setopt($ch, CURLOPT_POSTFIELDS, $this->formvars);
function submit($uri, $formvars=null, $formfiles=null)
{
settype($formvars, “array”);
settype($formfiles, “array”);
$postdata = ”;
while(list($key,$val) = each($formvars)) {
if (is_array($val) || is_object($val)) {
while (list($cur_key, $cur_val) = each($val)) {
$postdata .= urlencode($key).”[]=”.urlencode($cur_val).”&”;
}
} else
$postdata .= urlencode($key).”=”.urlencode($val).”&”;
}
$this->formvars = $postdata;
//$this->formfiles = $formfiles;
$this->httpmethod = “POST”;
return $this->fetch($uri);
}
I followed the recommendation from Chris and I also only can recommend the Zend_Http compenent. It’s by far superior to Snoopy.