• php

    Posted on November 3rd, 2008

    Written by Jose (Jossi) Fresco Benaim

    Tags

    ,

    Sphider and .htaccess protection

    Sphider and .htaccess protection

    Sphider, the open source PHP spider (aka Web crawler) and search engine, uses the fsockopen() function to get files that are spidered. This means that if the site you are spidering is protected via .htaccess or the Apache directive to protect realms, Sphider will return a “401 unreachable” error when attempting to fetch files during [...]

  • php

    Posted on October 28th, 2008

    Written by Jose (Jossi) Fresco Benaim

    Tags

    , ,

    Sphider mod

    Sphider mod

    Off-the-shelf, Sphider the open source php crawler and search engine, (http://www.sphider.eu/) only returns one portion of a document in which the searched keyword is found. With this mod, Sphider will display as many as 5 portions of a document instead…