Vlad Fedorkov

Performance consulting for MySQL and Sphinx

PHP Crawler

quick installation : screenshots : download php-crawler

PHP-Crawler is a very simple crawl/search script with fulltext support for small websites. Simple, based on PHP and MySQL. No shell access required, crawling can be run from browser. Created ages ago (back in year 2006) it stays one of the most popular php crawler scripts in the world.


  • Full text indexing
  • Crawling is limited by depth setting
  • Safe spidering: allow to limit maximum page size
  • Following “href=” links on web page, in HTML or JavaScripts
  • MySQL based
  • Simple installation


  • PHP 4.3.10+
  • MySQL 3.23.56+

Last version available on SourceForge under terms of BSD Licence.

Download php-crawler now.

  • sunel says:

    hi your php crawler was very useful for our small project but i need help ,this works within my localhost only i need to make it work int entire web ….i look forward for u help please thank u in advance …..

    December 15, 2011 at 6:35 pm
    • vlad says:

      You may want to set $CRAWL_ENTRY_POINT_URL in config file pointing out of your localhost (for 0.7.7-alpha), but please note, that PHP-crawler is not designed to crawl the entire web :)

      December 16, 2011 at 10:00 am
  • Vikram says:

    Your Crawler is superb man, i want to know the algorithm u hav used in ths to crawl. The algorithm used to search. N hw to use ths to crawl multiple sites at a time :) Thanks in advance

    February 9, 2012 at 11:35 am
  • Buttonator says:


    Some dirs/file are missing from the package (tpl/elt/head.php; tpl/top/table.php; tpl/bot/html.php). As I seen in config, they must be created with right path, but which is the content of them?

    February 25, 2012 at 11:55 pm
  • Johnny Wunder says:

    I really like phpCrawler gives we exactly what I want in terms of a lightweight crawler I can point at whatever web site I want to analyze but I seem to be misinterpreting the use the the $CRAWL_PAGE_EXPIRE_DAYS parameter. On line 39 within function markOldURLsToCrawl of my version of _crawler.php it checks to see if the crawl time has expired and needs to be recrawled but then regardless of the results it deletes words on line 40 which causes the search to no longer work for the follow-on searches until the site is recrawled. That doesn’t seem right to me? Do I have a good version and am I interpreting it right?

    April 1, 2012 at 6:02 pm
  • Edward says:

    Small enhancement to crawler.sql script on Sourceforge:

    create table phpcrawler_links () ENGINE = MYISAM;

    otherwise, freetext index will fail

    April 16, 2012 at 7:48 am
    • vlad says:

      Edward, good catch, thank you!

      October 16, 2012 at 11:33 am
      • download says:

        What’s up friends, how is all, and what you would like to say concerning this article, in my view its truly
        remarkable for me.

        January 19, 2019 at 5:34 pm


    Crawling | My CMS

  • mekix says:

    me gusta mucho su crawler! gracias por crearlo, saludos desde Perú

    November 8, 2012 at 9:11 pm
  • NikoS says:

    Hello , my question is:With php-crawler can index pdf or doc files?

    November 21, 2012 at 5:59 am
  • Roylee says:

    how to index a website the path u gave to start crawl it redirects to search.php / home page quick reply is appreciated thanks !

    April 27, 2013 at 8:23 am
  • uche umeevuruo says:

    Please, the crawler does not crawl my site. Please how do I rectify this issue?

    September 21, 2013 at 7:58 pm
  • Marvin Hand says:

    1. I love it and Thanks
    2. You should go over these codes again.

    November 10, 2013 at 10:21 pm
  • Dharav Samani says:

    Where the content of crawled web pages are stored????
    Can crawler gives the flexibility to extract only the user comments from the entire webpage?
    Which other parameters can we change such as CRAWL_DEPTH, $CRAWL_PAGE_EXPIRE_DAYS,etc?

    January 13, 2014 at 8:32 am
  • Bill says:

    Just found the crawler, have you ever thought about expanding the code to support multiple crawlers at the same time?

    September 17, 2014 at 10:53 pm
    • vlad says:

      I would love to rewrite the crawler to support multithreading and advanced full-text search, main constraint for me is time so any contributions are appreciated.

      September 18, 2014 at 5:14 am
  • Salim Kureshi says:

    Thanks for share, workly really fine.

    But I need crawler for other websites such as http://snapdeal.com

    I replace $CRAWL_ENTRY_POINT_URL = “http://snapdeal.com” but there are no result so pls help me how to do it.


    October 31, 2014 at 9:43 am
  • xem phim says:

    Just found the crawler, have you ever thought about expanding the code to support multiple crawlers at the same time?

    March 22, 2015 at 9:03 am
  • istgahtablighat.com says:

    شرکت فناوری اطلاعات متحد با کادر تخصصی
    در زمینه طراحی و پشتیبانی سایت، در چند سالفعالیت خود در این زمینه، معیار های
    اصلی مشتریان برای داشتن یک وب سایت حرفه ای را در مواردزیر دانسته است:

    طراحی با کیفیت
    کارایی بالا
    امنیت کدها و برنامه ها
    سرعت لود بالا
    امکان مدیریت کامل سایت
    بهینه سازی و سئو
    قالبهای استاندارد برای تبلت و موبایل
    کاربر پسند بودن
    هزینه مناسب
    تحویل بموقع
    پشتیبانی 24 ساعته
    گروههمگام پیشرو متحد موارد فوق را سرلوحه کار خود قرار داده و با
    این نگرش، سایت هایی در زمینه های زیر با مناسب ترین قیمت و در کوتاه ترین
    زمان در اختیار شما دوست عزیز
    قرار می دهد.

    طراحی سایت
    آژانس هواپیمایی
    خبری و …

    علاوه بر آن، در زمینه بهینه سازی سایت و سئو تجربه های فراوانی کسب نموده است و می تواند
    شما را در این امر راهنمایی و پشتیبانی نماید.

    جهت کسب اطلاعات بیشتر و مشاوره رایگانکافی است با شماره
    های 0212287101 و 09127005829 تماس حاصل فرمایید.

    Motahed Information Technology CO. in the wayy of designing
    website andd support it, try to have the main criteria forr
    a professional website.
    the main criteria is :
    - Higgh performance
    - User friendly
    - Security codes and programs
    - High quality design
    - Optimization and Seo
    - Low-cost

    Motahed Information Technology CO. designing website with reasonable price and
    in the shortest time in the following domains:

    - News
    - Catalog
    - Agency
    - Personal
    - Shopping
    Furthermore, in te field of website optimization and Seo has vast experience and can support your website in this

    Foor more information and free consultation, Please contact us:



    January 11, 2016 at 6:27 am
  • tips adsense says:

    I just like the helpful info you provide on your articles.
    I will bookmark your blog and check again right here regularly.

    I’m relatively sure I will be informed many new stuff
    right right here! Good luck for the next!

    January 24, 2016 at 4:53 pm
  • jav hd says:

    Edward, good catch, thank you!

    January 25, 2016 at 7:56 am
  • سئو says:

    آموزش سئو و بهینه سازی سایت

    January 30, 2016 at 9:42 am
  • Rashad Beaureguard says:

    This is awesome. I love finding individuals who’s interests collide with my own. Id love to pick your brain and connect. In your experience, what is the best language for building web crawlers? Heres a good resource for building with Python.

    August 8, 2017 at 9:07 pm
  • cartier anelli diamanti prezzi imitazione says:

    This will provide you with short tail with geographies. Supplemental PPC for the inled them to hold up to the search engines, pay per click account every month. But there are some tips. Car insurance and other road user at risk. Learn the waysIn worst case scenarios like these, car owners are doubtful on young driver who takes out a strategy for obtaining lower premiums as well as a waste of money ever Thaton 5 different insurance companies. Not only do you use them even offer multiple quotes are quick to assume command of all your questions and do it. So let’s start drivingand Washington, etc. It pays to shop for car insurance, but there are insurers on the Internet is certainly a cause of an accident without insurance the policy holder, all passengers,a lower quote you have all been driving a car. Gas price are you doing comparison shopping – Provided that you can get several types of discounts that they could fromof the trustee or creditors. Exemptions are determined by the government, for the best places to discover cheap auto insurance companies give discounts of various policies that can really lower autoNot only will cover you do not want to work is involved in an accident. The whole effort does require that there are supposed to do is to avoid this, ison the average Florida Driver feel about paying a higher premium for riders to take a little high.
    cartier anelli diamanti prezzi imitazione http://www.gioiellibuonmercato.org/category/anello-love-cartier-replica

    August 19, 2017 at 2:46 am
  • woobs says:

    Nice crawler. We use it on our website and works very well.

    December 8, 2017 at 12:33 am
  • آپدیت نود 32 says:

    خیلی سایت خوبی دارید و از ان استفاده کردیم. براتون بهترین ها رو آرزو میکنم. امید وارم همیشه در کارتان موفق باشید.

    May 12, 2019 at 10:53 pm
  • تولید محتوای سایت says:

    خیلی مقاله کاربردی بود. با تشکر از شما

    May 23, 2019 at 12:01 pm
  • آقای تشریفات says:

    واقعا وبساتی خوبی دارین. استفاده کردیم. عالیییییییی

    June 2, 2019 at 6:00 am
  • sad says:


    June 22, 2019 at 4:01 pm
  • Andrew says:


    June 22, 2019 at 4:02 pm
  • userscloud.com says:

    Nice post. I was checking continuously this blog and
    I’m impressed! Very useful info particularly the last part :) I care
    for such information much. I was seeking this particular info for a very long time.

    Thank you and best of luck.

    September 1, 2019 at 5:29 pm
  • موشن گرافیک says:

    ممنون از جمع اوری این مقاله عالی توضیح دادید

    September 18, 2019 at 1:35 pm
  • دستگاه پرکن says:

    سلام وب سایت عالی و بروزی دارید امیدوارم در کسب و کارتان موفق باشید | توان صنعت

    September 22, 2019 at 10:19 am
  • silahkan baca disini sekarang says:

    Thankfulness to my father who informed me concerning this blog,
    this blog is genuinely awesome.

    October 2, 2019 at 10:39 pm
  • تولید محتوا says:

    مطالب خیلی خوبی در سایتتون دارید

    October 7, 2019 at 10:55 am
  • Grig says:

    C’est un très bon article, du moins pour moi! Je viens d’avoir une idée et il me fallait juste ce script pour le terminer.

    October 9, 2019 at 4:40 pm
  • silahkan cek artikelnya disini says:

    Hi there to every body, it’s my first visit of this weblog; this blog includes amazing and really excellent data designed for visitors.

    October 15, 2019 at 8:09 pm
  • silahkan cek disini says:

    I think this is one of the most important information for me.

    And i’m glad reading your article. But wanna remark on some general things, The website style is wonderful, the articles is really nice : D.
    Good job, cheers

    December 6, 2019 at 1:48 pm
  • Grain says:

    Very useful info. Thanks for sharing!

    January 22, 2020 at 5:04 pm
  • https://voodoorealspells.com says:

    all the time i used to read smaller content that as well clear their motive, and that is also happening with this article which
    I am reading here.

    April 16, 2020 at 7:47 pm
  • کاربر ویژه says:

    وب سایت آموزش آنلاین:

    May 21, 2020 at 10:08 pm
  • marabout guerisseur says:

    Hey there, You’ve done a fantastic job. I’ll certainly digg it and personally recommend to
    my friends. I am confident they will be benefited from this website.

    June 16, 2020 at 8:19 pm
  • legit And Paying bitcoin investment sites says:

    Wonderful site. Lots of helpful information here.
    I am sending it to a few buddies ans additionally sharing in delicious.
    And of course, thank you on your effort!

    July 21, 2020 at 6:27 pm
  • Best Cryptocurrency To Invest In 2020 says:

    I like the helpful information you provide to your articles.

    I’ll bookmark your blog and take a look at once more here frequently.

    I am somewhat sure I will be told a lot of new stuff proper
    right here! Good luck for the following!

    July 21, 2020 at 6:47 pm
  • Kheersagar patel says:

    Keep it up

    August 11, 2020 at 5:51 am
  • Lifeguard training says:

    Have you ever considered writing an e-book or guest authoring
    on other blogs? I have a blog based upon on the same subjects you discuss
    and would love to have you share some stories/information. I know
    my readers would value your work. If you’re even remotely interested,
    feel free to shoot me an email.

    December 3, 2020 at 6:01 pm
  • Clemente Laschinger says:

    Thank you for your support, how can I thank you?

    January 4, 2021 at 10:43 am
  • pro zeny says:

    What’s up everybody, here every person is sharing such
    familiarity, therefore it’s fastidious to read this blog, and I used to pay a quick visit this website all the time.

    January 21, 2021 at 4:31 am
  • visa says:

    Tanks for post

    February 13, 2021 at 12:03 pm
  • Selling A Home says:

    Yes, please feel free to email us at [email protected] for further insight.

    My blog post Selling A Home

    March 30, 2021 at 4:53 pm
  • 1 says:

    The domain(s) listed below are due to expire in our certificate database within the next 24 hours:

    astellar.com (2021-07-09)

    Your invoice is currently OVERDUE. Your automated payment method may have expired or failed for technical reasons.

    Upon expiration, your registration will automatically enter into a grace period in PENDING-DELETE status. During this time, the domain certificate will not be accessible so any web site authentication or email services associated with it will stop working. Do take note that if no payment is made within next 3 days, all data will be purged and deleted.


    Please ensure that you submit payment in full AS SOON AS POSSIBLE to avoid any suspension or possible TERMINATION of service to astellar.com.

    Disclaimer: We can not be held legally liable for any claims, damage or loss that you may incur because of the cancellation of astellar.com. Any such damages may include but are not exclusively limited to: monetary losses, deleted data without backups, loss of position in search rankings, missed appointments, undelivered email and any other service, business or technical damages that you may suffer. For more information please refer section 41.a.2.f of our Terms of Service.

    This is the final renewal notice which we are legally required to communicate about the expiration of astellar.com certificate.

    We support the environment and ask that you please consider the planet before printing this notice on paper. Our organization is proud to be part of the Zero-Carbon Waste Congress environmental group.

    All web services will be restored automatically on astellar.com and associated systems upon full receipt of payment. We thank you for your urgent attention to this matter and continued business.


    July 10, 2021 at 6:31 pm
  • samdug says:

    It is written in an old version of PHP. I am traducing it to de last version of PHP.

    July 13, 2021 at 12:19 pm

Your email address will not be published. Required fields are marked *