quick installation : screenshots : download php-crawler
PHP-Crawler is a very simple crawl/search script with fulltext support for small websites. Simple, based on PHP and MySQL. No shell access required, crawling can be run from browser. Created ages ago (back in year 2006) it stays one of the most popular php crawler scripts in the world.
Features
- Full text indexing
- Crawling is limited by depth setting
- Safe spidering: allow to limit maximum page size
- Following “href=” links on web page, in HTML or JavaScripts
- MySQL based
- Simple installation
Requirements
- PHP 4.3.10+
- MySQL 3.23.56+
Distribution
Last version available on SourceForge under terms of BSD Licence.
sunel says:
hi your php crawler was very useful for our small project but i need help ,this works within my localhost only i need to make it work int entire web ….i look forward for u help please thank u in advance …..
vlad says:
You may want to set $CRAWL_ENTRY_POINT_URL in config file pointing out of your localhost (for 0.7.7-alpha), but please note, that PHP-crawler is not designed to crawl the entire web
Vikram says:
Your Crawler is superb man, i want to know the algorithm u hav used in ths to crawl. The algorithm used to search. N hw to use ths to crawl multiple sites at a time
Thanks in advance
Buttonator says:
Hi!
Some dirs/file are missing from the package (tpl/elt/head.php; tpl/top/table.php; tpl/bot/html.php). As I seen in config, they must be created with right path, but which is the content of them?
Johnny Wunder says:
I really like phpCrawler gives we exactly what I want in terms of a lightweight crawler I can point at whatever web site I want to analyze but I seem to be misinterpreting the use the the $CRAWL_PAGE_EXPIRE_DAYS parameter. On line 39 within function markOldURLsToCrawl of my version of _crawler.php it checks to see if the crawl time has expired and needs to be recrawled but then regardless of the results it deletes words on line 40 which causes the search to no longer work for the follow-on searches until the site is recrawled. That doesn’t seem right to me? Do I have a good version and am I interpreting it right?
Johnny
Edward says:
Small enhancement to crawler.sql script on Sourceforge:
create table phpcrawler_links () ENGINE = MYISAM;
otherwise, freetext index will fail
vlad says:
Edward, good catch, thank you!
Pingback/Trackback
Crawling | My CMS
mekix says:
me gusta mucho su crawler! gracias por crearlo, saludos desde Perú
NikoS says:
Hello , my question is:With php-crawler can index pdf or doc files?
Thanks
Roylee says:
how to index a website the path u gave to start crawl it redirects to search.php / home page quick reply is appreciated thanks !
uche umeevuruo says:
Please, the crawler does not crawl my site. Please how do I rectify this issue?
Marvin Hand says:
1. I love it and Thanks
2. You should go over these codes again.
Dharav Samani says:
Where the content of crawled web pages are stored????
Can crawler gives the flexibility to extract only the user comments from the entire webpage?
Which other parameters can we change such as CRAWL_DEPTH, $CRAWL_PAGE_EXPIRE_DAYS,etc?
Bill says:
Just found the crawler, have you ever thought about expanding the code to support multiple crawlers at the same time?
vlad says:
I would love to rewrite the crawler to support multithreading and advanced full-text search, main constraint for me is time so any contributions are appreciated.
Salim Kureshi says:
Thanks for share, workly really fine.
But I need crawler for other websites such as http://snapdeal.com
I replace $CRAWL_ENTRY_POINT_URL = “http://snapdeal.com” but there are no result so pls help me how to do it.
Thanks
Salim
Pingback/Trackback
نکست بلاگز » معرفی چند کراولر متن باز
xem phim says:
Just found the crawler, have you ever thought about expanding the code to support multiple crawlers at the same time?
istgahtablighat.com says:
شرکت فناوری اطلاعات متحد با کادر تخصصی
در زمینه طراحی و پشتیبانی سایت، در چند سالفعالیت خود در این زمینه، معیار های
اصلی مشتریان برای داشتن یک وب سایت حرفه ای را در مواردزیر دانسته است:
طراحی با کیفیت
کارایی بالا
امنیت کدها و برنامه ها
سرعت لود بالا
امکان مدیریت کامل سایت
بهینه سازی و سئو
قالبهای استاندارد برای تبلت و موبایل
کاربر پسند بودن
هزینه مناسب
تحویل بموقع
پشتیبانی 24 ساعته
گروههمگام پیشرو متحد موارد فوق را سرلوحه کار خود قرار داده و با
این نگرش، سایت هایی در زمینه های زیر با مناسب ترین قیمت و در کوتاه ترین
زمان در اختیار شما دوست عزیز
قرار می دهد.
طراحی سایت
شرکتی
فروشگاهی
شخصی
آژانس هواپیمایی
کاتالوگ
خبری و …
علاوه بر آن، در زمینه بهینه سازی سایت و سئو تجربه های فراوانی کسب نموده است و می تواند
شما را در این امر راهنمایی و پشتیبانی نماید.
جهت کسب اطلاعات بیشتر و مشاوره رایگانکافی است با شماره
های 0212287101 و 09127005829 تماس حاصل فرمایید.
Motahed Information Technology CO. in the wayy of designing
website andd support it, try to have the main criteria forr
a professional website.
the main criteria is :
- Higgh performance
- User friendly
- Security codes and programs
- High quality design
- Optimization and Seo
- Low-cost
Motahed Information Technology CO. designing website with reasonable price and
in the shortest time in the following domains:
- News
- Catalog
- Agency
- Personal
- Shopping
Furthermore, in te field of website optimization and Seo has vast experience and can support your website in this
master.
Foor more information and free consultation, Please contact us:
02122287101
09127005829
http://istgahtablighat.com/
%IstgahTablighat%
tips adsense says:
I just like the helpful info you provide on your articles.
I will bookmark your blog and check again right here regularly.
I’m relatively sure I will be informed many new stuff
right right here! Good luck for the next!
jav hd says:
Edward, good catch, thank you!
سئو says:
آموزش سئو و بهینه سازی سایت
Rashad Beaureguard says:
This is awesome. I love finding individuals who’s interests collide with my own. Id love to pick your brain and connect. In your experience, what is the best language for building web crawlers? Heres a good resource for building with Python.