Understanding Search Engine Spiders
By h3riCyber
Spider will be automatically taking web pages and bring it into Search Engine, some people call it also Web Crawler, Search engine sending spider for taking document as much as possible. Work mechanism of search engine spider when doing crawling web page look like browser when downloading the different is web browser will appear texts and images but spider has not visual components and work with html based.
Crawler assign to indexed, make ranked, arranged web page in order to structure index for faster finding by internet searcher. Crawler object are files, folders, web directory and the subject from robots.txt is search engine crawler, in this case crawler will filter which are web page, file, folder can be indexed or not. Most of web page contain links to other page normally spider will start from top left to right down.
Robot.txt is text file not html this will be placed on the web site pages use for inform to search robots, you can make that file using text editor all file name must use small caps for instant:
- Robot.txt format
- User-agent contain rules that will be followed by robot
Disallow means contain folders that wish to blocked, to blocking all web place use slash “disallow:/”, “disallow:/” for blocking folders, “disallow:/file_name.html” for blocking web site files.
