Complex Search Engine marketing: Technological SEO refers to Site and server optimization that helps the crawler in crawling, indexing and ranking functions to rank your internet site greater. txt file is then parsed and will instruct the robotic regarding which internet pages aren't to be crawled. Like a online search https://www.youtube.com/watch?v=p3IfcO3MzMs