Dignity nobility high quality condition highness standing course caste position superiority precedence trend distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and will instruct the robotic as to which web pages are certainly not to get crawled. Like a online search engine https://www.youtube.com/watch?v=p3IfcO3MzMs