plugin.includes parse-tika|protocol-http|urlfilter-suffix|scoring-opic Enable required plugins. content.server.port 55000 Port of http server serving content. fetcher.server.delay 0.2 The number of seconds the fetcher will delay between successive requests to the same server. http.agent.name test-nutch http.robots.agents test-nutch,* http.agent.name.check true http.robots.agents test-nutch,* The agent strings we'll look for in robots.txt files, comma-separated, in decreasing order of precedence. You should put the value of http.agent.name as the first agent name, and keep the default * at the end of the list. E.g.: BlurflDev,Blurfl,* io.serializations org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.JavaSerialization A list of serialization classes that can be used for obtaining serializers and deserializers.