created by sjw/mfgis 2feb07

Installing and Running Nutch Under Debian 'Etch'

Install Sun's Java

Sun Java is available as a set of Debian packages and may be easily installed using apt. To obtain Sun's Java, ensure that 'non-free' is included in /etc/apt/sources.list

• # apt-get install sun-java5-bin sun-java5-demo sun-java5-jdk sun-java5-jre

Since there may be more than one flavor of Java on the system (e.g. kaffe) ensure that Sun Java is the chosen alternative

• # update-alternatives --config java // then select sun java from the menu

If necessary edit /etc/profile to include the following lines:

• JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun-1.5.0.10
  export JAVA_HOME

Install Tomcat5.5 and Verify that it is functioning

• # apt-get install tomcat5.5 libtomcat5.5-java tomcat5.5-admin tomcat5.5-webapps

Verify Tomcat is running:

• # /etc/init.d/tomcat5.5 status
  #Tomcat servlet engine is running with Java pid /var/lib/tomcat5.5/temp/tomcat5.5.pid

Tomcat may be started and/or stopped using the following:

• # /etc/init.d/tomcat5.5 start
  # /etc/init.d/tomcat5.5 stop

It is NOT necessary to run '~/local/tomcat/bin/catalina.sh start' as noted elsewhere in the WIKI, nor is it necessary to start tomcat/catalina from any particular location
Tomcat5.5 under Debian Etch listens to port 8180, not 8080, so pointing your browser to http://blahblah:8180 will bring up the Tomcat home page, if everything is functioning properly.

Grant Yourself Tomcat Manager Permissions

Edit /usr/share/tomcat5.5/conf/tomcat-users.xml and include the following:

Enter the Tomcat Manager

Tomcat5.5 under Debian Etch comes pre-installed with a handfull of simple webapps. Clicking on the Tomcat Manager link from the Tomcat home page will show you a list of these applications and their execution status. Later we will return to this page to verify that our nutch applications are running.

Acquire, install and configure Nutch

Acquire a copy of nutch and unpack it in a new directory location. I suggest using /usr/local/nutch as the top-level directory, but this is of course optional

Configure for multiple, independent site crawls and searches

Follow the section Intranet:Configuration from the Nutch tutorial at http://lucene.apache.org/nutch/tutorial8.html. However, plan in advance for crawling and searching sites independently from one another:
Given two sites, site1 and site2 which you wish to crawl/index (and later search) independently from each other, you may make multiple copies of the conf directory:

• #cd /usr/local/nutch
  #cp -rp conf conf.site1
  #cp -rp conf conf.site2

And then work through steps one through four of the above mentioned section for each site.

Create simple shell scripts which allow for the independent crawling of each site, such as /usr/local/nutch/crawl_site1.sh

• NUTCH_CONF_DIR=conf.site1
  export NUTCH_CONF_DIR
  bin/nutch crawl urls/site1 -dir crawls/site1 -depth 10 -topN 100000

and the same for site2.

Then proceed to crawl each site:

• #sh crawl_site1.sh
  #sh crawl_site2.sh

Configure Tomcat's File and Webapp Paths

Under Debian Etch, the Catalina configuration files are located under /etc/tomcat5.5/policy.d At runtime they are combined into a single file,/usr/share/tomcat5.5/conf/catalina.policy Do not edit the latter, as it will be overwrittten.
At the end of /etc/tomcat5.5/policy.d/04webapps.policy include the following code:

{{{grant codeBase "file:/usr/share/tomcat5.5-webapps/-\" {

• permission java.util.PropertyPermission "user.dir", "read"; permission java.util.PropertyPermission "java.io.tmpdir", "read,write"; permission java.util.PropertyPermission "org.apache.*", "read,execute"; permission java.io.FilePermission "/usr/local/nutch/crawls/-" , "read"; permission java.io.FilePermission "/var/lib/tomcat5.5/temp", "read"; permission java.io.FilePermission "/var/lib/tomcat5.5/temp/-", "read,write,execute,delete"; permission java.lang.RuntimePermission "createClassLoader", ""; permission java.security.AllPermission; };}}}

Warning: The last line here was necessary in order to make things work for me. If anybody can supply a more restrictive permission set, please do so!!! The effects of this are unknown

Install Multiple Copies of Nutch under Tomcat5.5 and Prepare for Searching

Under Debian Etch & Tomcat5.5 the webapps path is located at

• /usr/share/tomcat5.5-webapps

Contrary to the Nutch tutorial(s) it is NOT NECESSARY to remove the ROOT context nor is it desirable. It was noted above that the Tomcat Manager allows us to view and control our multiple applications. Removing ROOT would break this functionality.
Create two new folders under /usr/share/tomcat5.5-webapps, and explode the nutch war file into each: {{{ #cd /usr/share/tomcat5.5-webapps #mkdir site1#mkdir site2 #cp /usr/local/nutch/nutch-0.8.1.war site1 #cp /usr/local/nutch/nutch-0.8.1.war site2 #cd site1; jar xvf nutch-0.8.1.war; rm nutch-0.8.1.war; cd .. #cd site2; jar xvf nutch-0.8.1.war; rm nutch-0.8.1.war; cd .. }}}

Configure the site1,site2 webapps

Edit the site1/WEB-INF/classes/nutch-default.xml file for the searcher.dir parameter, so that it points back to your crawl directory under /usr/local/nutch and save it as nutch-site.xml after making the following changes:
{{{searcher.dir /usr/local/nutch/crawls/site1 }}} And repeat for site2.
Create site1.xml and site2.xml under /usr/share/tomcat5.5-webapps by modifying the distribution nutch-site.xml

And repeat for site2.
Create symbolic links to these files under /usr/share/tomcat5.5/conf/Catalina/localhost

ln -s /usr/share/tomcat5.5-webapps/site1.xml /usr/share/tomcat5.5/conf/Catalina/localhost/site1.xml
ln -s /usr/share/tomcat5.5-webapps/site2.xml /usr/share/tomcat5.5/conf/Catalina/localhost/site2.xml

Restart Tomcat

 /etc/init.d/tomcat5.5 restart  Revisit the Tomcat Manager. You should see new entries for site1 and site2 and with luck their Running status should show asTrue

Search Your Sites!

Point your browser to http://blahblah:8180/site1 and conduct a search. 
Point your browser to http://blahblah:8180/site2 and conduct another search. 
If everything was configured properly you should see independent results representing independent searches on independent crawls.

FIN.

 Today I started to work on a little project that required a crawler, and Nutch seemed to do most of what I needed. The nutch team conveniently released Nutch 1.0 late in March 2009, so I had a brand new release to test out. Installing nutch 1.0 on a mac is not as straight forward as I thought, I ran into a lot of unexpected issues and here is my cook book description of how to successfully install nutch 1.0 on your mac.

1. Download the latest source code from the Apache SVN repositoryhttp://svn.apache.org/repos/asf/lucene/nutch/. I tried running it from the tarball without success, I also tried to compile the source from the tarball, but a post on the nutch forum clearly states that this will not work.
2. Set your JAVA_HOME and NUTCH_JAVA_HOME variables, again this is not straight forward, they both need to point to your real installation of Java 1.6 (earlier versions of Java will fail). I sat these variables to: /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home, I could not get the /Library/Java/Home symbolic link to work properly.
3. Compile the source code using Ant (I built it in Eclipse).
4. Setup your nutch configuration, by following the tutorial by Peter P. Wang
5. Run your first crawl with: ./bin/nutch crawl urls -dir crawl -depth 3 -topN 50

Most of the issues I encountered was related to the Java version and the fact that using/Application/Utilities/Java/Java preferences application do not really change the JAVA_HOMEdirectory /Library/Java/Home properly. So make sure you have set both JAVA_HOME andNUTCH_JAVA_HOME, and that your OSX does not fool you when it pretend to be symbolically linking to the 1.6 installation.

 This howto will explain how to get Nutch, Nutch-Gui, Sun JDK & Tomcat 6.0.16 working on Centos 5.x while maintaining a normally functioning Centos system. Currently, Centos 5.x ships with Tomcat 5.5, however, while it does run, there are problems with the default install of this version that results in errors which are undocumented and persistent at this time. If you have information or believe that these errors have been addressed and can point to a fix, please use the contact form on this website to let us know. The following instructions allow for easy removal of any software installed through following this howto by either using "rpm -e foo.rpm" or "rm -rf /opt/foo" returning your system to its original state.

Requirements

Getting Started

bin/nutch

Intranet Crawling

bin/nutch crawl urls -dir crawl.test -depth 3 >& crawl.log

Whole-web Crawling

mkdir db
mkdir segments

bin/nutch admin db -create

wget http://rdf.dmoz.org/rdf/content.rdf.u8.gz
gunzip content.rdf.u8.gz

bin/nutch inject db -dmozfile content.rdf.u8 -subset 3000

bin/nutch generate db segments

s1=`ls -d segments/2* | tail -1`
echo $s1

bin/nutch fetch $s1

bin/nutch updatedb db $s1

bin/nutch generate db segments -topN 1000
s2=`ls -d segments/2* | tail -1`
echo $s2

bin/nutch fetch $s2
bin/nutch updatedb db $s2

bin/nutch generate db segments -topN 1000
s3=`ls -d segments/2* | tail -1`
echo $s3

bin/nutch fetch $s3
bin/nutch updatedb db $s3

bin/nutch index $s1
bin/nutch index $s2
bin/nutch index $s3

bin/nutch dedup segments dedup.tmp

rm -rf ~/local/tomcat/webapps/ROOT*
cp nutch*.war ~/local/tomcat/webapps/ROOT.war

~/local/tomcat/bin/catalina.sh start

 This document contains instructions for downloading and installing Nutch and Lucene. Please beware that you must be logged into the csci571 computer to run Apache Tomcat and not on aludra or nunki.

1. Downloading and Installing Nutch
2. Downloading and Installing Lucene

Downloading and Installing Nutch
Chris A. Mattmann
mattmann@apache.org

Pre-requisites

1. Installation of Java 1.4 or above. You can download java from http://java.sun.com
   

2. Installation of Apache ANT 1.6 or above. You can download ANT from http://ant.apache.org 
3. Installation of Apache Tomcat 5.5.19 or above. You can download Tomcat from http://tomcat.apache.org
   
4. If you are using Windows OS, please install Cygwin: you can find Cygwin here: http://www.cygwin.com/ 
5. Install the subversion client, You can find Subversion at: http://subversion.tigris.org

Installation Instructions

1. Download Nutch from SVN, using the Subversion command line client:
   
   # svn co http://svn.apache.org/repos/asf/lucene/nutch/tags/release-0.8.1/ ./nutch

   1. This will install nutch into a directory called “nutch” local to wherever you ran this command. So, if you ran this command from /home/bogus, then you would have a directory called /home/bogus/nutch 
   2. We’ll call the directory where you unpacked Nutch to your $NUTCH_HOME 
2. Cd into the Nutch directory, and compile Nutch:
   
   # cd nutch
   # ant 
   
   1. You should see a message such as the following if all is well and the build ran successfully
      
      compile:
      
      job:
      [jar] Building jar: /Users/mattmann/tmp/nutch/build/nutch-0.8.1.job
      
      BUILD SUCCESSFUL
      
      Total time: 27 second

3. Okay, now that Nutch is built, you can fetch some content. There is a detailed, step-by-step set of instructions on the wiki, for how to fetch content. This page provides all the details: http://wiki.apache.org/nutch/NutchTutorial
   

4. Once you’ve fetched some content, you’ll probably want to browse it. To get Nutch set up on Tomcat, first build the Nutch webapp (run the below command from your nutch directory):
   
   # ant war 
5. The above command will construct a nutch-0.8.1.war file within $NUTCH_HOME/build. It will also construct a nutch.xml file within $NUTCH_HOME/build. The nutch.xml is a Tomcat context.xml file, that you can use to configure a WAR file for deployment within Tomcat. 
6. First, make a directory for your nutch war file, and your nutch context.xml file to live in. /usr/local/nutch is a good place.
   
   # mkdir /usr/local/nutch
   # cp –R $NUTCH_HOME/build/nutch-0.8.1.war /usr/local/nutch
   # cp –R $NUTCH_HOME/build/nutch.xml /usr/local/nutch 
7. Next, edit your /usr/local/nutch/nutch.xml file
   
   Inside the file, modify the property searcher.dir to the path where your Nutch index that you created separately (in step 3 above) exists. If that directory is /home/bogus/nutch/my.crawl, then you would set searcher.dir to /home/bogus/nutch/my.crawl. 
8. Edit your /usr/local/nutch/nutch.xml file again
   
   Edit the docBase attribute on the Context tag to be the FULL path to your Nutch 0.8.1 WAR file, e.g., /usr/local/nutch/nutch-0.8.1.war 
9. Now, assuming that you have installed Tomcat according to the pre-requisites, and assuming that you have set the $TOMCAT_HOME environment variable (that points to your Tomcat installation directory), first shutdown tomcat:
   
   # cd $TOMCAT_HOME/bin
   # ./shutdown.sh
   
   Now, symlink your context.xml file for Nutch to the Tomcat conf directory
   
   # ln -s /usr/local/nutch/nutch.xml $TOMCAT_HOME/conf/Catalina/localhost/nutch.xml
   
   Now, restart your Tomcat server:
   
   # cd $TOMCAT_HOME/bin
   # ./startup.sh 
10. If everything went right in step 9, then you should open up your browser, and point it at your tomcat installation (e.g., http://localhost:8080), and then append the path “/nutch” at the end of it. So, if you installed tomcat to run on port 8080, then you would visit: http://localhost:8080/nutch/

That’s it!

If you have any further questions, please feel free to contact me at the email address provided above.