lucene 3.5.0 stack overflow error on indexing

Go To StackoverFlow.com

0

I started learning to use Lucene and first tried to compile the example for a Indexer class from a book i found. The class looks like this:

public class Indexer 
{

private IndexWriter writer;

public static void main(String[] args) throws Exception 
{
    String indexDir = "src/indexDirectory";
    String dataDir = "src/filesDirectory";

    long start = System.currentTimeMillis();
    Indexer indexer = new Indexer(indexDir);
    int numIndexer = indexer.index(dataDir);
    indexer.close();
    long end = System.currentTimeMillis();

    System.out.println("Indexarea a " + numIndexer + " fisiere a durat "
            + (end - start) + " milisecunde");
}


public Indexer(String indexDir) throws IOException
{
    Directory dir = new FSDirectory(new File(indexDir), null) {};

    writer = new IndexWriter(dir, new StandardAnalyzer(Version.LUCENE_35), true, IndexWriter.MaxFieldLength.UNLIMITED);
}


public void close() throws IOException
{
    writer.close();
}


public int index(String dataDir) throws Exception
{
    File[] files = new File(dataDir).listFiles();

    for (int i=0;i<files.length; i++)
    {
        File f = files[i];

        if (!f.isDirectory() && !f.isHidden() && f.exists() && f.canRead() && acceptFile(f))
        {
            indexFile(f);
        }
    }

    return writer.numDocs();
}


protected boolean acceptFile(File f)
{
    return f.getName().endsWith(".txt");
}


protected Document getDocument(File f) throws Exception
{
    Document doc = new Document();
    doc.add(new Field("contents", new FileReader(f)));
    doc.add(new Field("filename", f.getCanonicalPath(), Field.Store.YES, Field.Index.NOT_ANALYZED));

    return doc;
}


private void indexFile(File f) throws Exception
{
    System.out.println("Indexez " + f.getCanonicalPath());
    Document doc = getDocument(f);
    if (doc != null)
    {
        writer.addDocument(doc);
    }
}

}

When i run it, i get

Exception in thread "main" java.lang.StackOverflowError
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)
at org.apache.lucene.store.Directory.openInput(Directory.java:143)

and goes like this for tens of times.

The constructor of my class

public Indexer(String indexDir) throws IOException
{
    Directory dir = new FSDirectory(new File(indexDir), null) {};

    writer = new IndexWriter(dir, new StandardAnalyzer(Version.LUCENE_35), true, IndexWriter.MaxFieldLength.UNLIMITED);
}

has a IndexWriter call that is deprecated (because the book is written for lucene 3.0.0) and uses this method IndexWriter.MaxFieldLength.UNLIMITED(also deprecated). Could this be causing the overflow? If so, what should i use instead?

2012-04-04 08:06
by MRM
Can you paste the full stack trace please - jpountz 2012-04-04 08:14
@jpountz i edited the pos - MRM 2012-04-04 08:19


3

No, don't make your own anonymous subclass of FSDirectory! Use FSDirectory.open instead, or instantiate a concrete subclass of FSDirectory provided by Lucene, like NIOFSDirectory (but in this case you must carefully read the Javadoc on your chosen implementation, every one has OS-specific pitfalls). Lucene never, even in version 3.0, expected you to make your own subclasses of FSDirectory.

2012-04-04 11:05
by Marko Topolnik
As i said, i use version 3.5, but the example is for 3.0 (Lucene in Action, Second Edition by Michael McCandless, Erik Hatcher, and Otis Gospodnetić). When i write Directory dir = new FSDirectory(new File(indexDir), null) NetBeans tells me i have to implement all abstract classes to use this call - MRM 2012-04-04 11:31
Yes, that is beacause you either have to instantiate a specific subclass of FSDirectory (provided with Lucene, like NIOFSDirectory), or (recommended) leave that choice to Lucene by calling FSDirectory.open - Marko Topolnik 2012-04-04 11:39
In your book (an excellent book to have, by the way) read 2.10 on page 56. BTW the book never shows an example where FSDirectory is instantiated - Marko Topolnik 2012-04-04 11:52
indeed, calling FSDirectory.open(new File(indexDir), null); instead of new FSDirectory(new File(indexDir), null) {}; worked. Thank you - MRM 2012-04-04 11:58
I looked again and this example of Indexer instantiates FSDirectory(page 20, Listing 1.1). But thank you very much for help - MRM 2012-04-04 12:04
That's interesting, then you might have a pre-release draft of the book in your hands. My copy says public Indexer(String indexDir) throws IOException { Directory dir = FSDirectory.open(new File(indexDir)); ...Marko Topolnik 2012-04-04 12:18
Ads