How to index wikipedia files in .xml format into solr - 【StackMirror】|xml|solr|indexing|wikipedia

I want to index xml files of Wikipedia into Solr.

But I am getting an error, it is unable to index. Solr has a specific format for xml files. I changed the schema.xml and data-config.xml files to suit the tags of the wikipedia files.

Still it is unable to index the files. My actual intention is to index wikipedia which is an xml file of 30 GB.

How would I go about indexing all wikipedia files into Solr?

2012-04-03 19:54
by Satyam Roy

I solved the same issue in this link http://stackoverflow.com/questions/20473798/indexing-wikipedia-with-solr. I hope it helps - Marcelo 2014-01-06 08:40

There's an example section in the DataImportHandler documentation for exactly this: indexing Wikipedia.

Basically, you use the DataImportHandler and some XPath to pull the metadata you care about out of the Wikipedia XML, and put it in flat Solr field listings.

2012-04-03 20:27
by Dan Fitch

I tried it but it is not working .... The file gets committed but when I search for it it does't find any fil - Satyam Roy 2012-04-03 20:30

Are you sure the file is in there? What happens when you search for *:* - Dan Fitch 2012-04-03 20:35

When I search it doesn't show any result even for : as nothing is getting indexe - Satyam Roy 2012-04-04 08:54

Okay, one more sanity check for you. Are you doing a <commit> operation after adding the file? See this page for how to do that - most Solr libraries and wrappers will make this very easy - Dan Fitch 2012-04-04 20:54