so I've been getting this error "multiple_values_encountered_for_non_multiValued_field_title" every once in a while when I am trying to run solrindex. I can now say that this is being caused by index-more plug in (MoreIndexingFilter.java)

private NutchDocument resetTitle(NutchDocument doc, ParseData data, String url) {
String contentDisposition =
data.getMeta(Metadata.CONTENT_DISPOSITION);
if (contentDisposition == null)
return doc;

for (int i=0; i<patterns.length; i++) {
Matcher matcher = patterns[i].matcher(contentDisposition);
if (matcher.find()) {
doc.add("title", matcher.group(1));
break;
}
}
return doc;
}


the problem here is that in my case this function is not reseting but it is just adding a new title. it seems that the original idea was that if CONTENT_DISPOSITION exist then the document will not have a title set from other plug ins (namely index-basic). unfortunately this seems not to be always the case as you can see by running this command:

bin/nutch indexchecker http://www.2modern.com/site/gift-registry.html

what i do get (the part that is relevant) is:


tstamp : Tue Feb 21 13:18:13 PST 2012
type : text/html
type : text
type : html
date : Tue Feb 21 13:18:13 PST 2012
url : http://www.2modern.com/site/gift-registry.html
content : 2Modern Gift Registry Modern Furniture & Lighting items in cart 0 checkout Returning 2Modern cu
user_ranking : 25.0
title : 2Modern Gift Registry
title : gift-registry.html
plutoz_ranking : 10.0
categories : Furniture Home
contentLength : 12924

and as you can see there are 2 titles. I think it would be very easy to fix that. just check to see if a title exist already before setting the name of the file as title:

if (contentDisposition == null || null != doc.getField("title"))
return doc;


or if the substitution must happen in presence of CONTENT_DISPOSITION, at least remove the old one:

if (matcher.find()) {
doc.remove("title");
doc.add("title", matcher.group(1));
break;
}


now that being said, the real problem here is why NutchDocument doesn't observe the schema.xml file and alway assumes that all fields are multi value?

public void add(String name, Object value) {
53 NutchField field = fields.get(name);
54 if (field == null) {
55 field = new NutchField(value);
56 fields.put(name, field);
57 } else {
58 ----> field.add(value); <---
59 }
60 }

--
Kaveh Minooie

www.plutoz.com