Hi all,
Inspired by another thread here (Question about CustomScoreQuery) I am using
this solution which is working really well (with one drawback):
I discovered that some of my problems were due to the fact that my assumption
was wrong:
I did have many fields/queries terms with the same field ID.
This ruined my approach because the query boost was aggregated and my
calculations were wrong.

What I did was during indexing I added the field value to the field id
(concatenated it by '_') and as filed value used the desired score.

At search time I am using simple FieldScoreQuery (As-is, no modifications
needed) with the complex field ID.
Here I can still use the setBoost to set the score because now my filed are
unique.

Logic wise this is perfect - dot product using Lucene.

Drawback - Lots of lots of different types of fields - effects the memory usage
dramatically.

If anyone has better ideas - please share!

-----Original Message-----
From: Alan Woodward [mailto:alan.woodward@xxxxxxxxxxxxxxxxxxxx]
Sent: Wednesday, February 22, 2012 4:00 PM
To: java-user@xxxxxxxxxxxxxxxxx
Subject: Re: Custom lucene scoring - Dot product between field boost and query
boost

Hi Yuval,

You can just override Similarity, rather than DefaultSimilarity - that way you
don't burn any CPU cycles on TF/IDF calculations.

Alan

On 22 Feb 2012, at 07:17, Yuval Kesten wrote:

> Hi Em,
> 1. Regarding the performances - the similarity class (And my subtype as well)
> gets the IDF and TF and SQUARED SUMS calculations as inputs - they just
> factor them differently. Even though I ignore the values they are being
> computed.
> 2. I have written this code:
> static {
> Similarity.setDefault(new MySimilarity());
> }
> Which means that I am setting the default similarity before doing the
> indexing and obviously before the searching.
> Thanks!
>
> -----Original Message-----
> From: Em [mailto:mailformailinglists@xxxxxxxx]
> Sent: Tuesday, February 21, 2012 6:07 PM
> To: java-user@xxxxxxxxxxxxxxxxx
> Subject: Re: Custom lucene scoring - Dot product between field boost
> and query boost
>
> Hi Yuval,
>
>> 1. Performances: I am calculating all the TF/IDF stuff and NORMS for
>> nothing...
> You aren't calculating that much, since you declared all those values as
> constants. What are you worried about?
>
>> 2. The score I get from the TopScoreDocCollector is not the same as I
> get from the Explanation.
>> Here is part of my code:
> Could you provide us the code where you are setting the Similarity, please?
>
> Kind regards,
> Em
>
> Am 21.02.2012 16:18, schrieb Yuval Kesten:
>> Hi,
>> I want to use Lucene with the following scoring logic:
>> When I index my documents I want to set for each field a score/weight.
>> When I query my index I want to set for each query term a score/weight.
>>
>> I will NEVER index or query with many instances of the same field - In each
>> query (document) there will be 0-1 instances with the same field name.
>> My fields/query term are not analyzed - they are already made out of one
>> token.
>>
>> I want the score to be simply the dot product between the fields of the
>> query to the fields of the document if they have the same value.
>>
>> For example:
>> Query:
>> Field Name
>>
>> Field Value
>>
>> Field Score
>>
>> 1
>>
>> AA
>>
>> 0.1
>>
>> 7
>>
>> BB
>>
>> 0.2
>>
>> 8
>>
>> CC
>>
>> 0.3
>>
>>
>> Document 1:
>> Field Name
>>
>> Field Value
>>
>> Field Score
>>
>> 1
>>
>> AA
>>
>> 0.2
>>
>> 2
>>
>> DD
>>
>> 0.8
>>
>> 7
>>
>> CC
>>
>> 0.999
>>
>> 10
>>
>> FFF
>>
>> 0.1
>>
>>
>> Document 2:
>> Field Name
>>
>> Field Value
>>
>> Field Score
>>
>> 7
>>
>> BB
>>
>> 0.3
>>
>> 8
>>
>> CC
>>
>> 0.5
>>
>>
>> The scores should be:
>> Score(q,d1) = FIELD_1_SCORE_Q * FILED_1_SCORE_D1 = 0.1 * 0.2 = 0.02
>> Score(q,d2) = FIELD_7_SCORE_Q * FILED_7_SCORE_D2 + FIELD_8_SCORE_Q *
>> FILED_8_SCORE_D2 = (0.2 * 0.3) + (0.3 * 0.5)
>>
>> What would be the best way implement it? In terms of accuracy and
>> performances (I don't need TF and IDF calculations).
>>
>> I currently implemented it by setting boosts to the fields and query terms.
>> Then I overwritten the DefaultSimilarity class:
>>
>> public class MySimilarity extends DefaultSimilarity {
>>
>> @Override
>> public float computeNorm(String field, FieldInvertState state) {
>> return state.getBoost();
>> }
>>
>> @Override
>> public float queryNorm(float sumOfSquaredWeights) {
>> return 1;
>> }
>>
>> @Override
>> public float tf(float freq) {
>> return 1;
>> }
>>
>> @Override
>> public float idf(int docFreq, int numDocs) {
>> return 1;
>> }
>>
>> @Override
>> public float coord(int overlap, int maxOverlap) {
>> return 1;
>> }
>>
>> }
>>
>> And based on
>> http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/scoring.html
>> this should work.
>> Problems:
>> 1. Performances: I am calculating all the TF/IDF stuff and NORMS for
>> nothing...
>> 2. The score I get from the TopScoreDocCollector is not the same as I get
>> from the Explanation.
>> Here is part of my code:
>>
>> indexSearcher = new IndexSearcher(IndexReader.open(directory, true));
>> TopScoreDocCollector collector = TopScoreDocCollector.create(iTopN,
>> true); indexSearcher.search(query, collector); ScoreDoc[] hits =
>> collector.topDocs().scoreDocs; for (int i = 0; i < hits.length; ++i)
>> { int docId = hits[i].doc; Document d = indexSearcher.doc(docId);
>> double score = hits[i].score; String id = d.get(FIELD_ID);
>> Explanation explanation = indexSearcher.explain(query, docId); }
>>
>> Thanks!
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx