Bool Queries | Elasticsearch.Net and NEST: the .NET clients [2.x]

You are looking at documentation for an older release. Not what you want? See the current release documentation.

» »

« Query DSL Verbatim and Strict Query Usage »

Bool Queriesedit

Writing boolean queries can grow verbose rather quickly when using the query DSL. For example, take a single bool query with only two clauses

var searchResults = this.Client.Search<Project>(s => s
    .Query(q => q
        .Bool(b => b
            .Should(
                bs => bs.Term(p => p.Name, "x"),
                bs => bs.Term(p => p.Name, "y")
            )
        )
    )
);

Now, imagine multiple nested bools; you’ll realise that this quickly becomes an exercise in hadouken indenting

Figure 2. hadouken indenting

Operator Overloadingedit

For this reason, NEST introduces operator overloading so complex bool queries become easier to write. The previous example now becomes the following with the fluent API

var searchResults = this.Client.Search<Project>(s => s
    .Query(q => q.Term(p => p.Name, "x") || q.Term(p => p.Name, "y"))
);

or, using the object initializer syntax

searchResults = this.Client.Search<Project>(new SearchRequest<Project>
{
    Query = new TermQuery { Field = "name", Value= "x" }
        || new TermQuery { Field = Field<Project>(p=>p.Name), Value = "y" }
});

A naive implementation of operator overloading would rewrite

term && term && term to

bool
|___must
   |___term
       |___bool
           |___must
               |___term
               |___term

As you can imagine this becomes unwieldy quite fast the more complex a query becomes, NEST can spot these and join them together to become a single bool query

bool
|___must
   |___term
   |___term
   |___term

Assert(
    q => q.Query() && q.Query() && q.Query(),
    Query && Query && Query,
    c => c.Bool.Must.Should().HaveCount(3)
    );

The bool DSL offers also a shorthand notation to mark a query as a must_not using the ! operator

Assert(q => !q.Query(), !Query, c => c.Bool.MustNot.Should().HaveCount(1));

And to mark a query as a filter using the + operator

Assert(q => +q.Query(), +Query, c => c.Bool.Filter.Should().HaveCount(1));

Both of these can be combined with && to form a single bool query

Assert(q => !q.Query() && !q.Query(), !Query && !Query, c => c.Bool.MustNot.Should().HaveCount(2));

Assert(q => +q.Query() && +q.Query(), +Query && +Query, c => c.Bool.Filter.Should().HaveCount(2));

Combining/Merging bool queriesedit

When combining multiple queries some or all possibly marked as must_not or filter, NEST still combines to a single bool query

bool
|___must
|   |___term
|   |___term
|   |___term
|
|___must_not
   |___term

Assert(
    q => q.Query() && q.Query() && q.Query() && !q.Query(),
    Query && Query && Query && !Query,
    c=>
    {
        c.Bool.Must.Should().HaveCount(3);
        c.Bool.MustNot.Should().HaveCount(1);
    });

c.Bool.Must.Should().HaveCount(3);

c.Bool.MustNot.Should().HaveCount(1);

Even more involved term && term && term && !term && +term && +term still only results in a single bool query:

bool
|___must
|   |___term
|   |___term
|   |___term
|
|___must_not
|   |___term
|
|___filter
   |___term
   |___term

Assert(
    q => q.Query() && q.Query() && q.Query() && !q.Query() && +q.Query() && +q.Query(),
    Query && Query && Query && !Query && +Query && +Query,
    c =>
    {
        c.Bool.Must.Should().HaveCount(3);
        c.Bool.MustNot.Should().HaveCount(1);
        c.Bool.Filter.Should().HaveCount(2);
    });

c.Bool.Must.Should().HaveCount(3);

c.Bool.MustNot.Should().HaveCount(1);

c.Bool.Filter.Should().HaveCount(2);

You can still mix and match actual bool queries with the bool DSL e.g bool(must=term, term, term) && !term would still merge into a single bool query.

Assert(
    q => q.Bool(b => b.Must(mq => mq.Query(), mq => mq.Query(), mq => mq.Query())) && !q.Query(),
    new BoolQuery { Must = new QueryContainer[] { Query, Query, Query } } && !Query,
    c =>
    {
        c.Bool.Must.Should().HaveCount(3);
        c.Bool.MustNot.Should().HaveCount(1);
    });

c.Bool.Must.Should().HaveCount(3);

c.Bool.MustNot.Should().HaveCount(1);

NEST will also do the same with should`s or `|| when it sees that the boolean queries in play only consist of should clauses. This is because the bool query does not quite follow the same boolean logic you expect from a programming language.

To summarize, the latter:

term || term || term

becomes

bool
|___should
   |___term
   |___term
   |___term

but term1 && (term2 || term3 || term4) does not become

bool
|___must
|   |___term1
|
|___should
   |___term2
   |___term3
   |___term4

This is because when a bool query has only should clauses, at least one of them must match. When that bool query also has a must clause then the should clauses start acting as a boost factor and none of them have to match, drastically altering its meaning.

So in the previous you could get back results that only contain term1. This is clearly not what you want in the strict boolean sense of the input.

To aid with this, NEST rewrites the previous query to

bool
|___must
   |___term1
   |___bool
       |___should
           |___term2
           |___term3
           |___term4

Assert(
    q => q.Query() && (q.Query() || q.Query() || q.Query()),
    Query && (Query || Query || Query),
    c =>
    {
        c.Bool.Must.Should().HaveCount(2);
        var lastClause = c.Bool.Must.Last() as IQueryContainer;
        lastClause.Should().NotBeNull();
        lastClause.Bool.Should().NotBeNull();
        lastClause.Bool.Should.Should().HaveCount(3);
    });

c.Bool.Must.Should().HaveCount(2);

var lastClause = c.Bool.Must.Last() as IQueryContainer;

lastClause.Should().NotBeNull();

lastClause.Bool.Should().NotBeNull();

lastClause.Bool.Should.Should().HaveCount(3);

add parentheses to force evaluation order

Also note that using shoulds as boosting factors can be really powerful so if you need this always remember that you can mix and match an actual bool query with the bool dsl.

There is another subtle situation where NEST will not blindly merge 2 bool queries with only should clauses. Imagine the following:

bool(should=term1, term2, term3, term4, minimum_should_match=2) || term5 || term6

if NEST identified both sides of the OR operation as only containing should clauses and it would join them together it would give a different meaning to the minimum_should_match parameter of the first boolean query. Rewriting this to a single bool with 5 should clauses would break because only matching on term5 or term6 should still be a hit.

Assert(
    q => q.Bool(b => b
        .Should(mq => mq.Query(), mq => mq.Query(), mq => mq.Query(), mq => mq.Query())
        .MinimumShouldMatch(2)
        )
         || !q.Query() || q.Query(),
    new BoolQuery
    {
        Should = new QueryContainer[] { Query, Query, Query, Query },
        MinimumShouldMatch = 2
    } || !Query || Query,
    c =>
    {
        c.Bool.Should.Should().HaveCount(3);
        var nestedBool = c.Bool.Should.First() as IQueryContainer;
        nestedBool.Bool.Should.Should().HaveCount(4);
    });

c.Bool.Should.Should().HaveCount(3);

var nestedBool = c.Bool.Should.First() as IQueryContainer;

nestedBool.Bool.Should.Should().HaveCount(4);

Locked bool queriesedit

NEST will not combine bool queries if any of the query metadata is set e.g if metadata such as boost or name are set, NEST will treat these as locked.

Here we demonstrate that two locked bool queries are not combined

Assert(
    q => q.Bool(b => b.Name("leftBool").Should(mq => mq.Query()))
         || q.Bool(b => b.Name("rightBool").Should(mq => mq.Query())),
    new BoolQuery { Name = "leftBool", Should = new QueryContainer[] { Query } }
    || new BoolQuery { Name = "rightBool", Should = new QueryContainer[] { Query } },
    c => AssertDoesNotJoinOntoLockedBool(c, "leftBool"));

neither are two bool queries where either right query is locked

Assert(
    q => q.Bool(b => b.Should(mq => mq.Query()))
         || q.Bool(b => b.Name("rightBool").Should(mq => mq.Query())),
    new BoolQuery { Should = new QueryContainer[] { Query } }
    || new BoolQuery { Name = "rightBool", Should = new QueryContainer[] { Query } },
    c => AssertDoesNotJoinOntoLockedBool(c, "rightBool"));

or the left query is locked

Assert(
    q => q.Bool(b => b.Name("leftBool").Should(mq => mq.Query()))
         || q.Bool(b => b.Should(mq => mq.Query())),
    new BoolQuery { Name = "leftBool", Should = new QueryContainer[] { Query } }
    || new BoolQuery { Should = new QueryContainer[] { Query } },
    c => AssertDoesNotJoinOntoLockedBool(c, "leftBool"));

Perfomance considerationsedit

If you have a requirement of combining many many queries using the bool dsl please take the following into account.

You can use bitwise assignments in a loop to combine many queries into a bigger bool.

In this example we are creating a single bool query with a 1000 must clauses using the &= assign operator.

var c = new QueryContainer();

var q = new TermQuery { Field = "x", Value = "x" };

c &= q;

|===
|     Median|     StdDev|       Gen 0|  Gen 1|  Gen 2|  Bytes Allocated/Op
|  1.8507 ms|  0.1878 ms|    1,793.00|  21.00|      -|        1.872.672,28
|===

As you can see while still fast its causes a lot of allocations to happen because with each iteration we need to re evaluate the mergability of our bool query.

Since we already know the shape of our bool query in advance its much much faster to do this instead:

QueryContainer q = new TermQuery { Field = "x", Value = "x" };

var x = Enumerable.Range(0, 1000).Select(f => q).ToArray();

var boolQuery = new BoolQuery
{
    Must = x
};

|===
|      Median|     StdDev|   Gen 0|  Gen 1|  Gen 2|  Bytes Allocated/Op
|  31.4610 us|  0.9495 us|  439.00|      -|      -|            7.912,95
|===

The drop both in performance and allocations is tremendous!

If you assigning many bool queries prior to NEST 2.4.6 into a bigger bool using an assignment loop the client did not do a good job flattening the result in the most optimal way and could cause a stackoverflow when doing ~2000 iterations. This only applied to bitwise assigning many boolean queries. Other queries behave fine in earlier versions. Since NEST 2.4.6 you can combine as many bool queries as you’d like this way too. See PR #2335 on github for more information

« Query DSL Verbatim and Strict Query Usage »

Bool Queriesedit

Operator Overloadingedit

Combining/Merging bool queriesedit

Locked bool queriesedit

Perfomance considerationsedit

Top Videos

Be in the know with the latest and greatest from Elastic.