RESTful Search with Spring and Hibernate Search 6

2020/11/18

Introduction

One possible extension of a simple REST-API is to enable searching. So decided to write a short introduction of how one can get a simple search with pagination in the Spring Framework.

The excellent Hibernate Search project got an refresh in it’s 6.0 version. At the time of writing it has already arrived at the candidate release stage. I highly recommend the original doc for all features (1. Getting Started). But I imagine the topic of search is completely new to many developers. This walkthrough that ends up at a minimal application should speed up your understanding of search concepts or help out if your stuck.

This post is dedicated to the awesome https://discourse.hibernate.org/u/yrodiere that helps out a lot of people on advanced Hibernate Search questions here.


Baseline

I want to get fast to the search stuff. Please check all this stuff below. I expect you to already know it (or learn it before going on):


This image shows a quick overview of the components that we are going to build now:

1 Spring @RestController and Pagination

Before we get started with searching, we’ll need a REST-Endpoint. For our example it should also support Pagination. This can be done with the Spring @RestController and Pageable.

# Example request on the endpoint
$ curl localhost:8080/person?query=mia\&page=0\&size=2 #(use escaped \& in shell)
{
  "content" : [ {
    "id" : 4,
    "name" : "Mia Banks",
    "address" : "418-6140 Nec, Rd."
  }, {
    "id" : 66,
    "name" : "Joan Sheppard",
    "address" : "Ap #726-7766 Mi, Av."
  } ],
  "pageable" : {
    "sort" : { ... },
    "offset" : 0,
    "pageNumber" : 0,
    "pageSize" : 2,
    "paged" : true,
    "unpaged" : false
  },
  "totalPages" : 2,
  "totalElements" : 3,
  "last" : false,
  "size" : 2,
  "number" : 0,
  "sort" : { ... },
  "numberOfElements" : 2,
  "first" : true,
  "empty" : false
}

This endpoint is achieved with the following code:

@RestController
@RequestMapping(value = "/person")
public class PersonController {

    @Autowired
    PersonSearchService searchService;

    // Spring automatically handles the Pageable parameter and Page result
    @GetMapping
    public ResponseEntity<Page<Person>> searchPersons(Pageable pageable, @RequestParam("query") String query) {
        Page<Person> result = searchService.search(pageable, query);
        return ResponseEntity.ok(result);
    }
}

2 JPA Setup

To get an example Entity Person in an in-memory sql database (called H2) we need to set up some classes. You are expected to already understand and know how to do this. So if you get stuck in this section with JPA/Repositories/@Entity please go to some simpler JPA tutorial and come back here.

Our Entity

@Entity
@Table(name = "PERSON")
public class Person {

    @Column(name = "ID")
    @Id
    @GeneratedValue
    private int id;

    @Column(name = "NAME")
    private String name;

    @Column(name = "ADDRESS")
    private String address;

    // getters and setters
}

The Repository (for Database access)

@Repository
public interface PersonRepository extends PagingAndSortingRepository<Person, Integer> {
  // intentionally left blank, see PagingAndSortingRepository interface definition
}

3 Mapping the Entity to the Search Index

For Searching with a Search-Framework we want to put our entity into a Search-Index. The technical part is handled by Hibernate Search and Lucene (or Elastic-Search) but we need to tell the framework what entity fields we want to put into the index and where.

You can image the search index like the following flat file with a “document” per person:

# from {"id": 4, "name": "Mia Banks", "address": "418-6140 Nec, Rd."} 
# we get something like
person.name: [mia, banks]
person.adress: [418, 6140, nec, rd.]

# {"id": 4, "name": "Joan Sheppard", "address": "Ap #726-7766 Mi, Av."}
person.name: [joan, sheppard]
person.adress: [ap, 726, 7766, mi, av]

This index mapping is a whole topic on its own, but you can read more in the official doc at 6.3 Entity/Index Mapping.

For this minimal Example we need to:

  1. set up Hibernate Search and Lucene dependencies, see 1. Getting Started

    <properties>
      // ...
      <hibernate.version>5.4.23.Final</hibernate.version>
    </properties>
    
    <dependencies>
      // spring-starter-data-jpa, spring-starter-web, h2 database
      // ...
    
      <dependency>
        <groupId>org.hibernate.search</groupId>
        <artifactId>hibernate-search-mapper-orm</artifactId>
        <version>6.0.0.CR2</version>
      </dependency>
      <dependency>
        <groupId>org.hibernate.search</groupId>
        <artifactId>hibernate-search-backend-lucene</artifactId>
        <version>6.0.0.CR2</version>
      </dependency>
    
      // ...
    </dependencies> 
    
  2. map our entity to the index (with annotations), see 6.3 Enity/Index Mapping

    @@ -7,6 +7,7 @@
     import javax.persistence.*;
    
     @Entity
    +@Indexed
     @Table(name = "PERSON")
     public class Person {
    
    @@ -16,9 +17,11 @@
         private int id;
    
         @Column(name = "NAME")
    +    @FullTextField
         private String name;
    
         @Column(name = "ADDRESS")
    +    @FullTextField
         private String address;
    
         public int getId() {
    
  3. set up a service class for searching, see 10. Searching

    @Service
    public class PersonSearchService {
    
        @PersistenceContext
        EntityManager entityManager;
    
        @Transactional(readOnly = true)
        public Page<Person> search(Pageable pageable, String query) {
            SearchSession session = Search.session(entityManager);
    
            SearchResult<Person> result = session.search(Person.class)
                    .where(
                            f -> f.match().fields("name", "address")
                                    .matching(query)
                                    .fuzzy(1)
                    )
                    .fetch((int) pageable.getOffset(), pageable.getPageSize());
    
            return new PageImpl<>(result.hits(), pageable, result.total().hitCount());
        }
    }
    

    (this is a minimal example, of course Hibernate Search is capable of sorting and way more complex searches)

  4. set up automatic indexing on startup; so that even if the database changes while our server is down, the index will get updated on startup. Indexing/Reindexing is also a big topic to consider, see 9. Indexing Hibernate ORM entities

    @Component
    public class IndexOnStartup implements CommandLineRunner {
    
        @PersistenceContext
        EntityManager entityManager;
    
        @Override
        @Transactional(readOnly = true)
        public void run(String... args) throws Exception {
            SearchSession searchSession = Search.session( entityManager );
            MassIndexer indexer = searchSession.massIndexer( Person.class );
            indexer.startAndWait();
        }
    }
    

4 Searching

Now everything should be set up correctly. To try out the search we need some example data, which I put into the src/main/resources/persons.json file.

To load the persons into the database on startup, you can use another CommandLineRunner Component:

@Component
public class LoadTestData implements CommandLineRunner {
    @Autowired
    private ObjectMapper objectMapper;

    @Autowired
    private PersonRepository repo;


    @Override
    @Transactional
    public void run(java.lang.String... args) throws Exception {
        Person[] persons = objectMapper.readValue(new ClassPathResource("persons.json").getFile(), Person[].class);
        repo.saveAll(Arrays.asList(persons));
    }
}

Now you can play around with this test data and some queries:

5 Draft: Testing

TODO: Example Testcase TODO: Example Massindexer Test

Notes:

6 Draft: Long running Transactions

TODO: Explain @Transactional(readOnly = true) here and its problemantic effects on the default spring hikari connection pool.

@Override
        @Transactional(readOnly = true)
        public void run(String... args) throws Exception {
            SearchSession searchSession = Search.session( entityManager );
            MassIndexer indexer = searchSession.massIndexer( Person.class );
            indexer.startAndWait();
        }

7 Conclusion

Finally a small recap how a search runs through our little application:

  1. Our PersonController gets an incoming search at the @GetMapping

    e.g.: localhost:8080/person?query=mia&page=0&size=2

  2. It reads the page size (2) and that its the first page (0)

  3. It reads the @RequestParam(“query”) to be "mia"

  4. These parameters are sent to our PersonSearchService

  5. The PersonSearchService runs the query on the Index using Hibernate-Search and Lucene

  6. The PersonSearchService returns the result as a Spring Page

  7. The PersonController returns the page as JSON.

Further topics

Appendix