Introduction
One possible extension of a simple REST-API is to enable searching. So decided to write a short introduction of how one can get a simple search with pagination in the Spring Framework.
The excellent Hibernate Search project got an refresh in it’s 6.0 version. At the time of writing it has already arrived at the candidate release stage. I highly recommend the original doc for all features (1. Getting Started). But I imagine the topic of search is completely new to many developers. This walkthrough that ends up at a minimal application should speed up your understanding of search concepts or help out if your stuck.
This post is dedicated to the awesome https://discourse.hibernate.org/u/yrodiere that helps out a lot of people on advanced Hibernate Search questions here.
Baseline
I want to get fast to the search stuff. Please check all this stuff below. I expect you to already know it (or learn it before going on):
- REST I assume you know what it is, it’s semantic implications, and know the parts of an URL are used. (e.g. a:
GET http://google.de/?q=cats
) - Spring (for Web) I assume you can set up a basic Spring Web project (use Spring Initializr, or Learn It).
- Hibernate I assume you know how to map JPA Entities. (I like this summary of annotations, ch. 2.2 Domain-Driven Design in a Spring application)
- Spring Repositories I assume you know how to use Spring Repositories to save and load the Entities. Introduction to Spring Data JPA
This image shows a quick overview of the components that we are going to build now:
1 Spring @RestController and Pagination
Before we get started with searching, we’ll need a REST-Endpoint. For our example it should also support Pagination. This can be done with the Spring @RestController and Pageable.
# Example request on the endpoint
$ curl localhost:8080/person?query=mia\&page=0\&size=2 #(use escaped \& in shell)
{
"content" : [ {
"id" : 4,
"name" : "Mia Banks",
"address" : "418-6140 Nec, Rd."
}, {
"id" : 66,
"name" : "Joan Sheppard",
"address" : "Ap #726-7766 Mi, Av."
} ],
"pageable" : {
"sort" : { ... },
"offset" : 0,
"pageNumber" : 0,
"pageSize" : 2,
"paged" : true,
"unpaged" : false
},
"totalPages" : 2,
"totalElements" : 3,
"last" : false,
"size" : 2,
"number" : 0,
"sort" : { ... },
"numberOfElements" : 2,
"first" : true,
"empty" : false
}
This endpoint is achieved with the following code:
@RestController
@RequestMapping(value = "/person")
public class PersonController {
@Autowired
PersonSearchService searchService;
// Spring automatically handles the Pageable parameter and Page result
@GetMapping
public ResponseEntity<Page<Person>> searchPersons(Pageable pageable, @RequestParam("query") String query) {
Page<Person> result = searchService.search(pageable, query);
return ResponseEntity.ok(result);
}
}
2 JPA Setup
To get an example Entity Person in an in-memory sql database (called H2) we need to set up some classes. You are expected to already understand and know how to do this. So if you get stuck in this section with JPA/Repositories/@Entity please go to some simpler JPA tutorial and come back here.
Our Entity
@Entity
@Table(name = "PERSON")
public class Person {
@Column(name = "ID")
@Id
@GeneratedValue
private int id;
@Column(name = "NAME")
private String name;
@Column(name = "ADDRESS")
private String address;
// getters and setters
}
The Repository (for Database access)
@Repository
public interface PersonRepository extends PagingAndSortingRepository<Person, Integer> {
// intentionally left blank, see PagingAndSortingRepository interface definition
}
3 Mapping the Entity to the Search Index
For Searching with a Search-Framework we want to put our entity into a Search-Index. The technical part is handled by Hibernate Search and Lucene (or Elastic-Search) but we need to tell the framework what entity fields we want to put into the index and where.
You can image the search index like the following flat file with a “document” per person:
# from {"id": 4, "name": "Mia Banks", "address": "418-6140 Nec, Rd."}
# we get something like
person.name: [mia, banks]
person.adress: [418, 6140, nec, rd.]
# {"id": 4, "name": "Joan Sheppard", "address": "Ap #726-7766 Mi, Av."}
person.name: [joan, sheppard]
person.adress: [ap, 726, 7766, mi, av]
This index mapping is a whole topic on its own, but you can read more in the official doc at 6.3 Entity/Index Mapping.
For this minimal Example we need to:
-
set up Hibernate Search and Lucene dependencies, see 1. Getting Started
<properties> // ... <hibernate.version>5.4.23.Final</hibernate.version> </properties> <dependencies> // spring-starter-data-jpa, spring-starter-web, h2 database // ... <dependency> <groupId>org.hibernate.search</groupId> <artifactId>hibernate-search-mapper-orm</artifactId> <version>6.0.0.CR2</version> </dependency> <dependency> <groupId>org.hibernate.search</groupId> <artifactId>hibernate-search-backend-lucene</artifactId> <version>6.0.0.CR2</version> </dependency> // ... </dependencies>
-
map our entity to the index (with annotations), see 6.3 Enity/Index Mapping
@@ -7,6 +7,7 @@ import javax.persistence.*; @Entity +@Indexed @Table(name = "PERSON") public class Person { @@ -16,9 +17,11 @@ private int id; @Column(name = "NAME") + @FullTextField private String name; @Column(name = "ADDRESS") + @FullTextField private String address; public int getId() {
-
set up a service class for searching, see 10. Searching
@Service public class PersonSearchService { @PersistenceContext EntityManager entityManager; @Transactional(readOnly = true) public Page<Person> search(Pageable pageable, String query) { SearchSession session = Search.session(entityManager); SearchResult<Person> result = session.search(Person.class) .where( f -> f.match().fields("name", "address") .matching(query) .fuzzy(1) ) .fetch((int) pageable.getOffset(), pageable.getPageSize()); return new PageImpl<>(result.hits(), pageable, result.total().hitCount()); } }
(this is a minimal example, of course Hibernate Search is capable of sorting and way more complex searches)
-
set up automatic indexing on startup; so that even if the database changes while our server is down, the index will get updated on startup. Indexing/Reindexing is also a big topic to consider, see 9. Indexing Hibernate ORM entities
@Component public class IndexOnStartup implements CommandLineRunner { @PersistenceContext EntityManager entityManager; @Override @Transactional(readOnly = true) public void run(String... args) throws Exception { SearchSession searchSession = Search.session( entityManager ); MassIndexer indexer = searchSession.massIndexer( Person.class ); indexer.startAndWait(); } }
4 Searching
Now everything should be set up correctly. To try out the search we need some example data, which I put into the src/main/resources/persons.json file.
To load the persons into the database on startup, you can use another CommandLineRunner Component:
@Component
public class LoadTestData implements CommandLineRunner {
@Autowired
private ObjectMapper objectMapper;
@Autowired
private PersonRepository repo;
@Override
@Transactional
public void run(java.lang.String... args) throws Exception {
Person[] persons = objectMapper.readValue(new ClassPathResource("persons.json").getFile(), Person[].class);
repo.saveAll(Arrays.asList(persons));
}
}
Now you can play around with this test data and some queries:
- http://localhost:8080/person?query=mia&page=0&size=2
- http://localhost:8080/person?query=aliquam&page=0&size=100
- http://localhost:8080/person?query=street&page=1&size=2
- …
5 Draft: Testing
TODO: Example Testcase TODO: Example Massindexer Test
Notes:
- If you use Elasticsearch as Backend, consider
spring.jpa.properties.hibernate.search.automatic_indexing.synchronization.strategy: sync
for the tests because there is the short delay before the index gets updated
6 Draft: Long running Transactions
TODO: Explain @Transactional(readOnly = true) here and its problemantic effects on the default spring hikari connection pool.
@Override
@Transactional(readOnly = true)
public void run(String... args) throws Exception {
SearchSession searchSession = Search.session( entityManager );
MassIndexer indexer = searchSession.massIndexer( Person.class );
indexer.startAndWait();
}
7 Conclusion
Finally a small recap how a search runs through our little application:
-
Our PersonController gets an incoming search at the @GetMapping
-
It reads the page size (2) and that its the first page (0)
-
It reads the @RequestParam(“query”) to be
"mia"
-
These parameters are sent to our PersonSearchService
-
The PersonSearchService runs the query on the Index using Hibernate-Search and Lucene
-
The PersonSearchService returns the result as a Spring Page
-
The PersonController returns the page as JSON.
Further topics
- The Spring Pageable and Hibernate Search both handle sorts. You could extend the PersonSearchService to support sorting.
- The indexing could be optimized or tweaked by using custom tokenizers, analyzers or an extended mapping.
- To support an efficient includes-like (or wildcard-like) search NGRAM or EdgeNGRAM tokenizers could be used.
- JPA Value-Objects that were defined with @Embeddable can be included in the index with @IndexedEmbedded (also see nested Documents)
Appendix
- Gitlab Demo Project Sources: https://gitlab.com/peter-mueller/spring-hibernate-search-6-demo
- Official Hibernate Search Documentation: https://docs.jboss.org/hibernate/search/6.0/reference/en-US/html_single/