为什么Hibernate MassIndexer说索引已经完成，但实际上它还没有完成
为什么Hibernate MassIndexer说索引已经完成，但实际上它还没有完成
提问于 2019-06-20 11:22:04
我正在尝试使用MassIndexer在弹性搜索中索引大数据(1,350万条记录，与7-8个表关联)。它显示消息，它索引了39.08%之后的所有记录。我在本地和生产环境中也遇到了同样的问题，每次执行的百分比都不同。
fullTextEntityManager
.createIndexer(XYZ.class)
.batchSizeToLoadObjects(500).cacheMode(CacheMode.IGNORE).threadsToLoadObjects(2).idFetchSize(Integer.MIN_VALUE)
.startAndWait();日志：
23:05:25,338 (Hibernate Search: Elasticsearch transport thread-2) INFO SimpleIndexingProgressMonitor:90 - HSEARCH000031: Indexing speed: 1085.105591 documents/second; progress: 39.08%
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2) INFO SimpleIndexingProgressMonitor:87 - HSEARCH000030: 5322450 documents indexed in 4904960 ms
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2) INFO SimpleIndexingProgressMonitor:90 - HSEARCH000031: Indexing speed: 1085.115845 documents/second; progress: 39.08%
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2) INFO SimpleIndexingProgressMonitor:87 - HSEARCH000030: 5322500 documents indexed in 4904961 ms
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2) INFO SimpleIndexingProgressMonitor:90 - HSEARCH000031: Indexing speed: 1085.125854 documents/second; progress: 39.08%
23:05:36,103 (Hibernate Search: Elasticsearch transport thread-3) DEBUG request:194 - HSEARCH400082: Executed Elasticsearch HTTP POST request to path '/xyz/_forcemerge' with query parameters {} in 16734ms. Response had status 200 'OK'.
23:05:37,666 (Hibernate Search: Elasticsearch transport thread-3) DEBUG request:194 - HSEARCH400082: Executed Elasticsearch HTTP POST request to path '/xyz/_flush' with query parameters {} in 1562ms. Response had status 200 'OK'.
23:05:37,668 (Hibernate Search: Elasticsearch transport thread-3) DEBUG request:194 - HSEARCH400082: Executed Elasticsearch HTTP POST request to path '/xyz/_refresh' with query parameters {} in 1ms. Response had status 200 'OK'.
23:05:37,668 (main) INFO SimpleIndexingProgressMonitor:78 - HSEARCH000028: Reindexed 13618954 entities它应该显示仅在对所有记录进行索引之后才完成索引。
回答 2
Stack Overflow用户
回答已采纳
发布于 2019-07-15 15:14:42
Your log清楚地显示在批量索引过程中有错误，这在你最初的帖子中没有提到。
您会定期收到如下错误：
10:48:28,125 (Hibernate Search: Elasticsearch transport thread-2) ERROR LogErrorHandler:71 - HSEARCH000058: Exception occurred org.hibernate.search.exception.SearchException: HSEARCH400007: Elasticsearch request failed.
Request: POST /_bulk with parameters {refresh=false}
Response: null
Subsequent failures:
Entity com.example.model.XXXXXX Id 855665929073643520 Work Type org.hibernate.search.backend.AddLuceneWork
org.hibernate.search.exception.SearchException: HSEARCH400007: Elasticsearch request failed.
Request: POST /_bulk with parameters {refresh=false}
Response: null
at org.hibernate.search.elasticsearch.work.impl.BulkWork.lambda$execute$1(BulkWork.java:77)
at org.hibernate.search.util.impl.Futures.lambda$handler$1(Futures.java:57)
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.hibernate.search.elasticsearch.client.impl.DefaultElasticsearchClient$1.onFailure(DefaultElasticsearchClient.java:123)
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:605)
at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:396)
at org.elasticsearch.client.RestClient$1.failed(RestClient.java:375)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException
... 11 more而本质上是指一些索引请求由于Elasticsearch花费太长时间来回答而失败。
原因可能有很多。
您的Hibernate搜索配置看起来非常保守(只有两个线程)，所以我不认为您对Elasticsearch集群施加了太多压力。
我建议仔细检查一下你的Elasticsearch设置( Elasticsearch文档可能提供了一些可以帮助你的注意事项)。检查您是否有一个适当大小的Elasticsearch集群，服务器的大小是否合适，...
您可能还想调优与Elasticsearch集群通信相关的hibernate.search配置属性:超时、连接数量……请参阅https://docs.jboss.org/hibernate/search/5.11/reference/en-US/html_single/#elasticsearch-integration-configuration
Stack Overflow用户
发布于 2019-06-20 14:23:44
这看起来很像HSEARCH-3462，它在6.0.0.Alpha2中得到了修复，但没有向后移植到5.11。
长话短说:这是一个日志问题，而不是索引问题。最后一行声明所有内容都已重新索引，这是您应该相信的一行。
我会看看我们是否可以轻松地将修复移植到5.10/5.11，但可能需要一段时间才能再次释放这些分支。Backport工单(如果您需要跟踪进度)：https://hibernate.atlassian.net/browse/HSEARCH-3622
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/56678524
复制相关文章
相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号