GCE上にESインスタンスを3つつくってNode3台のクラスター構成にしている
これがあるときストレージがいっぱいになったことで一つのNodeがNODE_LEFT
のステータスになってしまった
$ curl -XGET host:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason ... index-1 3 r UNASSIGNED NODE_LEFT ...
この復旧までにおこなったメモ
まずはシャードの自動割り当てを止める
curl -XPUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.enable": "none" } }'
そして、レプリカシャード数を0に
curl -XPUT "localhost:9200/scored-vacation-rental-v9-vrbo/_settings" -H 'Content-Type: application/json' -d' { "settings": { "index": { "number_of_replicas": 0 } } }'
その後、NODE_LEFT
ステータスになっているインスタンスに入って、ESを起動させてみたがエラーになって起動しなかった
$ sudo systemctl start elasticsearch
その時のログ
Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]]; at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113) at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086) at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer. at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100) at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.Command.main(Command.java:90) at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43) Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29] at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204) at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198) at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620) at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109) ... 10 more Exception in thread "main" 2022-07-10 15:57:43,551 main ERROR No Log4j 2 configuration file found. Usin SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplica at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]]; at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113) at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086) at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer. at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100) at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.Command.main(Command.java:90) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29] at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204) at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198) at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620) at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109) ... 9 moreJul 10 15:57:35 es7-staylist-elasticsearch-8t1x systemd-entrypoint[21075]: Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]]; at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113) at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086) at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer. at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100) at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.Command.main(Command.java:90) at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43) Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29] at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204) at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198) at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620) at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109) ... 10 more Exception in thread "main" 2022-07-10 15:57:43,551 main ERROR No Log4j 2 configuration file found. Usin SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplica at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]]; at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113) at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086) at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer. at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100) at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.Command.main(Command.java:90) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29] at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204) at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198) at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620) at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109) ... 9 more
これはインスタンス作り直したほうが早そうだったので、GCEのインスタンスを停止し再起動したらうまくESも起動された
レプリカシャード数とシャードの自動割り当てを元に戻して復旧を完了した