topic Re: Elasticsearch failed to start during deployment in Administration and Deployment

Elasticsearch failed to start during deployment

Symphony — Thu, 19 Dec 2019 03:20:31 GMT

I'm encountering an problem during the deployment of VI 10.51.

The ERROR message is like this:elasticsearch (pid 11502) is running...", "Elasticsearch not started correctly, please check logs under /opt/sas/viya/config/var/log/svi-elasticsearch"

and elasticsearch.log shows :

[2019-12-18T13:51:37,641][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [N3-Q2Nd] SSL Problem Client requested protocol TLSv1 not enabled or not supported
javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported...

I've uploaded part of the elasticsearch.log, saslogon.log and also elasticsearch.yml under /opt/viya/config/etc/elasticsearch/default/.

I've tried to change SSLProtocol to TLSv1 TLSv1.1 TLSv1.2, and added -Djdk.tls.rejectClientInitiatedRenegotiation=true to jvm.options, neither worked.

I also checked http.enabled_protocols under elasticsearch.yml and it's pointing to TLSv1.2.

The OS version is Red Hat Enterprise Linux Server release 6.7 (Santiago)

httpd and mod_ssl version: 2.2.15-45

java version: jdk1.8.0_231

It would be great if there's any suggestions!

Re: Elasticsearch failed to start during deployment

AnandVyas — Thu, 19 Dec 2019 04:31:49 GMT

Hi @Symphony

Have you checked this section from the link -> https://docs.search-guard.com/latest/configuring-tls

By default Search Guard disables TLSv1 because it is unsecure.

If you need to use TLSv1 and you know what you are doing, you can re-enable it like:**

searchguard.ssl.http.enabled_protocols:
  - "TLSv1"
  - "TLSv1.1"
  - "TLSv1.2"

Re: Elasticsearch failed to start during deployment

Symphony — Thu, 19 Dec 2019 05:06:44 GMT

Hi @AnandVyas

I've just added TLSv1 and TLSv1.1 to the elasticsearch.yml,

After I restarted sas-viya-svi-elasticsearch-default, the ERROR still exists:

[2019-12-19T12:58:41,628][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [N3-Q2Nd] SSL Problem Client requested protocol TLSv1 not enabled or not supported
javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1537) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_231]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:255) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1162) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]

Re: Elasticsearch failed to start during deployment

AnandVyas — Thu, 19 Dec 2019 06:13:26 GMT

Okay, I would suggest you to check the TLS version in the java you are using and also see if that has been configured in the elasticsearch.yml config file.

you can use the below command to check:
path-to-openssl s_client -connect myhost.example.com:secured–port -CAfile trustedcerts.pem

Also, would advise to reach out to SAS TS for quick resolution.

Re: Elasticsearch failed to start during deployment

Symphony — Fri, 20 Dec 2019 02:53:37 GMT

Hi @AnandVyas ，

Thanks a lot for your suggestions!

Here's the result I get :

[root@svi bin]# ./openssl s_client -connect svi.local:443 -CAfile /home/viya/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem
CONNECTED(00000003)
depth=0 C = US, O = Self-Signed Certificate, CN = svi.local
verify return:1
---
Certificate chain
0 s:/C=US/O=Self-Signed Certificate/CN=svi.local
i:/C=US/O=Self-Signed Certificate/CN=svi.local
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDsjCCApqgAwIBAgIIJTbscKjKqlQwDQYJKoZIhvcNAQELBQAwQzELMAkGA1UE
BhMCVVMxIDAeBgNVBAoTF1NlbGYtU2lnbmVkIENlcnRpZmljYXRlMRIwEAYDVQQD
EwlzdmkubG9jYWwwHhcNMTkxMjE4MDMxNzA2WhcNMjYxMjE4MDMzMjA2WjBDMQsw
CQYDVQQGEwJVUzEgMB4GA1UEChMXU2VsZi1TaWduZWQgQ2VydGlmaWNhdGUxEjAQ
BgNVBAMTCXN2aS5sb2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AMGNvgqmkxFCevL+jae6tZtJMcL2l9wlHTd2FaCyl6rEHZ8YDWPsf/mr2nVlcc/t
6QCmdhdW533iDTlxzgej7pT1irBxG2ob8XpK+MYDPPDDG55wOCabMu6tL0JQoDVQ
ygKfWGrsJ/UgPOt6o2z7Nn+oeGS22f1w++NUIIUDgncDrm4neWlN319TfvCzqL1I
3mugPbhUowcSsoUZuw8oz4fSqb8pU7U/zSBIt3PL5NR7v4Lr9emArj9xk+o360pU
6cBQ//R9B05FtxnX+7796+JI7wedCoNKWwgAqJ+bFg0x1oXLYviQqCX/CzMizDdg
ep21ME+APRziGd+PLyV9Xb0CAwEAAaOBqTCBpjAdBgNVHQ4EFgQURVyLI6JZta3q
hWm5fWuZOOGmiV4wgYQGA1UdEQR9MHuCCXN2aS5sb2NhbIIJc3ZpLmxvY2Fsgglz
dmkubG9jYWyCCXN2aS5sb2NhbIIDc3ZpggsqLnN2aS5sb2NhbIILKi5zdmkubG9j
YWyCBSouc3Zpgglsb2NhbGhvc3SHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAGHBAq2
wN0wDQYJKoZIhvcNAQELBQADggEBAHh+cM/cdXwJsMbck7vPRkRBdLm3MTjcC6TG
Lu6lcoQpWip8GR3LjIUi8dLnMN4a2pvknCoKmABk3O+tkY3Vkvel/iwTzzYSwBl/
3eQuEYsyAD6LjbxLEk0VYHync5YVmj62zM3v6YweSWK68KNrf8fIZNQklkOKycXS
Rdlvyx+IvKivybOq667HvsoZUWizC95Xcr2udr20KQylTGAORkI+7jG4meqlYWBd
glvnRpSe9sYAK5uc1Hadsd1vm5FEikoQ5dLCwxDjBGuKS7/akI7rIYpTOj1gKTu/
W4MzIFkABa/MJvyGj1TOip2kjoawhoX+oqCqop9E7mr6M+DAirI=
-----END CERTIFICATE-----
subject=/C=US/O=Self-Signed Certificate/CN=svi.local
issuer=/C=US/O=Self-Signed Certificate/CN=svi.local
---
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 1641 bytes and written 375 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: BE7ABAD54AAEF08BF04A04998793150F64CBC345FE58E05CDFC11B5843695BFE
Session-ID-ctx:
Master-Key: F1727AB92A4BE35AA765E6D0D17A37080569EE6D56F33DCD610AFDD9EDE47B8155AD6736158B74EBE9F417B6A8B802B6
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 51 a7 c7 7e 49 12 6e 97-43 1c b2 a1 1b 09 a7 f1 Q..~I.n.C.......
0010 - 2c af 04 21 c1 f0 f8 b8-11 1e 69 d1 30 76 1c c5 ,..!......i.0v..
0020 - 93 a3 b4 66 94 47 26 c3-b5 5f 01 17 c6 c8 61 67 ...f.G&.._....ag
0030 - e2 55 23 dc 68 f6 44 66-39 6b f9 52 05 4d 2d 6d .U#.h.Df9k.R.M-m
0040 - be 48 5b 18 e4 43 0e aa-ea 63 63 bb 47 2f 1b 36 .H[..C...cc.G/.6
0050 - ac 25 5b a6 f5 e7 d2 0a-8f a2 6c da 49 77 31 ef .%[.......l.Iw1.
0060 - f8 62 45 f5 18 c4 b2 a5-d6 c8 61 b5 98 5c 3a d9 .bE.......a..\:.
0070 - fe f7 cf 9c 62 73 83 b5-46 4c 3a 30 13 bf 82 0d ....bs..FL:0....
0080 - 59 b3 38 3f 16 83 49 c9-4a 71 39 1e bc 3d 1d 13 Y.8?..I.Jq9..=..
0090 - fe 43 b6 ff 39 ac 8f 43-af 67 66 0b f6 ba be 77 .C..9..C.gf....w
00a0 - a7 aa d3 5e 21 58 c4 ea-4d 2e c3 c8 b6 ea bf 7c ...^!X..M......|
00b0 - e5 f1 4f 65 ae a7 0a 48-b9 4a d1 ef be ca 87 a6 ..Oe...H.J......

Start Time: 1576808860
Timeout : 300 (sec)
Verify return code: 0 (ok)

---

Here's another thing, the environment has two Java environments, one is java-1.8.0-openjdk-1.8.0.45-35.b13.el6.x86_64 installed by yum. Another is jdk-8u231-linux-x64.tar.gz downloaded straight from official website. I modified java parameters under vars.yml :

# deployment and set the path to Java in sasenv_java_home.
sas_install_java: false

# User specific location of where Java resides on the host.
# If not set, the deployment will use the Java defined on the host.
# This is an optional setting.
# Example:
# sasenv_java_home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre
sasenv_java_home: /home/java/javatgz/jdk1.8.0_231/jre

I wonder if that has any effect to this situation.

Should I change soft links for java under /usr/bin and /etc/alternatives?

Also I've already emailed SAS TS with attachments of logs and yml files, still waiting for reply.

Re: Elasticsearch failed to start during deployment

Symphony — Fri, 20 Dec 2019 07:22:54 GMT

Hi @AnandVyas ,

Is there a way to config Elasticsearch not to use TLS before deployment?

I tried to change searchguard.ssl.http.enabled to false under elsticsearch.yml but it seems to be reset to true automatically when I rerun the ansible-playbook command.