Trouble-Shooting

<- Back to Index

This section describes some common errors with this library and grpc in general, and how to solve them. Please note that this page can never cover all cases, please also search the existing issues/PRs (both opened and closed ones) for related topics. If a corresponding topic already exists, leave us a comment / info so that we know that you are also affected. If there is no such topic, feel free to open a new one as described at the bottom of this page.

Table of Contents

NoClassDefFoundError, ClassNotFoundException, NoSuchMethodError, AbstractMethodError

Example

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'client' defined in file [~/.../MyGrpcClient.class]: Initialization of bean failed; nested exception is java.lang.NoClassDefFoundError: io/grpc/TlsChannelCredentials$Feature
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602)
    [...]
Caused by: java.lang.NoClassDefFoundError: io/grpc/TlsChannelCredentials$Feature
    at io.grpc.netty.ProtocolNegotiators.<clinit>(ProtocolNegotiators.java:92)

The Problem

The server/client does not start because some class or method is missing.
This is usually the case if the grpc-libraries use slightly different versions.

The solution

Make sure to use exactly the same version for all grpc-java versions.

Add the following entry to your dependencyManagement section of your project:

<dependency>
    <groupId>io.grpc</groupId>
    <artifactId>grpc-bom</artifactId>
    <version>${grpcVersion}</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

You can use a similar approach for gradle:

dependencyManagement {
    imports {
        mavenBom "io.grpc:grpc-bom:${grpcVersion}"

Note: grpc-spring-boot-starter isn’t strictly bound to a specific version of grpc-java, so you can also use this to change the version of grpc-java you are using in your project.

See also Could not find TLS ALPN provider

Transport failed

Server-side

2019-07-07 10:05:46.217  INFO 6552 --- [-worker-ELG-3-5] i.g.n.s.i.g.n.N.connections              : Transport failed

io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 16030100820100007e0303aae6126974cbb4638b325d6bdb
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85) ~[grpc-netty-shaded-1.21.0.jar:1.21.0]
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:318) ~[grpc-netty-shaded-1.21.0.jar:1.21.0]
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:251) ~[grpc-netty-shaded-1.21.0.jar:1.21.0]
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450) [grpc-netty-shaded-1.21.0.jar:1.21.0]

Client-side

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235)
    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
    at net.devh.boot.grpc.examples.lib.SimpleGrpc$SimpleBlockingStub.sayHello(SimpleGrpc.java:178)
    [...]
Caused by: io.grpc.netty.shaded.io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00001204000000000000037fffffff000400100000000600002000000004080000000000000f0001
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1204)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)

The Problem

The server runs in PLAINTEXT mode, but the client tries to connect it in TLS (default) mode.

The simple solution

a.k.a.: Configure the client to connect in PLAINTEXT mode (Not recommended for production).

Add the following entry to your client side application config:

grpc.client.__name__.negotiationType=PLAINTEXT

The better solution

a.k.a.: Configure the server to run in TLS mode (Recommended).

Add the following entry to your sever side application config:

grpc.server.security.enabled=true
grpc.server.security.certificateChain=file:certificates/server.crt
grpc.server.security.privateKey=file:certificates/server.key

First received frame was not SETTINGS

Client-side

Status{code=INTERNAL, description=http2 exception, cause=...Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f

The Problem

485454502f -> HTTP/ from HTTP/1.1

You are probably connecting to a normal web server and not a HTTP/2 grpc server or proxy.

The Solution

Check your address configuration. Make sure that you are connecting to the correct port.

e.g. dns:///my-service.grpc.example -> dns:///my-service.grpc.example:12345

Network closed for unknown reason

Client-side

io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason

The Problem

You are either (1) trying to connect to an grpc-server in TLS mode using a PLAINTEXT client or (2) the target is not a grpc-server (e.g. a web-server).

The Solution

  1. Configure your client to use TLS mode.

    grpc.client.__name__.negotiationType=TLS

    or remove the negotiationType config completely as TLS is the default.

  2. Validate that the configured server is running and is a grpc-server using grpcurl or a similar tool.

Could not find TLS ALPN provider

Server-side

org.springframework.context.ApplicationContextException: Failed to start bean 'nettyGrpcServerLifecycle'; nested exception is java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185) ~[spring-context-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    [...]
Caused by: java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
    at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:258) ~[grpc-netty-1.21.0.jar:1.21.0]
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171) ~[grpc-netty-1.21.0.jar:1.21.0]
    at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:130) ~[grpc-netty-1.21.0.jar:1.21.0]
    [...]

Client-side

[...]
Caused by: java.lang.IllegalStateException: Failed to create channel: <name>
    at net.devh.boot.grpc.client.inject.GrpcClientBeanPostProcessor.processInjectionPoint(GrpcClientBeanPostProcessor.java:118) ~[grpc-client-spring-boot-autoconfigure-2.4.0.RELEASE.jar:2.4.0.RELEASE]
    at net.devh.boot.grpc.client.inject.GrpcClientBeanPostProcessor.postProcessBeforeInitialization(GrpcClientBeanPostProcessor.java:77)
    [...]
Caused by: java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
    at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:258) ~[grpc-netty-1.21.0.jar:1.21.0]
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171) ~[grpc-netty-1.21.0.jar:1.21.0]
    at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120) ~[grpc-netty-1.21.0.jar:1.21.0]
    [...]

Both sides

AbstractMethodError: io.netty.internal.tcnative.SSL.readFromSSL()

The Problem

There is no (compatible) netty TLS implementation available on the classpath.

The Solution

Either switch from grpc-netty to grpc-netty-shaded or add a dependency to netty-tcnative-boringssl-static (Please use the exact same (compatible) versions that are listed in the table in grpc-java’s netty security section.

Note: You need a 64bit Java JVM.

Dismatching certificates

Client-side

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[...]
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
[...]
Caused by: java.security.cert.CertificateException: No subject alternative names present

or

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[...]
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
[...]
Caused by: java.security.cert.CertificateException: No name matching <name> found

The Problem

The certificate does not match the target’s address/name.

The Solution

Configure an override for the name comparison by adding the following to your client config:

grpc.client.__name__.security.authorityOverride=<authority>

Untrusted certificates

Client-side

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[...]
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
[...]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The Problem

The certificate used by the server is not in the trust store of the client.

The Solution

Either add the certificate to java’s truststore by using java’s keytool or configure the client to use a custom trusted certificate file:

grpc.client.__name__.security.trustCertCollection=file:certificates/trusted-servers-collection.crt.list

Note: Both stores are currently read only at creation time and updates won’t be picked up.

Server port already in use

Server-side

Caused by: java.lang.IllegalStateException: Failed to start the grpc server
    at net.devh.boot.grpc.server.serverfactory.GrpcServerLifecycle.start(GrpcServerLifecycle.java:51) ~[grpc-server-spring-boot-autoconfigure-2.4.0.RELEASE.jar:2.4.0.RELEASE]
    [...]
Caused by: java.io.IOException: Failed to bind
    at io.grpc.netty.shaded.io.grpc.netty.NettyServer.start(NettyServer.java:246) ~[grpc-netty-shaded-1.21.0.jar:1.21.0]
    at io.grpc.internal.ServerImpl.start(ServerImpl.java:177) ~[grpc-core-1.21.0.jar:1.21.0]
    at io.grpc.internal.ServerImpl.start(ServerImpl.java:85) ~[grpc-core-1.21.0.jar:1.21.0]
    at net.devh.boot.grpc.server.serverfactory.GrpcServerLifecycle.createAndStartGrpcServer(GrpcServerLifecycle.java:90) ~[grpc-server-spring-boot-autoconfigure-2.4.0.RELEASE.jar:2.4.0.RELEASE]
    at net.devh.boot.grpc.server.serverfactory.GrpcServerLifecycle.start(GrpcServerLifecycle.java:49) ~[grpc-server-spring-boot-autoconfigure-2.4.0.RELEASE.jar:2.4.0.RELEASE]
    ... 13 common frames omitted
Caused by: java.net.BindException: Address already in use: bind

The Problem

The port the grpc server is trying to use is already used.

There are four common cases where this error might occur.

  1. The application is already running
  2. Another application is using that port
  3. The grpc server uses a port that is already used for something else (e.g. spring-web)
  4. You are running tests and spring didn’t shutdown the grpc-server after each test

The Solution

  1. Try searching for the application using the task manager or jps
  2. Try searching for the port using netstat
  3. Check/Change your configuration. This library uses port 9090 by default
  4. Adding @DirtiesContext to your test classes and methods Please note that the error will only occur from the second test onwards, so you have to annotate the first one as well!

Client fails to resolve domain name

Client-side

WARN  io.grpc.internal.ManagedChannelImpl - [Failed to resolve name. status=Status{code=UNAVAILABLE, description=No servers found for `discovery-server:443`}
ERROR n.d.b.g.c.n.DiscoveryClientNameResolver - No servers found for `discovery-server:443`

The Problem

The discovery service library or it’s configuration failed to specify the scheme how discovery-server:443 should be resolved. If you don’t have a service discovery, then the default is dns, but if you use a discovery service, then that will be the default and thus failing to resolve that address.

The same applies to other libraries, such as tracing or reporting libraries, which report their results via grpc to an external server.

The Solution

See also client target configuration.

Creating issues

Creating issues/asking questions on GitHub isn’t hard, but with a little bit of your effort you can help us solving your issues faster.

If your issue/question is about grpc in general consider asking it over at grpc-java.

Use the provided templates to create new issues, these contain sections for the required/helpful information we need.

In general, you should include the following information in your issue:

  1. What type of request do you have?
    • Question
    • Bug-report
    • Feature-Request
  2. What do you wish to achieve?
  3. What’s The Problem? What’s not working? What’s missing and why do you need it?
  4. Any relevant stacktraces/logs (very important)
  5. Which versions do you use?
    • Spring (boot)
    • grpc-java
    • grpc-spring-boot-starter
    • Other relevant libraries
  6. Additional context
    • Did it ever work before?
    • How can we reproduce it?
    • Do you have a demo?

<- Back to Index