Upgrade Netty dependency and fix Thread shutdown

[tgpfeiffer]

I am working on a Scala project that uses both Apache Spark (http://spark.apache.org/) and msgpack-rpc. Spark (transitively) depends on io.netty:netty:3.6.6.Final while msgpack-rpc depends on org.jboss.netty:netty:3.2.1.Final.

If I just include the dependencies to Spark and msgpack-rpc in my build.sbt file, then Spark will refuse starting with

java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker signature: (Ljava/util/concurrent/Executor;)Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong return type in function
at akka.remote.transport.netty.NettyTransport.(NettyTransport.scala:282)
at akka.remote.transport.netty.NettyTransport.(NettyTransport.scala:239)
...

So I edited my sbt file and excluded the transitive dependency on org.jboss.netty:netty from the msgpack-rpc dependency; that is, the (newer) netty version 3.6.6.Final is used. msgpack-rpc will still run fine, but has problems with shutdown: client.getEventLoop.shutdown() will not stop the Netty threads, a lot of "New I/O worker" and one "New I/O boss" thread will stay alive.

This is a problem, for instance, when using sbt run, because this will wait for all non-daemon threads to exit before exiting/returning to the sbt shell. This is never the case with the newer Netty version.

So I was wondering if it's possible to update the msgpack-rpc code to also work with a newer Netty version when it comes to thread shutdown. Or is it in fact a Netty bug?

[tgpfeiffer]

I examined this issue a bit further, and it seems like the following code is necessary to shut down all Netty threads properly when using netty 3.6.6:

client.close()
client.getEventLoop.shutdown()
client.getEventLoop.asInstanceOf[NettyEventLoop].getClientFactory.shutdown()

Just calling client.close() will leave us with

Thread[Hashed wheel timer #1,5,main] - Hashed wheel timer #1, 26, TIMED_WAITING, false
Thread[New I/O boss #17,5,main] - New I/O boss #17, 27, RUNNABLE, false
Thread[New I/O worker #1,5,main] - New I/O worker #1, 10, RUNNABLE, false
...
Thread[New I/O worker #16,5,main] - New I/O worker #16, 25, RUNNABLE, false
Thread[pool-3-thread-1,5,main] - pool-3-thread-1, 28, TIMED_WAITING, false

Calling just client.getEventLoop.shutdown() leads to remaining threads

Thread[Hashed wheel timer #1,5,main] - Hashed wheel timer #1, 26, TIMED_WAITING, false
Thread[New I/O boss #17,5,main] - New I/O boss #17, 27, RUNNABLE, false
Thread[New I/O worker #1,5,main] - New I/O worker #1, 10, RUNNABLE, false
...
Thread[New I/O worker #16,5,main] - New I/O worker #16, 25, RUNNABLE, false

(note that the "pool-3-thread-1" is gone) and just client.getEventLoop.asInstanceOf[NettyEventLoop].getClientFactory.shutdown() will result in

Thread[pool-1-thread-1,5,main] - pool-1-thread-1, 10, TIMED_WAITING, false
...
Thread[pool-1-thread-16,5,main] - pool-1-thread-16, 25, TIMED_WAITING, false

In all three cases, the program will not terminate when run from within sbt or an IDE. The combination of all three lines is required to exit cleanly.

However, the shutdown() method was added to Netty only in 3.6 (see netty/netty@7aa2cfa), it seems. Can we somehow incorporate calling this method, thereby also allowing to use newer versions of Netty?