Rick

Rick
Rick

Thursday, June 11, 2015

Curlable stats and health checks... for QBit

You can wire in stats and get a single "ok" end point for your services. 

You can use AdminBuilder to create an admin utils.
        final AdminBuilder adminBuilder = AdminBuilder.adminBuilder();

        final ServiceEndpointServer adminServer =
                adminBuilder.build();

        adminServer.startServer();

        final HealthServiceAsync healthService = adminBuilder.getHealthService();
        healthService.register("foo", 1, TimeUnit.DAYS);
        healthService.checkInOk("foo");
        healthService.register("bar", 1, TimeUnit.DAYS);
        healthService.checkInOk("bar");
You can also register for service queue health checks.
    @Bean
    public AdminBuilder qbitAdminBuilder() {

        final int port = environment.getProperty("qbit.admin.server.port", Integer.class);
        final String host = environment.getProperty("qbit.admin.server.host", String.class);


        final AdminBuilder adminBuilder = AdminBuilder.adminBuilder()
                .setPort(port).setHost(host);

        return adminBuilder;
    }


    @Bean
    public ServiceEndpointServer adminServiceEndpointServer(
            final AdminBuilder adminBuilder) {
        final ServiceEndpointServer adminServer =
                adminBuilder.build();

        adminServer.startServer();
        return adminServer;

    }

....

        final Integer healthCheckTTL = env.getProperty("qbit.app.healthCheckTTLSeconds", Integer.class);
...

        final ServiceBuilder serviceBuilder = ServiceBuilder.serviceBuilder();

        final AdminBuilder qbitAdminBuilder = 
                 applicationContext.getBean("qbitAdminBuilder", AdminBuilder.class);
        final HealthServiceAsync healthServiceAsync = 
               qbitAdminBuilder.getHealthServiceBuilder().buildHealthSystemReporter();

        serviceBuilder.registerHealthChecksWithTTLInSeconds(qbitAdminBuilder.getHealthService(), longName,
                healthCheckTTL == null ? 5 : healthCheckTTL);

....
You can also register for stats health checks (stats system has a statsd replicator so you can replicate stats to statsD)
        final ServiceBuilder serviceBuilder = ServiceBuilder.serviceBuilder();
...
        final StatsCollector statsCollector = applicationContext.getBean("qbitStatsCollector", StatsCollector.class);

        serviceBuilder.registerStatsCollections(longName, statsCollector, flushTimeSeconds, sampleEvery );
Once you do this, then you can query for health status.
Returns true if all internal service queues are running. Returns false if any are failing.
Returns list of nodes healthy or not:
[
    "api.proxy.unpackerService",
    "api.proxy.healthService",
    "api.proxy.forwarderService",
    "api.proxy.bouncerService"
]
[
    "api.proxy.unpackerService",
    "api.proxy.healthService",
    "api.proxy.forwarderService",
    "api.proxy.bouncerService"
]
Return extended stats
[
    {
        "name": "api.proxy.unpackerService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    },
    {
        "name": "api.proxy.healthService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    },
    {
        "name": "api.proxy.forwarderService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    },
    {
        "name": "api.proxy.bouncerService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    }
]
Registering for health checks can be done with the service bundle builder and the service endpoint server builder as well:
        final ServiceBundleBuilder.serviceBundleBuilder = 
                ServiceBundleBuilder.serviceBundleBuilder().getRequestQueueBuilder();

        final ServiceBundle serviceBundle = serviceBundleBuilder
                .setStatsCollector(statsCollector)
                .build();

        serviceBundle.start();

        serviceBundle.addService(new MyService());
Every service added to the bundle will get stats support.

Wednesday, June 10, 2015

What is QBit?

For a quick overview check out these two slide decks QBit Early Slide Deck and QBit Java Microservices Lib. For support from the community see this qbit google group. There are also tons of documentation in the QBit WIKI. The wiki home page does a good job of covering QBit basics, and tries to touch on every area. If you are more looking for the high-level What is QBit trying to solve then look no further than Java Microservices architectureHigh Speed MicroservicesReactive Microservices, and Microservice Monitoring.
Here is a high-level bullet list style description. (After a few more paragraphs.. right to the bulleted list).
QBit looks like Spring MVC but is more like Akka with a lot of inspiration from Vert.x.
QBit core ideas were built when using Vert.x at large media company to be able to handle 100M users on fraction of servers (13 but could do it with 6 servers similar in scope to another app done by another vendor which used 2,000 servers. Similar Java EE app done by competing media company used 150 servers). For this project the service was capable of handling 2,000,000 CPU intensive requests per second on 10 servers. (Only able to test this service to 400K TPS across cluster due to F5 starting to implode, but was able to test single node at 150K TPS using cloud load tester that simulated clients from iOS, Android and Web. 150K was limitation of the license agreement of the cloud load tool.).
Ideas were expanded while using QBit to build OAuth based application id rate limiter.
Uses messaging, event busses, and queues. Similar in architecture to Vert.x/Akka. Not quite as complex as Akka, not quite as low level as Vert.x.
A service looks just like a Java class. It has methods. It has fields. The fields do not need to be synchronized because only one thread can access the service at a time. Services calls are sent in batches over a queue (internally). For CPU intensive services there are low level checks to see if the service is busy. If the service is busy, the batch sizes get larger, which means there is less overhead using low level thread-safe hand-off constructs, which means it runs faster. Batching also helps with remote services to maximize IO bandwidth.
If you are building high-speed in-memory services, QBit is an excellent option. If you are building async services, QBit is an excellent option. If you are building services that must work with streams of calls or message, then QBit is an excellent option. If you are building microservices and you want to implement back pressure detection, circuit breakers, coordinate calls to N number of other internal / external services, QBit is an excellent option.

Tier 1: Core Features

  1. Fast queue lib (200M messages a second)
  2. Fast service queue lib (internal queue to queue, typed actor like)
  3. Proxy to service queue programming model is Java idiomatic (smaller learning curve).
  4. Ability to handle queue states (queue empty, queue busy, queue limit, etc.)
  5. Reactor (call coordination, callbacks, timing out detection, async call workflow). Easy to use. Focused on services and handling calls.
  6. Workers Services can be round-robin workers (pooled, each member has its own state but stateless wrt to client session) or sharded workers (shard calls so that in-memory services can split the data into shards so that any single shard of in-memory data is only accessed by one thread).
Note: Tier 1 support is similar to Akka typed Actors. A service is a Java class that sits behind one or two queues (request/response queues).

Tier 2: Core Features

  1. EventService: Pub/Sub events (works with service queue lib to deliver events) (20M TPS)
  2. StatsService: collect stats (high-speed, can be clustered)
  3. HealthService: collects health status of service queues (new)
  4. ServiceDiscovery service (Consul or JSON file watcher)

Tier 2: IO: Http Web Server / Http Web Client:

  1. Simplified HTTP dev, async
  2. One liners for sending and receiving HTTP requests/responses
  3. HttpClientHttpServer
  4. Easy to use WebSocket lib for servers and clients
  5. Many utility function and builders for building complex REST calls (postGzipJson).
  6. Thread safe access to handlers just like QBit services

Tier 3: REST/HTTP

  1. Builds support for REST on top of Tier 1 and Tier 2
  2. Focused on JSON only responses and bodies (uses high-speed JSON parser)
  3. Supports subset of Spring MVC annotations @RequestMapping
  4. Easier smaller set. Main focus is on REST services with JSON.
  5. Faster than mainstream (4x faster than Spring/Tomcat) 96K TPS to 200K TPS
  6. Thread safe access to handlers just like QBit services

Tier 3: Remote call proxy

  1. Easy to use. Use Builders to build a proxy
  2. Remoting with WebSocket
  3. Programming model same as ServiceQueue
  4. Wire protocol is JSON and ASCII
  5. We use the fasted Java JVM parser
  6. Proxy similar to local proxy
  7. 4x to 5x faster than QBit REST/HTTP (should be 10x faster) 400K TPS (800K messages a second)
  8. Program to interfaces. Simple.
  9. A service can have a remote and local proxy interfaces

Admin Features:

  1. Metadata provider (creates large catalog of metadata about services used for REST mapping calls) (Replaced older hackier way)
  2. REST end point to see status of internal services (rudimentary http page as well)
  3. Auto-register health checks with services
  4. Auto register rudimentary stats with services (counts, sampling for timing, creation, queue size, etc. Periodic low over-head)

EventService

  1. Pub/Sub events (works with service queue lib to deliver events) (20M TPS)
  2. Easy to use api, register(), subscribe(), consume()
  3. Internal app messaging. Used by ServiceDiscovery for example.
  4. You could use to create workflows, or just to monitor an event
  5. Async.
  6. A Channel interface is used to implement the listener and to send messages. A Channel is just a plain java interface. It is easy to see who listens to the event from the IDE because all you have to do is do a find usage.
  7. You do not have to use the proxy channel. A channel name can be any String.
  8. EventService can be clustered! You can send events to all other nodes. You can use it to replicate calls or events to other nodes. The clustering using Consul.io to get a list of service members in a service group.
  9. By default, it is not clustered. You can easily integrate with any 3rd party messaging system. The EventService has an event replication mechanism to replicate event messages to other pipes (Kafka, JMS, RabbitMQ), etc.

StatsService

  1. Tracks counts, timings, and levels
  2. Can be clustered (with QBit ServiceDiscovery service and Console)
  3. Can be integrated with StatsD
  4. Can be used to implement things like high-speed rate limiting based on header
  5. Can be used to implement things like track system and implement back pressure cool off

ServiceDiscovery

  1. Keeps track of services
  2. Lets you find services
  3. Lets you know when services have been added or removed
  4. Removes services that are unhealthy
  5. Uses Consul.io, but can also just poll a JSON file for easy integration with Consul or etcd, (Chef push), etc.

HealthService

  1. Internal single node tracking (single node meaning process)
  2. Tracks N number of internal health nodes
  3. Services can auto-register with the health system.
  4. Uses TTL. It is a watchdog type service.

Monday, June 8, 2015

Perf QBit versus Spring Boot

First question people usually ask me. How does QBit compare to X?
Where X is the favorite framework of the person or something they once read an article about. Sometimes the questions are interesting. Sometimes people are comparing things that do not make sense.
Often times that X is Spring Boot.
So how does QBit compare, keep in mind, we have not done any serious perf tuning of QBit. We are more focused on features, and you could use QBit on a Spring project. I have used QBit running inside of Spring Boot. QBit is a lib. It can run anywhere. QBit can run with Guice. You can run QBit inside of Vert.x.
Out of the box QBit REST compares to out of the box Spring REST as follows:

QBit code

package hello;


import io.advantageous.qbit.annotation.RequestMapping;

import java.util.Collections;
import java.util.List;

import static io.advantageous.qbit.server.EndpointServerBuilder.endpointServerBuilder;

/**
 * Example of a QBit Service
 * <p>
 * created by rhightower on 2/2/15.
 */
@RequestMapping("/myservice")
public class MyServiceQBit {

    @RequestMapping("/ping")
    public List<String> ping() {
        return Collections.singletonList("Hello World!");
    }

    public static void main(String... args) throws Exception {
        endpointServerBuilder()
                .setPort(6060)
                .build()
                .initServices(new MyServiceQBit()).startServer();
    }

}

Spring code

package hello;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.Scope;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;

import java.util.Collections;
import java.util.List;

@RestController
@EnableAutoConfiguration
@Scope
public class MyServiceSpring {

    @RequestMapping(value = "/services/myservice/ping",
            produces = "application/json")
    @ResponseBody
    List<String> ping() {
        return Collections.singletonList("Hello World!");
    }

    public static void main(String[] args) throws Exception {

        SpringApplication.run(MyServiceSpring.class, args);
    }

}
The code looks similar. Keep in mind that QBit only does JSON over HTTP or WebSocket. Also QBit is focused on queuing and messaging not just HTTP (it has more similarities with Akka than Spring Boot). Spring supports a lot more types of content and options. Also keep in mind that QBit can happily run inside of Spring. QBit is just a library. Spring Boot is using Tomcat and Tomcat does things being Java EE that QBit will never do and does not need to do. 
But, people ask.
Now how do they compare when you are talking about performance.

Spring Boot

$  wrk -c 2000 -d 10s http://localhost:8080/services/myservice/ping
Running 10s test @ http://localhost:8080/services/myservice/ping
  2 threads and 2000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.03ms    2.12ms  31.37ms   67.26%
    Req/Sec    13.25k     1.61k   16.33k    65.50%
  264329 requests in 10.05s, 46.43MB read
  Socket errors: connect 0, read 150, write 0, timeout 0
Requests/sec:  26312.11
Transfer/sec:      4.62MB

Here we see that Spring Boot handles just over 26K TPS. This is just running it with wrk on OSX. You can expect higher speeds with a properly tuned Linux TCP/IP stack.

P.S. You can make Spring Boot 30% to 50% faster by using Jetty instead of Tomcat.

QBit RAW

$ wrk -c 2000 -d 10s http://localhost:6060/services/myservice/ping
Running 10s test @ http://localhost:6060/services/myservice/ping
  2 threads and 2000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.61ms   18.09ms 553.68ms   99.37%
    Req/Sec    48.78k     3.20k   52.86k    93.50%
  971148 requests in 10.02s, 80.58MB read
Requests/sec:  96910.31
Transfer/sec:      8.04MB


As you can see, QBit runs well over 3x faster. In fact it is approaching 4x faster. (Wait to see what happens when we start optimizing QBit.)
Now you are thinking but this is not a fair test. You are right. It is not fair to QBit. QBit has to funnel everything through the same thread. Spring does not.
QBit ensures that only one thread at a time runs the ping operation so you can put stateful things inside of the QBit services. You can't do that with the Spring Boot example. QBit is easier to program. Thus QBit has to do a lot more work, and it is still over 3x faster. The underlying network speed is coming form Vert.x which QBit uses as its network transport toolkit. (Thank you Mr. Fox.)
QBit also lets you easily run the same service as a WebSocket, and when you do that, you will get much higher throughput.
Performance breakdown:
  • QBit WebSocket speed 580K TPS (running on OSX macbook pro)
  • QBit HTTP speed 90K TPS (running on OSX macbook pro)
  • Spring Boot REST 26K TPS (running on OSX macbook pro)
Now this does not mean much. But I have built very real things with QBit and Vert.x. Things which handles millions and millions of users using a fraction of servers as other similar services which is how I feed my family and pay rent.
QBit is a lot more than this silly perf test. QBit has a health system, a stats system, integrates with Consul, integrates with StatsD, and much more. Here is a non-exhaustive list.
  • QBit implements a easy-to-use reactive API where you can easily manage async calls to N number of other systems.
  • QBit allows sharded in-memory services for CPU intensive needs.
  • QBit allows pooled/round-robin in-memory services for IO intensive needs.
  • QBits stats system can be clustered so you can do rate limiting across a pool of servers.
  • You can tweak QBit's queuing system to handle 200M+ TPS internal queuing.
  • QBit has a high-speed, type safe event bus which can connect to other systems or just run really fast in the same process.
It is no mistake that QBit annotations look a lot like Spring MVC annotations. I spent a lot of years using Spring and Spring MVC. I like it.
It also no mistake that you do not need Spring to run QBit. This allows QBit to be used in projects that are not using Spring (they do exist).

Think of QBit has a message based queuing system which can do a lot of really cool stuff with service discovery, events, reactive programming, etc., but does it in such a way that it will feel comfortable to Java developers who have used Spring or Java EE. 
Kafka and Cassandra support, training for AWS EC2 Cassandra 3.0 Training