We had a real-time API in Rails that needed much lower latency and massive throughput. We wanted to preserve our investment in business logic inside ActiveRecord models while scaling up to 1000X throughput and cutting latency in half. Conventional wisdom would say this was impossible in Ruby, but we succeeded and actually surpassed our expectations. We'll discuss how we did it, using Event-Machine for the reactor pattern, Synchrony to avoid callback hell and to make testing easy, Goliath as the non-blocking web server, and sharding across many cooperative processes.
 I'm Colin Kelly, CTO at Toastbouter, and I'm Dan Kulowski, I'm a software engineer at Nodoka. I've spent the last two years or so working on highly-performant APIs in Ruby. Typically, our phrases are in the same sense. But we've had an honor success using the lab without having to sacrifice the use of quoting that paragraph. So before we get to far into things, a little background on the API that we had to scale was necessary. It's called the Rinkable API. The Rinkable API allows you to track calls similarly to how you track a web click. You pass us a unique set of parameters and we'll give you a unique phone number back. And if that phone number gets called, it shows up in recording and allows you to make more intelligence use about where you're spending your marketing budget. Internally, a Rinkable is a collection of phone numbers. The customer can have many Rinkables, so you're usually attached to a particular marketing campaign. And each Rinkable will contain between two and five hundred numbers. To protect validity and call attribution, we reserve a phone number for a pretty set period of time after each application. This requires some architectural consideration because their time limit customer will have a number against a Rinkable that doesn't have any available. So all that's happening. So to ensure that we always have one to give out, each Rinkable has an add plus one number that we call the overflow. We'll give it out if you don't have any other numbers available. We don't track any call attribution data with it in some extreme situations. We've seen these overblown numbers handed out several hundred times a second. And so there's no way to correlate with an important call. So we were approached about two years ago by a client who wanted to use Rinkable API that had much higher throughput requirements than a rail stack was going to be able to support. Our existing API, the old rail stack, could support a connection of about five requests a second. We all went phone number four requests and the round trip 90%ile response time was about three or 15 milliseconds. The client requirements were a little bigger. They wanted to be handled on a panel at least a thousand requests per second for connection. They wanted at least 40 phone number allocations per request and they wanted a round trip total response time to 90%ile less than 200 milliseconds. So some quick back and forth math, that's an 8,000 full increase in throughput just for a single customer. So, well, the requirements were pretty clear. We needed a very predictable low latency and we needed to be a lot less scalable if we brought more users onto the network. Additionally, we wanted to leverage our existing API so we knew that the new system needed to be able to communicate with old systems as synchronously. We also knew that eventually we were going to want to share business logic, all the complex stuff we had over in our main app with this new API and other APIs somehow. And so bonus points if we could find a solution that was in Ruby. So, while Ruby brings a lot of awesome to the table, there's some considerations necessary when you start thinking about highly-performing applications. The main thing we were working about because we're on the Watson-Turbo-Rainford character was the global interpreter. For most dynamic languages, this is like the elephant in the room. It's worth noting that neither the JRuby nor the Ruby-ist interpreter is implementing it, and so they don't have this issue, but who is running Ruby-ist or JRuby? Okay, so it's still a problem for most of us. So, in short, the GIL is a wrapper placed around every Ruby statement and it prevents the process from consuming more than one CPU. It's nice because it simplifies the interpreter's design because the implementers don't have to think about all that writing. And it prevents a lot of data consistency issues that can pop up in highly-current code. The downside is that since we can only run one Ruby process per core, we can't write truly parallelized applications. In the ideal world, this is what your CPU uses would look like. You'd have one application running and it would even be spread its work across all the available CPUs on the machine. However, with the GIL, you end up totally pegging out one process and then just leaving the rest going absolutely nowhere. So the only way to get around this in the MRI, at least, is to run multiple block-related Ruby processes. So we sharded our API across the course. The second issue we needed to address was it's called the C10K problem. The question is, how can we best optimize our application to handle 10,000 synchronous incoming requests? We could write highly-threaded Ruby code, but that's really difficult to get right. Or we could avoid that complexity entirely by using the reactor pattern. The reactor pattern is one solution to the C10K problem we chose. The gist of it is that whenever we're writing code, it would block the main thread referred to as the reactor, like an HTTP request or a database insert. We toss on the bottom of the queue an asynchronous callback and pull the next action off the top of the queue and start working on it. The application then selects for events for which the block in IO is completed and runs the callback serial. This allows us an optimal level of concurrency without having to multiple-frame it. If you've seen Node.js, that's the reactor pattern in JavaScript, in Python, in the system library, and in Ruby, the reactor pattern library is a vet machine. Originally I heard VHH know that it's a keynote. This is an example of what a console or remote API would like in a vet machine. For any action we expect to block, in this case queries to quote Facebook and Forceware, we define a callback for when the request succeeds and an airback for when the request fails, one of which will be executed when the action needs to start unlocks. Enforcement in many cases, a callback and airback might be 80% similar. At Invoka, we have another API that runs on a machine, and these callbacks make the code really difficult to read, write, and test. Without very careful art touching, our code was becoming incredibly brittle and nearly un-colorable. You can end up with a complete mess of spaghetti code. If you've worked in Node.js, it's my favorite. Fortunately, because Ruby, there is a better way. Synchrony is a gem that sits on top of a vet machine, and it leverages the fibers introduced in Ruby 1.9 to make writing ace against code much easier. Instead of using callbacks, when a method blocks the reactor, it states store the fiber stack, and the reactor gets on to the next event, when the fiber unlocks. Sorry, unlocks. When the reactor unlocks, or when the method unlocks, the reactor assumes it from the stack. Synchrony handles all the complicated stuff in the background, and lets us program linearly without fear of being arbitrarily interrupted. This pattern of application scheduling using fibers also knows code routines. We'll be familiar for any of you who are working Go. This is a demo code Go routine. So, instead of this potential pile of smells and confusion, we end up with something that looks a lot more like the kind of code that we enjoy writing and testing. We wrap blocking code in fibers, and synchrony handles the rest. And our Goliath. Goliath is a Ruby web server that leverages a vet machine synchrony and the asynchronous, and handles each incoming request as a separate fiber. All asynchronous IO will transpan it with suspend, and later resume, without us having to do any of the heavy work. You define a response method that takes a single argument through which Goliath gives you all of their request. Do whatever work you need to do, and return the result to Goliath, which should go over the client's HTTP response. The last non-blanked out line here is the array that you can hand back to Goliath. Position zero is a status code in the space of 100. Position one is headers if you want to give back, in the space of nothing. And position two is the body of your request. So using a vet machine, synchrony and Goliath enabled us to build a really performance system. But the problem is going to all start the difference between threaded and reactant pattern Ruby code. And then talk so about the software design decisions we found to be very effective in synchronizing the code. Next slide. Congratulations, you see some code? That's my favorite thing. I go to that review. No. So where's that? There you go. We name all of our conferences. The physical was all glass, that's kind of a physical. So at the bottom level here, I can have a very simple web app that just counts. So here's the threaded version of the code. Then if you've written the threaded code, you're recognizing that you would have a new text to reactant. You've got a speed variable on the counter. Down here is the increment method. You do that inside the new text. You've got a special case for the one that you shut down. You shut this thing. Here's the counter itself when you want to retrieve it. You also grab the new text so you don't save copy of it. Here's my run method when I kick this thing off. All it does is create a print. Oh my god. And then I've got stop. Down here is increment forever. This is the actual part of the app. Here it just increments into where I have to stop. So there's a simple threaded version. Here it is invented. I basically deleted a quarter of the code. There's no need to mess around with new text because you're not going to be asynchronous at your own. So it's much simpler to read than that. All the difference down here is spider.new instead of thread.new. And this little piece actually kicks off the file. They give it two steps of files on one run. And so down here I do one other thing because this happens to be a CPU on application. I did need to periodically yield. This is fairly rare. You would have giant shops of CPU. The downside of writing cooperative code is you have to cooperate. So here I set it to 100,000 and then really I'm going to pause. And then here is the actual app itself. This is the little Goliath app. By the way, this is all up on GitHub. So here's the actual server in the AeroStrom Goliath API. It gets set up with an object. I've got one or two counters that we passed in. Do a little bit of square up here and then run the counter. And then this is the actual heart of the website. Every request comes in. These are the tabs and you are up. So on this slash counter, hands back this little bit of JSON. It also got is alive for your connected to an EOB or other load balancer. We have a way to go through live. And here's the 404 handle. It's that simple. There's also a new async scenario if you wanted to do one complex. A little bit of code down here at the bottom. Look at the command line to the side, which type of counter run. And then this is the boilerplate actually get the way I'm running. So let's fire this thing up. I've got... Let's see if this one... Let's see here. Okay. Use the command. Alright, so in one window here, I'm going to run the threaded one. And the other window... The other window here, I'm going to run the eventually one. So there's the venting, there's threaded. I'm going to give the threaded one a head start. And here's the eventually one. Alright, so I've got the log coming out to the screen here. Those are the live logs. So those things are running now. And down here is the window. I'm going to hit both of those. One's running at 4,000, and one's at 9,000 more. So they both started at zero. And the C in the speed, the eventually one is more than twice as fast. And that's because it's not that surrounded with mid-textions, and it's not being asynchronously interrupted. So also you might notice they happen to always be multiples of 100,000 in a single minute. That's exactly what you'd expect because it's cooperative multi-threaded. The only time it's going to have a chance to serve my slash counter action here is when the fiber goes in. Alright, so that concludes that demo. I think 2x is more than 2x here. I think 1.5x would be a more typical kind of number. You can see it on every lap. This is obviously kind of a generic, but significant difference in density. Okay. So I want to talk a little bit about some of the software design decisions that we made that really seemed to pay off in building this API system. To start with, Synchrony supports many popular Active Record stores, including MySQL. MySQL 2, try to go for that. Postgres, Mongo, Redis, Nincash, NDP, all of those. And there's plenty of third-party drivers as well, which are very visible patches to have a word with Synchrony. The biggest problem we had was dependencies. When we pulled in Active Record code like this, the auto loader in Rails kind of encourages or at least enables dependencies between models. In this example, my dependency in the user of the account is actually okay. Rails doesn't force the account to be loaded there. But as soon as I mentioned email address in user, it pulled in email in its class. And that class mentioned network in the program. You mentioned gen, it pulled the gen in. So we had to spend a fair amount of time sniffing dependencies. And we ultimately went with turning off the auto loader entirely in our live. We really wanted to give just the models that were actually necessary to be pulled in to our servers. You probably know Active Record is fairly slow. So for maximum density in your API server, CPU logs included loading objects. It's fairly slow. Validations in call max can add up. So TLDR used thin models. Models with few dependencies and few validations were the best fit. We also found we needed to encapsulate handling of exceptions. So we have a gen that we put together. This is open source on GitHub called exceptionalsynchronic. And it's just a layer that we added above EventScene and sacred. It has a couple shims, most important, rich. And that's exceptions across call max. I'm going to show that. We've also got proxies that sit in front of EventScene just to make the weapons a little safer. And parallel sync, where everyone helps with parallel. Each of those in turn. So here's the way we do call back exceptions. To start with, this is a straight back machine example. You've got a callback and an air back. And you may notice connection.close appears in both places. This is off the Pidcode and that machine. And that's typical for callbacks and air max. It's not an actual way to code. You wind up with some ingredient in the patient. Right there. It's not dry. One of the things that drew us to Ruby was that it has an automatic exception handle. We raise and rescue the silver and gold connections. We really found that to work great to keep your air handling separate from the new railway code. So here we can see that syncretty returned. Unified return path. This is the version that's using syncretty. We no longer have an air back. We can call back separate. The single result of being set back into that request and then the code is doing it else. This reminds me of coding in C. We kind of hope that your fellow developers might remember the check return codes. It's not a great way to build really wide code for your line hand. So inspired by Jason's syncretty code, we realized we could tunnel exceptions over the callback. By doing that, there's no need for a separate air back. So here's the insure callback method in this gem. It just calls that return exception and finds the method down there. It's very simple. It just takes the return value or the exception, whichever happens, you get that back. That's what we set across. We call it a success in all cases. It doesn't really matter because you're going to pop right out the other side where we have a way where it's fairly long to long to end here. We have to for a result and it takes those results and knows how to check them. And if they respond to his exception, it will raise them back. So those two put together, you get right back to straight linear angle really good. You can see here now what we have to do is insure that it can actually be exposed on the way up. Exactly what you decide. Well, for testing, we often found we could unit test these things. This is linear code that we find out the machinery involved in starting on the line. It was never true for our event machine projects. Unit testing was quite difficult. You always had to start reacting. Yeah. And speaking of exceptions, the exception gets loose. When the machine works into the fiber calls into your code, your process will exit. That is not any fun. We learned this the other day. We had some of our server processes get all over and get restarted by a process manager. It seemed to be random. We eventually got to the bottom of it and it was friendly file. Internal exceptions often for varying things. Final admission wasn't high or something. Something like that. And move up and shut the machine down. So after having been driven by that, we wound up with just a thin layer above all the event machine methods. They look like this. They just rescue and log errors and don't let them get any higher. These are basically the top of the stack. There's nobody above you at this point. You know the exception. All right. You might notice this is actually rescuing the exception, which you ordinarily wouldn't do, because that is the lowest level exception. You can rescue a script error if you have a typo. But you have always running APIs that are actually exactly the same. All right. So if that machine proxy wraps all these methods, basically we would call the top of the methods. All right. And the last piece is to talk about in this gem is parallel sync. The venture code is natural for writing in parallel, but it makes it hard to write in serial code. Covertive libraries like Syncrede reverse that. Serial code is easy to write. It's a trivial effect. But when you want to run code in parallel, you need to take some explicit steps. So here's the heart of our API scanner guide across the charts. You'll see that it says responses equals and then it kicks off this parallel block. And that adds all the work to be done to talk to the other charts. And then when you hit the end of that block where the time is there, all of the results are going to run into there. And it's available on the analysis array to be used on the next statement. This really should be about future processes pattern. We're working on that. I'm going to try to merge some other gems in to get back to where you're sitting. I also want to call attention to the last line there. It's a plain old Ruby object. We use those in specific. I'm going to talk about them a little bit. So we call them a mutable value classes. Plain old Ruby objects that you can use instead of a simple string or hash typically. We found this is great. I'm inspired by second net. You'll see here we have an initializer that sets the object up to just case it holds the array of responses and also the format. You'll notice it's designed by contracting force here that we don't allow formats to set those to. In fact, these days we don't even allow it to set them up. I left it in there just to show the point that it's a great pinch point for contracting force. There's no accesses or right because this is read only. That's the indivisible part. If you want to change it, you change a cop. That's the convention. These classes typically don't serialize themselves. Some of them, we had a previous talk. So in this case, the JSON knows how to serialize itself as an array of responses. And we've got a matching method from JSON, which is a factory method there. We found this pattern and we're back in range. So these things are all unit tested as individual pieces that are opposable. This has been a hash, which is the typical way this kind of code is written. Yeah, hash does have two JSONs, but it turns out that it almost never has exactly what you want in JSON. As soon as you have some internal bookkeeping that's different than what you want to share with your external clients, a hash would not work. And so it turned out the bulk of our code wound up being in these immutable values. So that was pretty neat to see it go like that. So these were Sandy Bess's rules. We actually read about them after because it's probably the first time. But they nicely point out, direct you toward plain overview objects. And I particularly recommend the immutable stuff. Yeah, this required a great functional programming and it was also perfect for concurrency because Ruby is a shared reference language. We'll have many requests coming in at the same time. They were mutating infrastructures even though with fibers they won't have to have any synchronous. You still have to worry about thinking you can do good blocks when you come back to the data structure and be different. And simplest answers don't change. So I would be very, very, very happy if there were a functional Ruby out that flydot made that for Ruby. And we found that by using these classes we naturally decompose into much smaller pieces of code and avoiding big models. The last piece I want to talk about, briefly, is symptoms. I don't mean that I need class-style symptoms and I mean you both need various symptoms. TLDR is their global variables. You can't control the lifetime, you can't pass the structure parameters, you can't use them, you can't use an intestine. And symptoms we get for citizens. They're hydrants. You start into those, you get more and more of them. So we have a modified symptom pattern that's great for us. This is an example class that is what keeps our secrets, credentials, we use to talk to other third party people and such. It's similar, it's got initialized method, but you notice it's actually allowed to take a constructor parameter. It's making its dependencies explicit. Turns out after we wrote the code this way about a month ago, we actually had a case that needed two secrets files. And this code were created, where the symptom pattern would have been dead in one, a Syngedemon 2. Down here you'll see there's a method that you set in a particular instance to be the global one and a class method to get that. So you still have the convenience of getting at things without having to pass the proof you needed, but you also have all the flexibility and address all the shortcomings that were stated. Okay, back to Dan. So we also learned some important lessons about architecting large concurrences in the cloud, the first of which was about TCP latency. So this diagram shows the handshake that's performed each time a TCP connection is established. It's performed and determined the bandwidth of the connection, but if you can see the x-axis, this handshake can take about 20 milliseconds. On the left side here is HTTP 1.0. Each time your client wants to send a request, they open a new HTTP connection, perform the handshake, send a single response, I'm sorry, send a single request, get a single response, and then close the connection. For instance, repeat. Given our SLA, losing 20 to 30 milliseconds to a handshake on each request was highly undesirable. Unfortunately, HTTP 1.1 introduced connection persistence. Subsequent HTTP requests are pipelined across the initial TCP connection. The slow start handshake is performed once, and the connection is kept open for as long as it's needed. So we used HTTP 1.1 versus the same connection to communicate between shards, and further optimize communication between shards and the same server by using domain sockets instead of TCP connections. One issue that we did run into was what ELB is, Amazon's last load balancers. They only forward request a single port for server and rotation. So, we had to put in AHA proxies on each server, on a PGAI server, in order to round robin request the individual shard process running on that server. This is what the setup ended up looking like in EC2. The client and ELB, which round robins to the available AHA proxies, which in turn round robin to local API shards. The receiving shards send requests to other shards, where they pool what they don't own, and then they receive the initial receiving shard, consolidates the responses into a single JSON body and returns that to the client. Rain pools are assigned round robin to API shards to balance load. The system also allows us to move more heavily traffic rain pools from one shard to a second, experiencing a lower average load. So, what happens when a shard falls over or doesn't respond in time? Initially, if we implemented what we call the watchshard, it would monitor the API shards and react quickly to reassign rain pools to one shard stop responding. However, we didn't like the complexity of that design, and we knew who's going to watch the watchshards. So instead, we taught each shard the overflow number for every rain pool in the system. So if a shard doesn't respond to the application request, we just hand back the overflow number. We also rate limits communication between shards. If one shard notices another shard is taking too long to process a request or responding, we hand back and start handing back overflow numbers instead of actually hitting the slow shard. Over time, the shard that noticed one shard was being slow starts to allow more and more requests through and start handing back overflows all the time. So, we took these measurements from EC2 using Jmeter, which ended up being a fantastic tool. With three Jmeter instances slamming our API, we had about a 93 millisecond median, 90th percentile, 124 milliseconds, and we were able to process 1,400 requests per second. With four Jmeter instances slamming the crap out of our API, we had a median of about 102 milliseconds and a 98 percentile of 34 milliseconds, and we were able to process 1,700 requests per second. With five, we were able to process 2,000 requests per second. We found, awesomely enough, and as it shows, the bottleneck wasn't the number of requests that Jmeter was able to hit us with, not our API. We later found that we could have used aws as you can see, to ensure that data goes through as long as private backbones, as opposed to going over the public internet, and that would have cut the additional 10 to 15 milliseconds off the Patreon trip. So, the final result exceeded the SLA with a fair amount of effort. So, from the fact, there are 5.5 billion thumb members in North America. At this rate, we could allocate every North American thumb number less than 12 hours. So, it was a tremendous success. Very, very, very easy results. So, tools, some other tools we used. We used MiniTest and FactoryGirl, our entire testing, going on less than 10 seconds. We used simple coverage to make sure we were working lazy, and that's 10 seconds with a 99.96 percent coverage ratio. And again, as mentioned, a Patreon Jmeter for load testing was awesome. You can run it headlessly up in the cloud, very easy to consolidate results, very easy to create very many complex testing sets for it, and we used a lot of the same tools with clients who were also load testing again. Sorry. We got a couple of shout-outs. We'd like to really thank the maintainers of the Vap Machine and Vap Machine Synchrony. I personally would like a shout-out to the IRC channel, Ruby, and Ruby Lang on Freeman. We'd like to thank Google Labs and Billaric Grugor, who are the maintainers of the MN Synchrony, respectively. And a shout-out to Sandy Metz for a lot of the best practices that we've made this project possible. Thank you. I've got a call to action here. I strongly encourage you to try GoLab as a stack for your mixed API server. If you have any questions, you can put it on Twitter. We'd love to talk about it. I think it's a really great stack, and I'd love to see a version in the future of Rails itself. I know Ilya Grigory was proposing that idea in 2011. It didn't quite happen, but I think it should have. And now that we heard DHH mention that machine as being part of Rails 5, I would love to see Synchrony as well. So I'm going to see if I can come up with that project and try to steer it that way. But it would be a great stack together with a fin, for instance, because I already built it on that machine. Ruby gets a lot more density than take a little bit of the momentum away from no-jagets, which is really cool. When it comes to density. Alright, so that's the end. Thank you.
 I'm Colin Kelly, CTO at Toastbouter, and I'm Dan Kulowski, I'm a software engineer at Nodoka.  I've spent the last two years or so working on highly-performant APIs in Ruby.  Typically, our phrases are in the same sense.  But we've had an honor success using the lab without having to sacrifice the use of quoting that paragraph.  So before we get to far into things, a little background on the API that we had to scale was necessary.  It's called the Rinkable API.  The Rinkable API allows you to track calls similarly to how you track a web click.  You pass us a unique set of parameters and we'll give you a unique phone number back.  And if that phone number gets called, it shows up in recording and allows you to make more intelligence use about where you're spending your marketing budget.  Internally, a Rinkable is a collection of phone numbers.  The customer can have many Rinkables, so you're usually attached to a particular marketing campaign.  And each Rinkable will contain between two and five hundred numbers.  To protect validity and call attribution, we reserve a phone number for a pretty set period of time after each application.  This requires some architectural consideration because their time limit customer will have a number against a Rinkable that doesn't have any available.  So all that's happening.  So to ensure that we always have one to give out, each Rinkable has an add plus one number that we call the overflow.  We'll give it out if you don't have any other numbers available.  We don't track any call attribution data with it in some extreme situations. We've seen these overblown numbers handed out several hundred times a second.  And so there's no way to correlate with an important call.  So we were approached about two years ago by a client who wanted to use Rinkable API that had much higher throughput requirements than a rail stack was going to be able to support.  Our existing API, the old rail stack, could support a connection of about five requests a second.  We all went phone number four requests and the round trip 90%ile response time was about three or 15 milliseconds.  The client requirements were a little bigger.  They wanted to be handled on a panel at least a thousand requests per second for connection.  They wanted at least 40 phone number allocations per request and they wanted a round trip total response time to 90%ile less than 200 milliseconds.  So some quick back and forth math, that's an 8,000 full increase in throughput just for a single customer.  So, well, the requirements were pretty clear.  We needed a very predictable low latency and we needed to be a lot less scalable if we brought more users onto the network.  Additionally, we wanted to leverage our existing API so we knew that the new system needed to be able to communicate with old systems as synchronously.  We also knew that eventually we were going to want to share business logic, all the complex stuff we had over in our main app with this new API and other APIs somehow.  And so bonus points if we could find a solution that was in Ruby.  So, while Ruby brings a lot of awesome to the table, there's some considerations necessary when you start thinking about highly-performing applications.  The main thing we were working about because we're on the Watson-Turbo-Rainford character was the global interpreter.  For most dynamic languages, this is like the elephant in the room.  It's worth noting that neither the JRuby nor the Ruby-ist interpreter is implementing it, and so they don't have this issue, but who is running Ruby-ist or JRuby?  Okay, so it's still a problem for most of us.  So, in short, the GIL is a wrapper placed around every Ruby statement and it prevents the process from consuming more than one CPU.  It's nice because it simplifies the interpreter's design because the implementers don't have to think about all that writing.  And it prevents a lot of data consistency issues that can pop up in highly-current code.  The downside is that since we can only run one Ruby process per core, we can't write truly parallelized applications.  In the ideal world, this is what your CPU uses would look like.  You'd have one application running and it would even be spread its work across all the available CPUs on the machine.  However, with the GIL, you end up totally pegging out one process and then just leaving the rest going absolutely nowhere.  So the only way to get around this in the MRI, at least, is to run multiple block-related Ruby processes.  So we sharded our API across the course.  The second issue we needed to address was it's called the C10K problem.  The question is, how can we best optimize our application to handle 10,000 synchronous incoming requests?  We could write highly-threaded Ruby code, but that's really difficult to get right.  Or we could avoid that complexity entirely by using the reactor pattern.  The reactor pattern is one solution to the C10K problem we chose.  The gist of it is that whenever we're writing code, it would block the main thread referred to as the reactor,  like an HTTP request or a database insert.  We toss on the bottom of the queue an asynchronous callback and pull the next action off the top of the queue and start working on it.  The application then selects for events for which the block in IO is completed and runs the callback serial.  This allows us an optimal level of concurrency without having to multiple-frame it.  If you've seen Node.js, that's the reactor pattern in JavaScript, in Python, in the system library,  and in Ruby, the reactor pattern library is a vet machine.  Originally I heard VHH know that it's a keynote.  This is an example of what a console or remote API would like in a vet machine.  For any action we expect to block, in this case queries to quote Facebook and Forceware,  we define a callback for when the request succeeds and an airback for when the request fails,  one of which will be executed when the action needs to start unlocks.  Enforcement in many cases, a callback and airback might be 80% similar.  At Invoka, we have another API that runs on a machine,  and these callbacks make the code really difficult to read, write, and test.  Without very careful art touching, our code was becoming incredibly brittle and nearly un-colorable.  You can end up with a complete mess of spaghetti code.  If you've worked in Node.js, it's my favorite.  Fortunately, because Ruby, there is a better way.  Synchrony is a gem that sits on top of a vet machine,  and it leverages the fibers introduced in Ruby 1.9 to make writing ace against code much easier.  Instead of using callbacks, when a method blocks the reactor, it states store the fiber stack,  and the reactor gets on to the next event, when the fiber unlocks.  Sorry, unlocks.  When the reactor unlocks, or when the method unlocks, the reactor assumes it from the stack.  Synchrony handles all the complicated stuff in the background,  and lets us program linearly without fear of being arbitrarily interrupted.  This pattern of application scheduling using fibers also knows code routines.  We'll be familiar for any of you who are working Go.  This is a demo code Go routine.  So, instead of this potential pile of smells and confusion,  we end up with something that looks a lot more like the kind of code that we enjoy writing and testing.  We wrap blocking code in fibers, and synchrony handles the rest.  And our Goliath.  Goliath is a Ruby web server that leverages a vet machine synchrony and the asynchronous,  and handles each incoming request as a separate fiber.  All asynchronous IO will transpan it with suspend, and later resume,  without us having to do any of the heavy work.  You define a response method that takes a single argument through which Goliath gives you all of their request.  Do whatever work you need to do, and return the result to Goliath,  which should go over the client's HTTP response.  The last non-blanked out line here is the array that you can hand back to Goliath.  Position zero is a status code in the space of 100.  Position one is headers if you want to give back, in the space of nothing.  And position two is the body of your request.  So using a vet machine, synchrony and Goliath enabled us to build a really performance system.  But the problem is going to all start the difference between threaded and reactant pattern Ruby code.  And then talk so about the software design decisions we found to be very effective in synchronizing the code.  Next slide.  Congratulations, you see some code?  That's my favorite thing.  I go to that review.  No.  So where's that?  There you go.  We name all of our conferences.  The physical was all glass, that's kind of a physical.  So at the bottom level here, I can have a very simple web app that just counts.  So here's the threaded version of the code.  Then if you've written the threaded code, you're recognizing that you would have a new text to reactant.  You've got a speed variable on the counter.  Down here is the increment method.  You do that inside the new text.  You've got a special case for the one that you shut down.  You shut this thing.  Here's the counter itself when you want to retrieve it.  You also grab the new text so you don't save copy of it.  Here's my run method when I kick this thing off.  All it does is create a print.  Oh my god.  And then I've got stop.  Down here is increment forever.  This is the actual part of the app.  Here it just increments into where I have to stop.  So there's a simple threaded version.  Here it is invented.  I basically deleted a quarter of the code.  There's no need to mess around with new text because you're not going to be asynchronous at your own.  So it's much simpler to read than that.  All the difference down here is spider.new instead of thread.new.  And this little piece actually kicks off the file.  They give it two steps of files on one run.  And so down here I do one other thing because this happens to be a CPU on application.  I did need to periodically yield.  This is fairly rare.  You would have giant shops of CPU.  The downside of writing cooperative code is you have to cooperate.  So here I set it to 100,000 and then really I'm going to pause.  And then here is the actual app itself.  This is the little Goliath app.  By the way, this is all up on GitHub.  So here's the actual server in the AeroStrom Goliath API.  It gets set up with an object.  I've got one or two counters that we passed in.  Do a little bit of square up here and then run the counter.  And then this is the actual heart of the website.  Every request comes in.  These are the tabs and you are up.  So on this slash counter, hands back this little bit of JSON.  It also got is alive for your connected to an EOB or other load balancer.  We have a way to go through live.  And here's the 404 handle.  It's that simple.  There's also a new async scenario if you wanted to do one complex.  A little bit of code down here at the bottom.  Look at the command line to the side, which type of counter run.  And then this is the boilerplate actually get the way I'm running.  So let's fire this thing up.  I've got...  Let's see if this one...  Let's see here.  Okay.  Use the command.  Alright, so in one window here, I'm going to run the threaded one.  And the other window...  The other window here, I'm going to run the eventually one.  So there's the venting, there's threaded.  I'm going to give the threaded one a head start.  And here's the eventually one.  Alright, so I've got the log coming out to the screen here.  Those are the live logs.  So those things are running now.  And down here is the window.  I'm going to hit both of those. One's running at 4,000, and one's at 9,000 more.  So they both started at zero.  And the C in the speed, the eventually one is more than twice as fast.  And that's because it's not that surrounded with mid-textions,  and it's not being asynchronously interrupted.  So also you might notice they happen to always be multiples of 100,000 in a single minute.  That's exactly what you'd expect because it's cooperative multi-threaded.  The only time it's going to have a chance to serve my slash counter action here is when the fiber goes in.  Alright, so that concludes that demo.  I think 2x is more than 2x here.  I think 1.5x would be a more typical kind of number.  You can see it on every lap.  This is obviously kind of a generic, but significant difference in density.  Okay.  So I want to talk a little bit about some of the software design decisions that we made that really seemed to pay off in building this API system.  To start with, Synchrony supports many popular Active Record stores, including MySQL.  MySQL 2, try to go for that.  Postgres, Mongo, Redis, Nincash, NDP, all of those.  And there's plenty of third-party drivers as well, which are very visible patches to have a word with Synchrony.  The biggest problem we had was dependencies.  When we pulled in Active Record code like this, the auto loader in Rails kind of encourages or at least enables dependencies between models.  In this example, my dependency in the user of the account is actually okay.  Rails doesn't force the account to be loaded there.  But as soon as I mentioned email address in user, it pulled in email in its class.  And that class mentioned network in the program.  You mentioned gen, it pulled the gen in.  So we had to spend a fair amount of time sniffing dependencies.  And we ultimately went with turning off the auto loader entirely in our live.  We really wanted to give just the models that were actually necessary to be pulled in to our servers.  You probably know Active Record is fairly slow.  So for maximum density in your API server, CPU logs included loading objects.  It's fairly slow.  Validations in call max can add up.  So TLDR used thin models.  Models with few dependencies and few validations were the best fit.  We also found we needed to encapsulate handling of exceptions.  So we have a gen that we put together. This is open source on GitHub called exceptionalsynchronic.  And it's just a layer that we added above EventScene and sacred.  It has a couple shims, most important, rich.  And that's exceptions across call max.  I'm going to show that.  We've also got proxies that sit in front of EventScene just to make the weapons a little safer.  And parallel sync, where everyone helps with parallel.  Each of those in turn.  So here's the way we do call back exceptions.  To start with, this is a straight back machine example.  You've got a callback and an air back.  And you may notice connection.close appears in both places.  This is off the Pidcode and that machine.  And that's typical for callbacks and air max.  It's not an actual way to code.  You wind up with some ingredient in the patient.  Right there.  It's not dry.  One of the things that drew us to Ruby was that it has an automatic exception handle.  We raise and rescue the silver and gold connections.  We really found that to work great to keep your air handling separate from the new railway code.  So here we can see that syncretty returned.  Unified return path.  This is the version that's using syncretty.  We no longer have an air back.  We can call back separate.  The single result of being set back into that request and then the code is doing it else.  This reminds me of coding in C.  We kind of hope that your fellow developers might remember the check return codes.  It's not a great way to build really wide code for your line hand.  So inspired by Jason's syncretty code, we realized we could tunnel exceptions over the callback.  By doing that, there's no need for a separate air back.  So here's the insure callback method in this gem.  It just calls that return exception and finds the method down there.  It's very simple.  It just takes the return value or the exception, whichever happens, you get that back.  That's what we set across.  We call it a success in all cases.  It doesn't really matter because you're going to pop right out the other side where we have a way where it's fairly long to long to end here.  We have to for a result and it takes those results and knows how to check them.  And if they respond to his exception, it will raise them back.  So those two put together, you get right back to straight linear angle really good.  You can see here now what we have to do is insure that it can actually be exposed on the way up.  Exactly what you decide.  Well, for testing, we often found we could unit test these things.  This is linear code that we find out the machinery involved in starting on the line.  It was never true for our event machine projects.  Unit testing was quite difficult.  You always had to start reacting.  Yeah.  And speaking of exceptions, the exception gets loose.  When the machine works into the fiber calls into your code, your process will exit.  That is not any fun.  We learned this the other day.  We had some of our server processes get all over and get restarted by a process manager.  It seemed to be random.  We eventually got to the bottom of it and it was friendly file.  Internal exceptions often for varying things.  Final admission wasn't high or something.  Something like that.  And move up and shut the machine down.  So after having been driven by that, we wound up with just a thin layer above all the event machine methods.  They look like this.  They just rescue and log errors and don't let them get any higher.  These are basically the top of the stack.  There's nobody above you at this point.  You know the exception.  All right. You might notice this is actually rescuing the exception, which you ordinarily wouldn't do,  because that is the lowest level exception.  You can rescue a script error if you have a typo.  But you have always running APIs that are actually exactly the same.  All right.  So if that machine proxy wraps all these methods,  basically we would call the top of the methods.  All right. And the last piece is to talk about in this gem is parallel sync.  The venture code is natural for writing in parallel,  but it makes it hard to write in serial code.  Covertive libraries like Syncrede reverse that.  Serial code is easy to write. It's a trivial effect.  But when you want to run code in parallel, you need to take some explicit steps.  So here's the heart of our API scanner guide across the charts.  You'll see that it says responses equals and then it kicks off this parallel block.  And that adds all the work to be done to talk to the other charts.  And then when you hit the end of that block where the time is there,  all of the results are going to run into there.  And it's available on the analysis array to be used on the next statement.  This really should be about future processes pattern.  We're working on that.  I'm going to try to merge some other gems in to get back to where you're sitting.  I also want to call attention to the last line there.  It's a plain old Ruby object.  We use those in specific. I'm going to talk about them a little bit.  So we call them a mutable value classes.  Plain old Ruby objects that you can use instead of a simple string or hash typically.  We found this is great. I'm inspired by second net.  You'll see here we have an initializer that sets the object up to just case it holds the array of responses and also the format.  You'll notice it's designed by contracting force here that we don't allow formats to set those to.  In fact, these days we don't even allow it to set them up.  I left it in there just to show the point that it's a great pinch point for contracting force.  There's no accesses or right because this is read only.  That's the indivisible part.  If you want to change it, you change a cop. That's the convention.  These classes typically don't serialize themselves.  Some of them, we had a previous talk.  So in this case, the JSON knows how to serialize itself as an array of responses.  And we've got a matching method from JSON, which is a factory method there.  We found this pattern and we're back in range.  So these things are all unit tested as individual pieces that are opposable.  This has been a hash, which is the typical way this kind of code is written.  Yeah, hash does have two JSONs, but it turns out that it almost never has exactly what you want in JSON.  As soon as you have some internal bookkeeping that's different than what you want to share with your external clients, a hash would not work.  And so it turned out the bulk of our code wound up being in these immutable values.  So that was pretty neat to see it go like that.  So these were Sandy Bess's rules.  We actually read about them after because it's probably the first time.  But they nicely point out, direct you toward plain overview objects.  And I particularly recommend the immutable stuff.  Yeah, this required a great functional programming and it was also perfect for concurrency because Ruby is a shared reference language.  We'll have many requests coming in at the same time.  They were mutating infrastructures even though with fibers they won't have to have any synchronous.  You still have to worry about thinking you can do good blocks when you come back to the data structure and be different.  And simplest answers don't change.  So I would be very, very, very happy if there were a functional Ruby out that flydot made that for Ruby.  And we found that by using these classes we naturally decompose into much smaller pieces of code and avoiding big models.  The last piece I want to talk about, briefly, is symptoms.  I don't mean that I need class-style symptoms and I mean you both need various symptoms.  TLDR is their global variables.  You can't control the lifetime, you can't pass the structure parameters, you can't use them, you can't use an intestine.  And symptoms we get for citizens.  They're hydrants.  You start into those, you get more and more of them.  So we have a modified symptom pattern that's great for us.  This is an example class that is what keeps our secrets, credentials, we use to talk to other third party people and such.  It's similar, it's got initialized method, but you notice it's actually allowed to take a constructor parameter.  It's making its dependencies explicit.  Turns out after we wrote the code this way about a month ago, we actually had a case that needed two secrets files.  And this code were created, where the symptom pattern would have been dead in one, a Syngedemon 2.  Down here you'll see there's a method that you set in a particular instance to be the global one and a class method to get that.  So you still have the convenience of getting at things without having to pass the proof you needed,  but you also have all the flexibility and address all the shortcomings that were stated.  Okay, back to Dan.  So we also learned some important lessons about architecting large concurrences in the cloud,  the first of which was about TCP latency.  So this diagram shows the handshake that's performed each time a TCP connection is established.  It's performed and determined the bandwidth of the connection, but if you can see the x-axis,  this handshake can take about 20 milliseconds.  On the left side here is HTTP 1.0. Each time your client wants to send a request,  they open a new HTTP connection, perform the handshake, send a single response,  I'm sorry, send a single request, get a single response, and then close the connection.  For instance, repeat.  Given our SLA, losing 20 to 30 milliseconds to a handshake on each request was highly undesirable.  Unfortunately, HTTP 1.1 introduced connection persistence.  Subsequent HTTP requests are pipelined across the initial TCP connection.  The slow start handshake is performed once, and the connection is kept open for as long as it's needed.  So we used HTTP 1.1 versus the same connection to communicate between shards,  and further optimize communication between shards and the same server  by using domain sockets instead of TCP connections.  One issue that we did run into was what ELB is, Amazon's last load balancers.  They only forward request a single port for server and rotation.  So, we had to put in AHA proxies on each server, on a PGAI server,  in order to round robin request the individual shard process running on that server.  This is what the setup ended up looking like in EC2.  The client and ELB, which round robins to the available AHA proxies,  which in turn round robin to local API shards.  The receiving shards send requests to other shards,  where they pool what they don't own,  and then they receive the initial receiving shard, consolidates the responses  into a single JSON body and returns that to the client.  Rain pools are assigned round robin to API shards to balance load.  The system also allows us to move more heavily traffic rain pools  from one shard to a second, experiencing a lower average load.  So, what happens when a shard falls over or doesn't respond in time?  Initially, if we implemented what we call the watchshard,  it would monitor the API shards and react quickly to reassign rain pools  to one shard stop responding.  However, we didn't like the complexity of that design,  and we knew who's going to watch the watchshards.  So instead, we taught each shard the overflow number for every rain pool in the system.  So if a shard doesn't respond to the application request,  we just hand back the overflow number.  We also rate limits communication between shards.  If one shard notices another shard is taking too long to process a request or responding,  we hand back and start handing back overflow numbers instead  of actually hitting the slow shard.  Over time, the shard that noticed one shard was being slow  starts to allow more and more requests through and start handing back overflows all the time.  So, we took these measurements from EC2 using Jmeter,  which ended up being a fantastic tool.  With three Jmeter instances slamming our API,  we had about a 93 millisecond median,  90th percentile, 124 milliseconds, and we were able to process 1,400 requests per second.  With four Jmeter instances slamming the crap out of our API,  we had a median of about 102 milliseconds and a 98 percentile of 34 milliseconds,  and we were able to process 1,700 requests per second.  With five, we were able to process 2,000 requests per second.  We found, awesomely enough, and as it shows, the bottleneck wasn't the number of requests  that Jmeter was able to hit us with, not our API.  We later found that we could have used aws as you can see,  to ensure that data goes through as long as private backbones,  as opposed to going over the public internet,  and that would have cut the additional 10 to 15 milliseconds off the Patreon trip.  So, the final result exceeded the SLA with a fair amount of effort.  So, from the fact, there are 5.5 billion thumb members in North America.  At this rate, we could allocate every North American thumb number less than 12 hours.  So, it was a tremendous success.  Very, very, very easy results.  So, tools, some other tools we used.  We used MiniTest and FactoryGirl, our entire testing,  going on less than 10 seconds.  We used simple coverage to make sure we were working lazy,  and that's 10 seconds with a 99.96 percent coverage ratio.  And again, as mentioned, a Patreon Jmeter for load testing was awesome.  You can run it headlessly up in the cloud, very easy to consolidate results,  very easy to create very many complex testing sets for it,  and we used a lot of the same tools with clients who were also load testing again.  Sorry.  We got a couple of shout-outs.  We'd like to really thank the maintainers of the Vap Machine and Vap Machine Synchrony.  I personally would like a shout-out to the IRC channel, Ruby, and Ruby Lang on Freeman.  We'd like to thank Google Labs and Billaric Grugor, who are the maintainers of the MN Synchrony, respectively.  And a shout-out to Sandy Metz for a lot of the best practices that we've made this project possible.  Thank you.  I've got a call to action here.  I strongly encourage you to try GoLab as a stack for your mixed API server.  If you have any questions, you can put it on Twitter.  We'd love to talk about it.  I think it's a really great stack, and I'd love to see a version in the future of Rails itself.  I know Ilya Grigory was proposing that idea in 2011.  It didn't quite happen, but I think it should have.  And now that we heard DHH mention that machine as being part of Rails 5, I would love to see Synchrony as well.  So I'm going to see if I can come up with that project and try to steer it that way.  But it would be a great stack together with a fin, for instance, because I already built it on that machine.  Ruby gets a lot more density than take a little bit of the momentum away from no-jagets, which is really cool.  When it comes to density.  Alright, so that's the end.  Thank you.
High_performance_APIs_48_155.png High_performance_APIs_300_2771.png High_performance_APIs_2772_2963.png High_performance_APIs_2964_3755.png High_performance_APIs_3756_6023.png High_performance_APIs_6024_6287.png High_performance_APIs_6288_8927.png High_performance_APIs_8928_9095.png High_performance_APIs_9096_10619.png High_performance_APIs_10620_11039.png High_performance_APIs_11040_12215.png High_performance_APIs_12216_12287.png High_performance_APIs_16764_17087.png High_performance_APIs_17088_17255.png High_performance_APIs_17304_17507.png High_performance_APIs_17508_17723.png High_performance_APIs_17724_17795.png High_performance_APIs_17796_17951.png High_performance_APIs_17952_18671.png High_performance_APIs_18720_19619.png High_performance_APIs_19620_19691.png High_performance_APIs_19692_35627.png High_performance_APIs_35628_35759.png High_performance_APIs_35760_46355.png High_performance_APIs_46356_46439.png High_performance_APIs_46440_46487.png High_performance_APIs_46656_46669.png