When background job processing needs arise, Sidekiq is the de facto choice. It's a great tool which has been around for years, but it doesn't provide recurring job processing out of the box. sidekiq-scheduler fills that gap, it's a Sidekiq extension to run background jobs in a recurring manner. In this talk, we'll cover how sidekiq-scheduler does its job, different use cases, challenges when running on distributed environments, its future, how we distribute capacity over open source initiatives, and as a bonus, how to write your own Sidekiq extensions.
 Awesome. Well, so this is a sponsor talk by Muvit, the company we work at. And we want to talk about Sidekick Scheduler, which is a Sidekick extension for, an extension for Sidekick. There we go. So my name is Andreas and this is Gian. We work for Muvit. We're based out of South America and Uruguay. And we've been growing. We've got people in Argentina and Colombia. And we also have an office in Austin. If you're in Austin, we're also always hiring. So go check us out. Yeah, so Sidekick Scheduler, that's where it's hosted at. And I'll leave you with Gian now for a while. He'll explain a little more about what it is. Okay. So let's start with a couple of use cases that we have. In 2012, we were working on a project that, among other things, needed to daily fetch information related to credit level scores based on financial data and apply interest to unpaid credit card debts. Also on Friday nights, we needed to send week activities, some of these reports to a couple of email addresses. The first approach, which we thought of, was to set up current entries, which will execute great tasks, which will execute great tasks, performing the processes that I mentioned before. We thought it was a reasonable approach, but with some drawbacks. So why not run? The answer to that question is composed of, because it needs to start the Ruby interpreter in each run. We are using JRuby and take some time to start up and consume a little bit of memory. So if we have a current entry that says run this at 3 p.m., for example, it will need to start another JRuby process and consume some memory. There's no easy way to programmatically add, remove, enable, and disable current entries. Current configuration will need to live outside of the app's configuration. It's not a portable solution because it relies on the operating system. And you can start and stop the scaler without affecting other current entries. There are also some drawbacks, like current minimum resolution is at minute level, not at second level, and deploying the app in a cluster will turn to trigger duplicate tasks when in fact you need to run only once. Despite those drawbacks, we didn't discard the current one, but we thought if there was some alternative solution. We found a gem in 2012, named a psychic scaler whose purpose was to scale psychic jobs at a specific time in the future. This wasn't what we were looking for, but we asked ourselves how hard it can be to add some current support to psychic scaler. Researching options, we stumbled upon Rufus Scaler, which is a gem where you can use current bugs to schedule a call to a block of root codes. So we came up with the idea of integrating Rufus into psychic scaler, adding that way current support. That was in August 2012, and after doing that, we started using psychic scaler in the project we were working on. In 2013, Morton was fighting hard to actively maintain the gem, so he transferred the ownership to us. And in 2016, we added support for real jobs when psychic is acting as active just for games. So how do we use psychic scaler? A psychic scaler is just an extension like applying to psychic. Using it is not really different than using psychic. Therefore, we need to declare a regular psychic worker, which is a Ruby class, including psychic worker module, and that also responds to the performance. The schedule configuration is placed inside the psychic configuration file at the schedule key. So in that example, we have a Hello World job that runs every minute when second is equal to zero, and when that specific point in time occurs, psychic scaler will push the Hello World job into psychic. Okay, then you install the gem, of course. With gem installed, or if you want to use Banner, you just add it into the gem file as usual. Then we run psychic as usual also. As we are here in the example, we are outside of race. We are telling psychic to require our Ruby file that contains the worker class. And that's it. Every minute, psychic scaler will enqueue the job, and then psychic will enqueue the job. Okay. There are different scaling types, and as we rely on Rufus, which is a separate gem, as we rely on Rufus, different scaling types are supported. The common one, the most common one, the most popular one is Krone, which runs every minute, and it's the same Krone-like syntax that you are used to, the Krone tool, the standard Unix tool, you can use it in psychic scaler. Add type, which pushes the job once at a specific point in time. Every, that's it, pushes the jobs in a current Y way following a given frequency. And in type, which pushes the job once after some time duration has elapsed. Okay. The main purpose of psychic scaler is to push jobs into psychic. This is the main idea. Letting psychic then random shows as usual. So how does psychic scaler works? When psychic scaler is required, when you require psychic scaler, it hooks into the startup and shut down psychic's lifecycle method. We will explain then what those lifecycle events or phases are. Then on the startup phase, we fetch the configuration, read the configuration, and starts Rufus scaler, which is just a thread that iterates over all the scalers and invokes each Ruby blog when needed. We start Rufus scaler and set up a scaler job for every one of the computer jobs. Each handler in Rufus is responsible for store execution info into RAIS, verify that in case of Chrome and add jobs, that show instance was not previously pushed, and finally pushed the show into psychic, letting psychic run the show as usual. While implementing this approach, we stumbled upon the challenge of managing multiple instances at the same time. At first, recording jobs were meant to be run on one psychic instance, only one. And if multiple nodes were needed, only one node will run psychic scaler. While the others will host regular psychic instances, running shows pushed by a single one. Support for multiple nodes was added later for Chrome and add scale types. So right now, it's possible to run multiple psychic scalers, scaler instances, having Chrome and add shows. Those Chrome and add shows will not run duplicate. All right. Well, here. Otherwise, I'll just shout and you'll hear me. Sorry about that. All right. Let's talk a little bit about the future, which, yeah. Honestly, the one thing that we want to do is we want to stop colluding the global psychic configuration, because right now, we kind of load the configuration into the psychic configuration. And kind of, you know, if psychic itself would try to do some other stuff, we could possibly have a collision there. So we want to take that away. We want to get every and end schedule types to work with multiple instances, you know, so you don't have to worry about that. And, well, so one of the things we don't want to do is we don't want to work on psychic scheduler with stuff that is outside of the scope of just scheduling, right? So we want to, that psychic scheduler just, you know, does that job and nothing more. Also, there's some test refactor that needs to be done and a little code-based refactor. All right. Demo time. Now, I did make a video out of this, so I don't, you know, so stuff works. You know, hopefully, right. So really, one idea that we had in mind for the demo was for our repository or open source repository, if you have a, right, that's not worth, that's right, ignore it. If you have some, you know, some issues and let's say, you know, the last comment on an issue was from you and you're not really trying to implement that issue and maybe it falls into an activity and the original poster maybe lost interest or really doesn't care about it anymore. We thought that maybe after a month or two, you might want to automatically close it. So we, that was the general idea we had, but since we didn't have a few months to generate all that test data, what we did is to have the invalid and duplicate marked issues. So we created, you know, a little code and psychic scheduler thing that runs every minute and checks those marked issues and just closes them. So, all right. So right now, you go to the code base. I'm sorry if that's a little small. You run psychic scheduler and you'll see we have a con job there in Q's which is called issue clean up there, scheduled to run every minute and then not in a minute, but in a few seconds we'll see here that we'll start running, we'll find the issues, just close them and so that you also believe me. We're going to go over to the browser now and refresh it. You'll see that they were closed and obviously if nothing is open, the job will run and it will do nothing so it will not find any issue. And so if we go back and now let's say a few months passed and no activity was on that issue, you know, it will simulate that by marking it as a duplicate or a, and the next time the job runs it will just clean it up. Okay. So as a bonus to this talk, we wanted to show you what it takes or what tools you have available to write a psychic extension. So you got these four events in the psychic lifecycle that you can hook into. Like John mentioned earlier, we hook into startup and shutdown events. So that's where we, so in the startup event we load up the configuration, start the roof of scheduler and then in the shutdown, I don't think it does a lot. It just dies. Clean up the thread and die. Yeah. And then one thing that I think we didn't display here, but when you run the web extension, a psychic scheduler would actually have create another type for you and it will show you the jobs that are currently in queue and it will also let you disable them. So you have, you know, without logging into the server you can control it via the web. And so to do that, you just got to create like a module extension where you got to hook into that method called self.registered and then you can, with that app.get and put, you can put the path in there and it's a way for you to, you know, define actions and return HTML or whatever data you want to render there on that tab. And once you do that, you have to, in the initializer, I think, you have to register the extension on psychic. So on the psychic web you got to register it and you can add the tab and there you put the name and the path that it points to. And the path really is back here when you do app.get my ex. That's the path you got to use there. And yeah, and you also have the option of adding locales in case you support various languages. With psychic scheduler it's just those four, I think. A little more. Okay. So we're just showing four. So you can, you know, support more than one language. And yeah, so this example is up on the move it, that URL. Although if you want to check out the psychic extension, maybe you want to check out psychic scheduler. All right. Now, about feedback and collaboration, since this is obviously an open source project, we, it has had, since, I mean, we didn't create it, we just, you know, took over in maintenance a few years ago. So over the course of the years it has had 43 contributors. And right now we have like six hours a week allocated for issues. So if you find anything wrong with psychic scheduler, you'll get some time for us to, you know, to fix it. And as always, help and feedback is very appreciated. And as a last thing, we have some other open source libraries. We got Roosin, which is a simple exception notification gem that you can use for logging and sending errors in any Ruby app that you have. We have Rui, which is a lightweight rules evaluator for context-driven conditional expressions. You can do some weird stuff with that. And then we have Angus. And it's, so the name for that was a real story. In Uruguay, we're a meat-loving country. We like to eat a lot of meat. And so Angus, you know, obviously is a famous cow, a good one. And, but it really doesn't have to do anything with that. We just thought it was cool. What it is, it's a REST-like API framework. And it will run on RAC. So if you're already using a RAC app, you can use that to serve your API. And it already generates, so while you're writing, it kind of generates documentation on the way. And, yeah, so you can use that for RESTful APIs. And then we have Fakeit, which is not really something for Ruby. It's an Android fake data, or fake, but realistic data generator. So you can generate names, email states, and country names, which, you know, it will be fake, but it will be kind of real. So, you know, you don't want to generate just a few letters for an email. You need a certain format. Yeah. So, but that is for Android tests, basically. Yeah. And that's it for us. Pretty short. So the question is, how do we manage the second resolution within the scheduler? Well, in fact, we use Rufus scheduler for that. Rufus just starts up a thread, and every, every, some, between some milliseconds, period, like, I don't know, 500, I think, it checks its loops through all the schedules, and asks if the scheduler, the schedule has to run. And in that case, it calls the Ruby block. So it's not just like hooks with some events from the, from the operative system. It just makes a loop, it loops, loops again, loops again. So it's, that means that your Ruby block will not run at 2 p.m. in a certain second, certain milliseconds. Okay. It will run at, I don't know, 190 milliseconds. Okay. It's not, it's not really precise. It's not perfect. Yeah. Yeah. What assures is that it will run in that second, in that second. All right. Thank you very much. Thanks for coming. Thank you.
 Awesome.  Well, so this is a sponsor talk by Muvit, the company we work at.  And we want to talk about Sidekick Scheduler, which is a Sidekick extension for,  an extension for Sidekick. There we go.  So my name is Andreas and this is Gian.  We work for Muvit. We're based out of South America and Uruguay.  And we've been growing. We've got people in Argentina and Colombia.  And we also have an office in Austin. If you're in Austin, we're also always hiring.  So go check us out.  Yeah, so Sidekick Scheduler, that's where it's hosted at.  And I'll leave you with Gian now for a while. He'll explain a little more about what it is.  Okay. So let's start with a couple of use cases that we have.  In 2012, we were working on a project that, among other things, needed to daily fetch  information related to credit level scores based on financial data and apply interest to  unpaid credit card debts. Also on Friday nights, we needed to send  week activities, some of these reports to a couple of email addresses.  The first approach, which we thought of, was to set up current entries, which will execute  great tasks, which will execute great tasks, performing the processes that I mentioned before.  We thought it was a reasonable approach, but with some drawbacks.  So why not run? The answer to that question is composed of, because it needs to start  the Ruby interpreter in each run. We are using JRuby and take some time to start up and  consume a little bit of memory. So if we have a current entry that says run this at 3 p.m.,  for example, it will need to start another JRuby process and consume some memory.  There's no easy way to programmatically add, remove, enable, and disable current entries.  Current configuration will need to live outside of the app's configuration.  It's not a portable solution because it relies on the operating system.  And you can start and stop the scaler without affecting other current entries.  There are also some drawbacks, like current minimum resolution is at minute level, not at  second level, and deploying the app in a cluster will turn to trigger duplicate tasks when  in fact you need to run only once. Despite those drawbacks, we didn't discard  the current one, but we thought if there was some alternative solution.  We found a gem in 2012, named a psychic scaler whose purpose was to scale psychic jobs at a  specific time in the future. This wasn't what we were looking for, but we asked ourselves  how hard it can be to add some current support to psychic scaler.  Researching options, we stumbled upon Rufus Scaler, which is a gem where you can use current  bugs to schedule a call to a block of root codes.  So we came up with the idea of integrating Rufus into psychic scaler, adding that way  current support. That was in August 2012, and after doing that, we started using psychic  scaler in the project we were working on. In 2013, Morton was fighting hard to actively  maintain the gem, so he transferred the ownership to us.  And in 2016, we added support for real jobs when psychic is acting as active just for  games. So how do we use psychic scaler? A psychic scaler is just an extension like  applying to psychic. Using it is not really different than using psychic.  Therefore, we need to declare a regular psychic worker, which is a Ruby class, including psychic  worker module, and that also responds to the performance.  The schedule configuration is placed inside the psychic configuration file at the schedule  key. So in that example, we have a Hello World job that runs every minute when second is  equal to zero, and when that specific point in time occurs, psychic scaler will push the  Hello World job into psychic.  Okay, then you install the gem, of course. With gem installed, or if you want to use  Banner, you just add it into the gem file as usual.  Then we run psychic as usual also. As we are here in the example, we are outside of race.  We are telling psychic to require our Ruby file that contains the worker class.  And that's it. Every minute, psychic scaler will enqueue the job, and then psychic will  enqueue the job.  Okay. There are different scaling types, and as we rely on Rufus, which is a separate  gem, as we rely on Rufus, different scaling types are supported.  The common one, the most common one, the most popular one is Krone, which runs every  minute, and it's the same Krone-like syntax that you are used to, the Krone tool, the  standard Unix tool, you can use it in psychic scaler.  Add type, which pushes the job once at a specific point in time.  Every, that's it, pushes the jobs in a current Y way following a given frequency.  And in type, which pushes the job once after some time duration has elapsed.  Okay. The main purpose of psychic scaler is to push jobs into psychic.  This is the main idea. Letting psychic then random shows as usual.  So how does psychic scaler works?  When psychic scaler is required, when you require psychic scaler, it hooks into the  startup and shut down psychic's lifecycle method.  We will explain then what those lifecycle events or phases are.  Then on the startup phase, we fetch the configuration, read the configuration, and  starts Rufus scaler, which is just a thread that iterates over all the scalers and  invokes each Ruby blog when needed.  We start Rufus scaler and set up a scaler job for every one of the  computer jobs.  Each handler in Rufus is responsible for store execution info into RAIS, verify that  in case of Chrome and add jobs, that show instance was not previously pushed, and  finally pushed the show into psychic, letting psychic run the show as usual.  While implementing this approach, we stumbled upon the challenge of managing multiple  instances at the same time.  At first, recording jobs were meant to be run on one psychic instance, only one.  And if multiple nodes were needed, only one node will run psychic scaler.  While the others will host regular psychic instances, running shows pushed by a single  one.  Support for multiple nodes was added later for Chrome and add scale types.  So right now, it's possible to run multiple psychic scalers, scaler instances, having  Chrome and add shows.  Those Chrome and add shows will not run duplicate.  All right.  Well, here.  Otherwise, I'll just shout and you'll hear me.  Sorry about that.  All right.  Let's talk a little bit about the future, which, yeah.  Honestly, the one thing that we want to do is we want to stop colluding the global  psychic configuration, because right now, we kind of load the configuration into the  psychic configuration.  And kind of, you know, if psychic itself would try to do some other stuff, we could  possibly have a collision there.  So we want to take that away.  We want to get every and end schedule types to work with multiple instances, you know,  so you don't have to worry about that.  And, well, so one of the things we don't want to do is we don't want to work on psychic  scheduler with stuff that is outside of the scope of just scheduling, right?  So we want to, that psychic scheduler just, you know, does that job and nothing more.  Also, there's some test refactor that needs to be done and a little code-based refactor.  All right.  Demo time.  Now, I did make a video out of this, so I don't, you know, so stuff works.  You know, hopefully, right.  So really, one idea that we had in mind for the demo was for our repository or open source  repository, if you have a, right, that's not worth, that's right, ignore it.  If you have some, you know, some issues and let's say, you know, the last comment on an  issue was from you and you're not really trying to implement that issue and maybe it falls  into an activity and the original poster maybe lost interest or really doesn't care about  it anymore.  We thought that maybe after a month or two, you might want to automatically close it.  So we, that was the general idea we had, but since we didn't have a few months to generate  all that test data, what we did is to have the invalid and duplicate marked issues.  So we created, you know, a little code and psychic scheduler thing that runs every minute  and checks those marked issues and just closes them.  So, all right.  So right now, you go to the code base.  I'm sorry if that's a little small.  You run psychic scheduler and you'll see we have a con job there in Q's which is called  issue clean up there, scheduled to run every minute and then not in a minute, but in a few  seconds we'll see here that we'll start running, we'll find the issues, just close them and  so that you also believe me.  We're going to go over to the browser now and refresh it.  You'll see that they were closed and obviously if nothing is open, the job will run and it  will do nothing so it will not find any issue.  And so if we go back and now let's say a few months passed and no activity was on that  issue, you know, it will simulate that by marking it as a duplicate or a, and the next  time the job runs it will just clean it up.  Okay.  So as a bonus to this talk, we wanted to show you what it takes or what tools you have  available to write a psychic extension.  So you got these four events in the psychic lifecycle that you can hook into.  Like John mentioned earlier, we hook into startup and shutdown events.  So that's where we, so in the startup event we load up the configuration, start the roof  of scheduler and then in the shutdown, I don't think it does a lot.  It just dies.  Clean up the thread and die.  Yeah.  And then one thing that I think we didn't display here, but when you run the web extension,  a psychic scheduler would actually have create another type for you and it will show you  the jobs that are currently in queue and it will also let you disable them.  So you have, you know, without logging into the server you can control it via the web.  And so to do that, you just got to create like a module extension where you got to hook  into that method called self.registered and then you can, with that app.get and put, you  can put the path in there and it's a way for you to, you know, define actions and return  HTML or whatever data you want to render there on that tab.  And once you do that, you have to, in the initializer, I think, you have to register  the extension on psychic.  So on the psychic web you got to register it and you can add the tab and there you put  the name and the path that it points to.  And the path really is back here when you do app.get my ex.  That's the path you got to use there.  And yeah, and you also have the option of adding locales in case you support various languages.  With psychic scheduler it's just those four, I think.  A little more.  Okay.  So we're just showing four.  So you can, you know, support more than one language.  And yeah, so this example is up on the move it, that URL.  Although if you want to check out the psychic extension, maybe you want to check out psychic  scheduler.  All right.  Now, about feedback and collaboration, since this is obviously an open source project,  we, it has had, since, I mean, we didn't create it, we just, you know, took over in maintenance  a few years ago.  So over the course of the years it has had 43 contributors.  And right now we have like six hours a week allocated for issues.  So if you find anything wrong with psychic scheduler, you'll get some time for us to,  you know, to fix it.  And as always, help and feedback is very appreciated.  And as a last thing, we have some other open source libraries.  We got Roosin, which is a simple exception notification gem that you can use for logging  and sending errors in any Ruby app that you have.  We have Rui, which is a lightweight rules evaluator for context-driven conditional expressions.  You can do some weird stuff with that.  And then we have Angus.  And it's, so the name for that was a real story.  In Uruguay, we're a meat-loving country.  We like to eat a lot of meat.  And so Angus, you know, obviously is a famous cow, a good one.  And, but it really doesn't have to do anything with that.  We just thought it was cool.  What it is, it's a REST-like API framework.  And it will run on RAC.  So if you're already using a RAC app, you can use that to serve your API.  And it already generates, so while you're writing, it kind of generates documentation on the way.  And, yeah, so you can use that for RESTful APIs.  And then we have Fakeit, which is not really something for Ruby.  It's an Android fake data, or fake, but realistic data generator.  So you can generate names, email states, and country names, which, you know, it will be fake,  but it will be kind of real.  So, you know, you don't want to generate just a few letters for an email.  You need a certain format.  Yeah.  So, but that is for Android tests, basically.  Yeah.  And that's it for us.  Pretty short.  So the question is, how do we manage the second resolution within the scheduler?  Well, in fact, we use Rufus scheduler for that.  Rufus just starts up a thread, and every, every, some, between some milliseconds, period,  like, I don't know, 500, I think, it checks its loops through all the schedules, and asks  if the scheduler, the schedule has to run.  And in that case, it calls the Ruby block.  So it's not just like hooks with some events from the, from the operative system.  It just makes a loop, it loops, loops again, loops again.  So it's, that means that your Ruby block will not run at 2 p.m.  in a certain second, certain milliseconds.  Okay.  It will run at, I don't know, 190 milliseconds.  Okay.  It's not, it's not really precise.  It's not perfect.  Yeah.  Yeah.  What assures is that it will run in that second, in that second.  All right.  Thank you very much.  Thanks for coming.  Thank you.
Recurring_Background_Jobs_with_Sidekiq-scheduler_2_0_1379.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_1380_1763.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_1764_1943.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_1944_2675.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_2676_2963.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_2964_3359.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_3360_4643.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_4644_6443.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_6444_6515.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_6516_8747.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_8748_9071.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_9072_9155.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_9156_9791.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_9792_10103.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_10104_10451.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_10452_12335.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_12336_12671.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_12672_12923.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_12924_13511.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_13512_13871.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_13872_15419.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_15420_19583.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_19584_23231.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_23232_23375.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_23376_24227.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_29604_29879.png Recurring_Background_Jobs_with_Sidekiq-scheduler_2_29880_29918.png