Awesome q2a theme

Periodically perform the task?

0 like 0 dislike
32 views
Hello. Please tell me. How can I resolve this problem. Need http request to pull data from website and save them in the database. It is implemented, but the question is what to do every minute?
How to use gem whenewer, then get frequent requests when one in the process execution and the other began, there may be problems with the locks?
Advised threads, like using gem 'cuncurrent-ruby' Concurrent::TimerTask, but then how would if be embedded in a rail application, then will start on each your Worker thread. And if you write a separate, it is unclear how best to do all this?
Tell me, can someone like implemented?
by | 32 views

2 Answers

0 like 0 dislike
There is some confusion

Using cron to run processes, not threads, so the problem with locks should be avoided, because every startup is its own process with its own address space. There may be problems with data integrity in the database.

Now about the implementation:

This is best done through a queue service like sidekiq.

How does it work?

Through the API, sidekiq, you can put task in the queue ( including put you and every minute, this can be specified in config), startup options are stored in redis (which need to be put on production). The foremost task of parsing must be set to 1 thread of execution.

Sidekiq on hanging out separate process in addition to the rails, just takes the top task from queue and executes it.

Thus, if the parsing more than a minute - simply increases in long queues. This value is monitored, but if it grows, the disaster will not.

You can do is probably using ActiveJob and in the main database - Rails, I haven't watched the innovations about it.
by
0 like 0 dislike
Kron adds a task to the queue. Task in the queue should take as little time as possible.
by

Related questions

0 like 0 dislike
1 answer
asked Sep 1, 2019 by AlanIkaev
0 like 0 dislike
3 answers
0 like 0 dislike
1 answer
0 like 0 dislike
1 answer
asked Aug 3, 2019 by AlanIkaev
0 like 0 dislike
3 answers
110,608 questions
257,187 answers
0 comments
40,796 users