fullstackpython.github.com/task-queues.html at patch-1 · hypertexthero/fullstackpython.github.com

History

485 lines (478 loc) · 24.4 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

<!DOCTYPE html>

<title>Task Queues - Full Stack Python</title>

<!--[if lt IE 9]>

<![endif]-->

var _gaq = _gaq || [];

_gaq.push(['_setAccount', 'UA-19910497-7']);

_gaq.push(['_trackPageview']);

(function() {

var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;

ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';

var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);

})();

</script>

</head>

<body>

</a>

<span class="logo-title"><a href="/">Full Stack Python</a></span>

</div>

</div> <div class="row">

<h1>Task queues</h1>

<p>Task queues manage background work that must be executed outside the usual

HTTP request-response cycle.</p>

<h2>Why are task queues necessary?</h2>

<p>Tasks are handled asynchronously either because they are not initiated by

an HTTP request or because they are long-running jobs that would dramatically

reduce the performance of an HTTP response.</p>

<p>For example, a web application could poll the GitHub API every 10 minutes to

collect the names of the top 100 starred repositories. A task queue would

handle invoking code to call the GitHub API, process the results and store them

in a persistent database for later use.</p>

<p>Another example is when a database query would take too long during the HTTP

request-response cycle. The query could be performed in the background on a

fixed interval with the results stored in the database. When an

HTTP request comes in that needs those results a query would simply fetch the

precalculated result instead of re-executing the longer query.

This precalculation scenario is a form of <a href="/caching.html">caching</a> enabled

by task queues.</p>

<p>Other types of jobs for task queues include</p>

<ul>

<li>

<p>spreading out large numbers of independent database inserts over time

instead of inserting everything at once</p>

</li>

<li>

<p>aggregating collected data values on a fixed interval, such as every

15 minutes</p>

</li>

<li>

<p>scheduling periodic jobs such as batch processes</p>

</li>

</ul>

<h2>Task queue projects</h2>

<p>The defacto standard Python task queue is Celery. The other task queue

projects that arise tend to come from the perspective that Celery is overly

complicated for simple use cases. My recommendation is to put the effort into

Celery's reasonable learning curve as it is worth the time it takes to

understand how to use the project.</p>

<ul>

<li>

<p>The <a href="http://www.celeryproject.org/">Celery</a> distributed task queue is the

most commonly used Python library for handling asynchronous tasks and

scheduling.</p>

</li>

<li>

<p>The <a href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python

library for queueing jobs and processing them in the background with workers.

RQ is backed by Redis and is designed to have a low barrier to entry.

The <a href="http://nvie.com/posts/introducing-rq/">intro post</a> contains information

on design decisions and how to use RQ.</p>

</li>

<li>

<p><a href="https://github.com/dcramer/taskmaster">Taskmaster</a> is a lightweight simple

distributed queue for handling large volumes of one-off tasks. </p>

</li>

</ul>

<h2>Hosted message and task queue services</h2>

<p>Task queue third party services aim to solve the complexity issues that arise

when scaling out a large deployment of distributed task queues.</p>

<ul>

<li>

<p><a href="http://www.iron.io/">Iron.io</a> is a distributed messaging service platform

that works with many types of task queues such as Celery. It also is built

to work with other IaaS and PaaS environments such as Amazon Web Services

and Heroku.</p>

</li>

<li>

<p><a href="http://aws.amazon.com/sqs/">Amazon Simple Queue Service (SQS)</a> is a

set of five APIs for creating, sending, receiving, modifying and deleting

messages.</p>

</li>

<li>

<p><a href="http://www.cloudamqp.com/">CloudAMQP</a> is at its core managed servers with

RabbitMQ installed and configured. This service is an option if you are

using RabbitMQ and do not want to maintain RabbitMQ installations on your

own servers.</p>

</li>

</ul>

<h2>Task queue resources</h2>

<ul>

<li>

<p><a href="http://justcramer.com/2012/05/04/distributing-work-without-celery/">Distributing work without Celery</a>

provides a scenario in which Celery and RabbitMQ are not the right tool

for scheduling asynchronous jobs.</p>

</li>

<li>

<p><a href="http://queues.io/">Queues.io</a> is a collection of task queue systems with

short summaries for each one. The task queues are not all compatible with

Python but ones that work with it are tagged with the "Python" keyword.</p>

</li>

<li>

<p><a href="http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619">Why Task Queues</a>

is a presentation for what task queues are and why they are needed. </p>

</li>

<li>

<p><a href="https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps">How to use Celery with RabbitMQ</a>

is a detailed walkthrough for using these tools on an Ubuntu VPS.</p>

</li>

<li>

<p>Heroku has a clear walkthrough for using

<a href="https://devcenter.heroku.com/articles/python-rq">RQ for background tasks</a>.</p>

</li>

<li>

<p><a href="http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/">Introducing Celery for Python+Django</a>

provides an introduction to the Celery task queue.</p>

</li>

<li>

<p>The "Django in Production" series by

<a href="https://twitter.com/robgolding63">Rob Golding</a> contains a post

specifically on <a href="http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/">Background Tasks</a>.</p>

</li>

<li>

<p><a href="http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/">Asynchronous Processing in Web Applications Part One</a>

and <a href="http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/">Part Two</a>

are great reads for understanding the difference between a task queue and

why you shouldn't use your database as one.</p>

</li>

</ul>

<h2>Task queue learning checklist</h2>

Pick a slow function in your project that is called during an HTTP request.</p>

Determine if you can precompute the results on a fixed interval instead of

during the HTTP request. If so, create a separate function you can call

from elsewhere then store the precomputed value in the database.</p>

Read the Celery documentation and the links in the resources section below

to understand how the project works.</p>

Install a message broker such as RabbitMQ or Redis and then add Celery to your

project. Configure Celery to work with the installed message broker.</p>

Use Celery to invoke the function from step one on a regular basis.</p>

Have the HTTP request function use the precomputed value instead of the

slow running code it originally relied upon.</p>

<h3>What's next after task queues?</h3>

How do I monitor my app and its task queues with logging?

</p>

</div>

</a>

How can I learn more about the users of my application?

</p>

</div>

What tools exist for monitoring a live web application?

</p>

</div>

</div> <style type="text/css">

#mc_embed_signup{background:#fff; clear:left; font:12px "Helvetica Neue",Arial,sans-serif; }

/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.

We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */

</style>

<hr/>

<h4>Interested in a complete Full Stack Python book with detailed tutorials and example code? Sign up here and you'll get an alert email if a book is created. No other emails will be sent other than sign up confirmation.</h4>

<label for="mce-EMAIL">Email Address</label>

</div>

</div>

</div>

</div>

</form>

</div>

var fnames = new Array();var ftypes = new Array();fnames[0]='EMAIL';ftypes[0]='email';fnames[1]='FNAME';ftypes[1]='text';fnames[2]='LNAME';ftypes[2]='text';

try {

var jqueryLoaded=jQuery;

jqueryLoaded=true;

} catch(err) {

var jqueryLoaded=false;

}

var head= document.getElementsByTagName('head')[0];

if (!jqueryLoaded) {

var script = document.createElement('script');

script.type = 'text/javascript';

script.src = '//ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js';

head.appendChild(script);

if (script.readyState && script.onload!==null){

script.onreadystatechange= function () {

if (this.readyState == 'complete') mce_preload_check();

}

var err_style = '';

try{

err_style = mc_custom_error_style;

} catch(e){

err_style = '#mc_embed_signup input.mce_inline_error{border-color:#6B0505;} #mc_embed_signup div.mce_inline_error{margin: 0 0 1em 0; padding: 5px 10px; background-color:#6B0505; font-weight: bold; z-index: 1; color:#fff;}';

}

var head= document.getElementsByTagName('head')[0];

var style= document.createElement('style');

style.type= 'text/css';

if (style.styleSheet) {

style.styleSheet.cssText = err_style;

} else {

style.appendChild(document.createTextNode(err_style));

}

head.appendChild(style);

setTimeout('mce_preload_check();', 250);

var mce_preload_checks = 0;

function mce_preload_check(){

if (mce_preload_checks>40) return;

mce_preload_checks++;

try {

var jqueryLoaded=jQuery;

} catch(err) {

setTimeout('mce_preload_check();', 250);

return;

}

var script = document.createElement('script');

script.type = 'text/javascript';

script.src = 'http://downloads.mailchimp.com/js/jquery.form-n-validate.js';

head.appendChild(script);

try {

var validatorLoaded=jQuery("#fake-form").validate({});

} catch(err) {

setTimeout('mce_preload_check();', 250);

return;

}

mce_init_form();

}

function mce_init_form(){

jQuery(document).ready( function($) {

var options = { errorClass: 'mce_inline_error', errorElement: 'div', onkeyup: function(){}, onfocusout:function(){}, onblur:function(){} };

var mce_validator = $("#mc-embedded-subscribe-form").validate(options);

$("#mc-embedded-subscribe-form").unbind('submit');//remove the validator so we can get into beforeSubmit on the ajaxform, which then calls the validator

options = { url: 'http://mattmakai.us2.list-manage.com/subscribe/post-json?u=b7e774f0c4f05dcebbfee183d&id=b22335388d&c=?', type: 'GET', dataType: 'json', contentType: "application/json; charset=utf-8",

beforeSubmit: function(){

$('#mce_tmp_error_msg').remove();

$('.datefield','#mc_embed_signup').each(

function(){

var txt = 'filled';

var fields = new Array();

var i = 0;

$(':text', this).each(

function(){

fields[i] = this;

i++;

});

$(':hidden', this).each(

function(){

var bday = false;

if (fields.length == 2){

bday = true;

fields[2] = {'value':1970};//trick birthdays into having years

}

if ( fields[0].value=='MM' && fields[1].value=='DD' && (fields[2].value=='YYYY' || (bday && fields[2].value==1970) ) ){

this.value = '';

} else if ( fields[0].value=='' && fields[1].value=='' && (fields[2].value=='' || (bday && fields[2].value==1970) ) ){

this.value = '';

} else {

if (/\[day\]/.test(fields[0].name)){

this.value = fields[1].value+'/'+fields[0].value+'/'+fields[2].value;

} else {

this.value = fields[0].value+'/'+fields[1].value+'/'+fields[2].value;

}

});

$('.phonefield-us','#mc_embed_signup').each(

function(){

var fields = new Array();

var i = 0;

$(':text', this).each(

function(){

fields[i] = this;

i++;

});

$(':hidden', this).each(

function(){

if ( fields[0].value.length != 3 || fields[1].value.length!=3 || fields[2].value.length!=4 ){

this.value = '';

} else {

this.value = 'filled';

}

});

return mce_validator.form();

success: mce_success_cb

};

$('#mc-embedded-subscribe-form').ajaxForm(options);

});

}

function mce_success_cb(resp){

$('#mce-success-response').hide();

$('#mce-error-response').hide();

if (resp.result=="success"){

$('#mce-'+resp.result+'-response').show();

$('#mce-'+resp.result+'-response').html(resp.msg);

$('#mc-embedded-subscribe-form').each(function(){

this.reset();

});

} else {

var index = -1;

var msg;

try {

var parts = resp.msg.split(' - ',2);

if (parts[1]==undefined){

msg = resp.msg;

} else {

i = parseInt(parts[0]);

if (i.toString() == parts[0]){

index = parts[0];

msg = parts[1];

} else {

index = -1;

msg = resp.msg;

}

} catch(e){

index = -1;

msg = resp.msg;

}

try{

if (index== -1){

$('#mce-'+resp.result+'-response').show();

$('#mce-'+resp.result+'-response').html(msg);

} else {

err_id = 'mce_tmp_error_msg';

html = '<div id="'+err_id+'" style="'+err_style+'"> '+msg+'</div>';

var input_id = '#mc_embed_signup';

var f = $(input_id);

if (ftypes[index]=='address'){

input_id = '#mce-'+fnames[index]+'-addr1';

f = $(input_id).parent().parent().get(0);

} else if (ftypes[index]=='date'){

input_id = '#mce-'+fnames[index]+'-month';

f = $(input_id).parent().parent().get(0);

} else {

input_id = '#mce-'+fnames[index];

f = $().parent(input_id).get(0);

}

if (f){

$(f).append(html);

$(input_id).focus();

} else {

$('#mce-'+resp.result+'-response').show();

$('#mce-'+resp.result+'-response').html(msg);

}

} catch(e){

$('#mce-'+resp.result+'-response').show();

$('#mce-'+resp.result+'-response').html(msg);

}

</script>

</div>

<div class="panel-heading"><h3 class="panel-head">Task Queues</h3></div>

<a href="/">Full Stack Python</a> is an open book that explains

each Python web application stack layer and provides the

best web resources for those topics.

<hr/>

<a href="http://twitter.com/mattmakai">Matt Makai</a> designed and

wrote this site with assistance from community pull

requests. If you're on GitHub you can

<a href="https://github.com/makaimc">follow him</a>

to see what daily changes are made or

<a href="https://github.com/makaimc/fullstackpython.github.com">star the repository</a> for future reference.

</div>

<h3 class="panel-head">Table of Contents</h3>

</div>

<a href="/introduction.html" class="list-group-item smaller-item ">Introduction</a>

<a href="/web-frameworks.html" class="list-group-item smaller-item ">Web Frameworks</a>

<a href="/django.html" class="list-group-item smaller-item ">Django</a>

<a href="/flask.html" class="list-group-item smaller-item ">Flask</a>

<a href="/bottle.html" class="list-group-item smaller-item ">Bottle</a>

<a href="/other-web-frameworks.html" class="list-group-item smaller-item ">Other Web Frameworks</a>

<a href="/deployment.html" class="list-group-item smaller-item ">Deployment</a>

<a href="/servers.html" class="list-group-item smaller-item ">Servers</a>

<a href="/operating-systems.html" class="list-group-item smaller-item ">Operating Systems</a>

<a href="/web-servers.html" class="list-group-item smaller-item ">Web Servers</a>

<a href="/platform-as-a-service.html" class="list-group-item smaller-item ">Platform-as-a-service</a>

<a href="/wsgi-servers.html" class="list-group-item smaller-item ">WSGI Servers</a>

<a href="/source-control.html" class="list-group-item smaller-item ">Source Control</a>

<a href="/application-dependencies.html" class="list-group-item smaller-item ">Application Dependencies</a>

<a href="/configuration-management.html" class="list-group-item smaller-item ">Configuration Management</a>

<a href="/databases.html" class="list-group-item smaller-item ">Databases</a>

<a href="/no-sql-datastore.html" class="list-group-item smaller-item ">NoSQL Data Stores</a>

<a href="/cascading-style-sheets.html" class="list-group-item smaller-item ">Cascading Style Sheets</a>

<a href="/javascript.html" class="list-group-item smaller-item ">JavaScript</a>

<a href="/application-programming-intefaces.html" class="list-group-item smaller-item ">Application Programming Interfaces</a>

<a href="/api-integration.html" class="list-group-item smaller-item ">API Integration</a>

<a href="/static-content.html" class="list-group-item smaller-item ">Static Content</a>

<a href="/caching.html" class="list-group-item smaller-item ">Caching</a>

<a href="/task-queues.html" class="list-group-item smaller-item active">Task Queues</a>

<a href="/logging.html" class="list-group-item smaller-item ">Logging</a>

<a href="/monitoring.html" class="list-group-item smaller-item ">Monitoring</a>

<a href="/web-analytics.html" class="list-group-item smaller-item ">Web Analytics</a>

<a href="/web-application-security.html" class="list-group-item smaller-item ">Web Application Security</a>

<a href="/best-python-resources.html" class="list-group-item smaller-item ">Best Python Resources</a>

<a href="/about-author.html" class="list-group-item smaller-item ">About the Author</a>

<a href="/change-log.html" class="list-group-item smaller-item ">Change Log</a>

<a href="/future-directions.html" class="list-group-item smaller-item ">Future Directions</a>

</div>

</div></div>

<hr/>

<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a>

2014

</div>

</body>

</html>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

task-queues.html

Latest commit

History

task-queues.html

File metadata and controls