DocPad can be blazing fast or horribly slow. This document will go into the various facets of DocPad performance.
Current issues:
-
If you are performing queries inside your documents, it will be very slow. Solution is to use live collections inside your DocPad Config File.
-
If you have 1000+ files or a few very large files, DocPad will be quite slow. Solution is to use the Raw Plugin. #446
-
If you have 1000+ documents (not files), DocPad will get quite slow (1000 jade documents with a layout takes 3 minutes). No one solution. #936
Why can DocPad be slow?
- Queries in Documents
- If inside your layout or partial you are doing something like
@getCollection('html').findAll(menu:true).modelsthen you are doing a query on the entirehtmlcollection every time that layout is rendered. That is sure to be slow (especially times 1000x documents with that layout or partial).- The solution here is to make use of Query Engine’s Live Collections as much as possible. This means adding the following to your DocPad configuration file
collections: menu: -> @getCollection('html').findAllLive(menu:true)then inside your template doing@getCollection('menu').models. Live collections only perform query comparisons when they need to (e.g. on changes or new models).
- The solution here is to make use of Query Engine’s Live Collections as much as possible. This means adding the following to your DocPad configuration file
- If inside your layout or partial you are doing something like
- In Memory Database
- To keep DocPad’s setup easy, DocPad utilises an in-memory database. That means DocPad keeps files and documents inside it’s memory, that includes their content. That means if you have a 64MB image inside
src/staticthen that is now a 64MB image in memory. It adds up.- The solution here is unless you require the file in memory (e.g. for rendering or cross-document referencing) then the file should be inside the Raw Plugin’s
src/rawdirectory instead.
- The solution here is unless you require the file in memory (e.g. for rendering or cross-document referencing) then the file should be inside the Raw Plugin’s
- To keep DocPad’s setup easy, DocPad utilises an in-memory database. That means DocPad keeps files and documents inside it’s memory, that includes their content. That means if you have a 64MB image inside
- Redundant Template Portions/Partials
- If you say each one of your blog posts utilises a single layout, and that single layout utilises a partial, then for each blog post that is 3 renders that occur (one for the blog post, one for the layout, one for the partial). Times that by each blog post and it adds up.
- The solution here is if your partial outputs the same content each time it will be rendered, you can add
cacheable: trueto its meta data, this means that it will now be cached after the first render of that generation and not rendered again (using the cached result from the first render for subsequent calls). This is especially useful if your layout contains other redundant templating, as they can be abstracted out into a cached partial! Awesome.
- The solution here is if your partial outputs the same content each time it will be rendered, you can add
- If you say each one of your blog posts utilises a single layout, and that single layout utilises a partial, then for each blog post that is 3 renders that occur (one for the blog post, one for the layout, one for the partial). Times that by each blog post and it adds up.
- Document References
- DocPad is intelligent… for the most part. It is able to detect when a document, like a blog post listing, or a stylesheet that is to be concatenated on render, references other documents. This allows DocPad to be able to keep blog post listings up to date by rendering them when a blog post changes. The problem is that while DocPad knows that certain documents references other documents, it does not yet know which documents that document references. This means when you modify a page, DocPad will also regenerate that blog post listing just to be safe. Yes this is dumb, but safe and slow is better than invalid incorrect outdated data.
- There is no current solution for this.
- DocPad is intelligent… for the most part. It is able to detect when a document, like a blog post listing, or a stylesheet that is to be concatenated on render, references other documents. This allows DocPad to be able to keep blog post listings up to date by rendering them when a blog post changes. The problem is that while DocPad knows that certain documents references other documents, it does not yet know which documents that document references. This means when you modify a page, DocPad will also regenerate that blog post listing just to be safe. Yes this is dumb, but safe and slow is better than invalid incorrect outdated data.
Other common workarounds:
- Keeping processing down:
- Use the
standalone: trueflag - @todo document the consequences of this and the sideeffects of this - Use the
referencesOthers: falseflag - @todo document the consequences of this and the sideeffects of this
- Use the
What DocPad has been doing about this:
Memory
Issues
- #545 memory usage optimisations discussion
- #527 stream not buffer - risks stuffing up content reading in general
- #276 raw directory - no risks - completed
- #528 forget content after it’s needed - risks stuffing up references inside dynamic documents
- #629 render documents as we need them
- #788 Get rid of Backbone Models for native javascript objects
Notes
- watch memory and decrease parallel tasks is another option (requires #528 to be effective) but is silly compared to just doing #527
- memory optimisations are essential for deployment to dotcloud
Performance
Issues
- #508 performance optimisations discussion
- #336 Keep references on which documents reference which
- #535 suspend collections - closed due to negligible performance impact
- #538 remove all traces of synchronous file system calls - completed
- #590 importer speed optimisations
- #788 Get rid of Backbone Models for native javascript objects
- implement a “no streams” option in caterpillar
QueryEngine:
- #24 specific (instead of generic) change listeners
Notes
- performance improves with native streams over readable-stream inside caterpillar (until we drop 0.8 support, tough)
Migrated from #529
