Scaling PhantomJS With Ghost Town

My first project at Buzzvil was to develop a system to reliably render images at scale with PhantomJS. The result of my efforts was Ghost Town: simple queued & clustered PhantomJS processing in a tiny Node.js module. Problem PhantomJS is not an easy library to work with. Crashing and freezing is common. High memory usage and slow startup time is expected, and scaling must be done manually. Ultimately, PhantomJS and phantomjs-node are neglected projects, so these issues can be expected to remain indefinitely. At Buzzvil, we needed the ability to reliably render a wide variety of images, with low latency and therefore high concurrency. No existing project effectively managed this, so I researched and designed a module that could both gracefully recover from crashes and scale automatically. Solution Enter Ghost Town. It does the heavy lifting of launching and managing PhantomJS and queuing and processing tasks, and then it gets out of your way to let you do the rest. Each item is stored in the masters queue until a worker is ready, and then assigned for processing. If the processing times out or the assigned worker fails, Ghost Town will requeue it. Reliability guaranteed! To prevent memory leaks, Ghost Town creates separate PhantomJS processes for each worker, and periodically relaunches them based on their number of pages created. Implementation Initializing Ghost Town is easy. (Several configuration options are available for tweaking the runtime and efficiency settings see thedocumentation for details.) With no configuration: var town = require(&quotghost-town")() Next, implement the master and worker as when using the normal cluster module.