Engine

The Spinach Engine is what connects tasks, jobs, brokers and workers together.

It is possible, but unusual, to have multiple Engines running in the same Python interpreter.

class spinach.engine.Engine(broker: spinach.brokers.base.Broker, namespace: str = 'spinach')

Spinach Engine coordinating a broker with workers.

This class does the orchestration of all components, it is the one that starts and terminates the whole machinery.

The Engine can be run in two modes:

  • client: synchronously submits jobs.
  • worker: asynchronously executes jobs.

Submitting jobs is quite easy, so running the Engine in client mode doesn’t require spawning any thread.

Executing jobs however is a bit more involved, so running the Engine in worker mode ends up spawning a few threads:

  • a few worker threads: they are only responsible for executing the task function and advancing the job status once it is finished.
  • a result notifier thread: sends back the result of job executions to the Broker backend, acts basically as a client.
  • an arbiter thread: fetches jobs from the Broker and gives them to the workers as well as doing some periodic bookkeeping.
  • a Broker subscriber thread: receives notifications from the backend when something happens, typically a job is enqueued.
  • the process main thread: starts all the above threads, then does nothing waiting for the signal to terminate the threads it started.

This means that a Spinach worker process has at least 5 threads.

Parameters:
  • broker – instance of a Broker
  • namespace – name of the namespace used by the Engine. When different Engines use the same Redis server, they must use different namespaces to isolate themselves.
attach_tasks(tasks: spinach.task.Tasks)

Attach a set of tasks.

A task cannot be scheduled or executed before it is attached to an Engine.

>>> tasks = Tasks()
>>> spin.attach_tasks(tasks)
namespace

Namespace the Engine uses.

schedule(task: Union[str, Callable, spinach.task.Task], *args, **kwargs) → spinach.job.Job

Schedule a job to be executed as soon as possible.

Parameters:
  • task – the task or its name to execute in the background
  • args – args to be passed to the task function
  • kwargs – kwargs to be passed to the task function
Returns:

The Job that was created and scheduled.
schedule_at(task: Union[str, Callable, spinach.task.Task], at: datetime.datetime, *args, **kwargs) → spinach.job.Job

Schedule a job to be executed in the future.

Parameters:
  • task – the task or its name to execute in the background
  • at – date at which the job should start. It is advised to pass a timezone aware datetime to lift any ambiguity. However if a timezone naive datetime if given, it will be assumed to contain UTC time.
  • args – args to be passed to the task function
  • kwargs – kwargs to be passed to the task function
Returns:

The Job that was created and scheduled.
schedule_batch(batch: spinach.task.Batch) → Iterable[spinach.job.Job]

Schedule many jobs at once.

Scheduling jobs in batches allows to enqueue them fast by avoiding round-trips to the broker.

Parameters:batch – Batch instance containing jobs to schedule
Returns:The Jobs that were created and scheduled.
start_workers(number: int = 5, queue: str = 'spinach', block: bool = True, stop_when_queue_empty=False, workers_class: Type[spinach.worker.BaseWorkers] = <class 'spinach.worker.ThreadWorkers'>)

Start the worker threads.

Parameters:
  • number – number of workers to launch, each job running uses one worker.
  • queue – name of the queue to consume, see Queues.
  • block – whether to block the calling thread until a signal arrives and workers get terminated.
  • stop_when_queue_empty – automatically stop the workers when the queue is empty. Useful mostly for one-off scripts and testing.
  • worker_class – Class to change the behavior of workers, defaults to threaded workers
stop_workers(_join_arbiter=True)

Stop the workers and wait for them to terminate.

Namespace

Namespaces allow to identify and isolate multiple Spinach engines running on the same Python interpreter and/or sharing the same Redis server.

Having multiple engines on the same interpreter is rare but can happen when using the Flask integration with an app factory. In this case using different namespaces is important to avoid signals sent from one engine to be received by another engine.

When multiple Spinach Engines use the same Redis server, for example when production and staging share the same database, different namespaces must be used to make sure they do not step on each other’s feet.

The production application would contain:

spin = Engine(RedisBroker(), namespace='prod')

While the staging application would contain:

spin = Engine(RedisBroker(), namespace='stg')

Note

Using different Redis database numbers (0, 1, 2…) for different environments is not enough as Redis pubsubs are shared among databases. Namespaces solve this problem.