#Processing Multi-Stage Tasks using Firebase Queue

This is an implementation of Pipes and Filter Pattern which is a commonly used Cloud Design Pattern. The most common approach to implement this pattern is via multiple queues as shown in the following figure.

#Pipeline Using Multiple Queues (Ref : MSDN)

The prosed solution here uses a bit different approach than above and use a single queue instead of multiple queues as shown below.

#Pipeline Using a Single Queue

To practically implement a solution like above the common approach would be to maintain a state for each queue message and assign a state for each workers(filters) to filter the messages accordingly before execution. However recently when I was trying out Firebase Realtime Database I found Firebase Queue which is a queue implementation on top of the Firebase Realtime Database and it turned out that it comes with a out of the box solution for maintaining a state (their term is specs) for queue messages as described above. So I worked on a PoC to evaluate this and in my personal opinion it's really cool. But this is not enough to make a dicission so I was looking at various benefits we can achieve here with the usage of Firebase Realtime Database.

#Benefits

• Using firebase realtime behaviour we can easily show the progress of the running tasks on a front-end
• Since it's single queue it's less hazzle to implement and maintain
• Uses push instead of polling to notify workers about new tasks
• Implementation has been made super easy with their library
• Nicely support to be run in a clustered environment distributing workload among nodes

#Use Cases

• In scenarios where REST APIs facilitate time consuming processing heavy back­end functions
• In scenarios where triggered background tasks run for streaming data