Gracefully manage hangs
Resolving processing pipeline hangs (#1) is a critical task. Our hashpipe plugin implementing a state machine has an unknown transition resulting in a hang. The idea is to add logic to detect that we are hanging and to notify the user (or log the event) and gracefully catch up and continue processing.
As mentioned in the reported hanging issue (#1) one case is identified in the net thread and signaled with the print statement NET: HANGING HERE!!!. An approach might be to exam that loop where the statement is first printed and then determine the ways we can enter that loop and identify if we can exit and begin processing again.