Processing pipeline hangs
One of the major problems we have in the processing pipeline are the hangs in our hashpipe plugin.
The reason why it hangs is unknown. We are unsure if it is limits in the network, memory limits being reached, or if we are to slow on the I/O transfers on and off the GPU device (i.e., I/O bandwidth). More work needs to be done to identify the stalls.
Currently there is one case of stalling that is reported by the net thread (it may be all cases that this occurs, but not sure). This is a print statement saying NET: HANGING HERE!!!. This is in the net thread when waiting for either the input or output buffer to be free. To get into this state we missed are hanging up stream somewhere and the semaphore is not triggering to release the buffers.