Live streaming

Some notes on receiving/sending live streams

Receiving streams

• LiveThread::slots_ is a vector of Connection instances.
  
• Let's assume there is an RTSPConnection instance in a slot. The RTSPConnection::client features a ValkkaRTSPClient instance, that is derived from the live555 RTSPClient class. ValkkaRTSPClient encapsulates all the usual live555 stuff, i.e. response handlers, callbacks, etc.
  
• ValkkaRTSPClient has the target FrameFilter where the frames are being fed. ValkkaRTSPClient::livestatus is a reference to a shared variable that is used by LiveThread::periodicTask to see if the client is alive or if it's been destroyed (calling the destructor happens within live555 callback chains) .. this part is a bit cumbersome, so fresh ideas are welcome.
• Signaling with the "outside world" is done by a periodic live555 task LiveThread::periodicTask that reads the signals sent from outside the thread (see LiveThread::handleSignals)

Sending streams

From the point of view of API users, sending frames happens simply by connecting a source to LiveFifo (see the examples). Behind the scene, frames are sent over the net, by calling LiveFifo::writeCopy - i.e. using the unified approach in Valkka library to handle frames; they are handled in the same way, whether they are passed to the decoder or sent to the screen by using OpenGLFrameFifo (see Library architecture).

Sending frames happens roughly as follows:

• LiveFifo has a reference to the associated LiveThread
• Let's remember that calls to LiveFifo::writeCopy are done outside the running LiveThread that's sending frames. Typically by FifoFrameFilter.
• LiveFifo::writeCopy calls LiveThread::triggerGotFrames which triggers an event using the scheduler and the triggerEvent method (this is the only allowed way to launch events outside the live555 event loop)
• The triggered event corresponds to LiveThread::gotFramesEvent
• LiveThread::gotFramesEvent inserts a new (immediate) task to the live555 event loop, namely LiveThread::readFrameFifoTask
• LiveThread::readFrameFifoTask (i) calls LiveThread::handleFrame and (ii) re-schedules itself if there are more than one frame in the fifo
• LiveThread::handleFrame checks the slot of the outgoing frame, takes the corresponding Outbound instance and calls Outbound::handleFrame

SDP Streams

Streams sent directly to UDP ports, as defined in an SDP file ("SDP" streams)

• SDPOutbound has a set of Stream instances in Outbound::streams. There is a Stream instance per media substream.
• Stream instances encapsulate the usual live555 stuff per substream: RTPSink, RTCPInstance, Groupsock, FramedSource, etc.
• Each Stream has also a reference to LiveFifo and Stream::buffer_source which is a BufferSource instance.
• BufferSource is a subclass of live555 FramedSource that is used for sending frames.
• BufferSource has it's own internal fifo BufferSource::internal_fifo. This fifo is used by BufferSource::doGetNextFrame, as it should in live555, i.e. if there are no frames left in the fifo, then BufferSource::doGetNextFrame exits immediately, otherwise it calls FramedSource::afterGetting and if necessary, re-schedules itself
• To summarize, the call chain looks like this: LiveThread::handleFrame => Outbound::handleFrame => Stream::handleFrame (transfers the frame from LiveFifo to BufferSource::internal_fifo) => BufferSource::handleFrame => FramedSource::afterGetting
• The Live555 filterchain looks like this:

  buffer_source  =new BufferSource(env, fifo, 0, 0, 4); // nalstamp offset: 4
  terminal       =H264VideoStreamDiscreteFramer::createNew(env, buffer_source);
  
  sink           = H264VideoRTPSink::createNew(env,rtpGroupsock, 96);
  
  sink->startPlaying(*(terminal), this->afterPlaying, this);
  

• Frames flow like this: BufferSource => H264VideoStreamDiscreteFramer => H264VideoRTPSink

  This diagram, where "{}" means enclosing object will help you to understand this:

  BufferSource (.., fifo) {
     - Live555 FramedSource class with method "doGetNextFrame"
     - Recycles frames back to fifo (in doGetNextFrame)
  }
  
  Stream {
     RTPSink,
     RTCPInstance,
     Groupsock,
     BufferSource *buffer_source,
     FrameFifo &fifo,
     
     methods:
         startPlaying
             - issues startPlaying on the live555 
               event loop (for the internal live555 
               filterchain defined in the Stream subclasses)
         afterPlaying
             - a live555 callback
  }
  
  H264 : public Stream {
     - Instantiates buffer_source = new BufferSource (.., fifo)
     - Creates the live555 internal filterchain
  }
  

RTSP Server

Some misc. observations / code walkthrough:

• Inheritance: H264ServerMediaSubsession => ValkkaServerMediaSubsession => OnDemandServerMediaSubsession
• H264ServerMediaSubsession is placed into LiveThread::server (which is RTSPServer instance)
• H264ServerMediaSubsession::createNewStreamSource is called from OnDemandServerMediaSubsession::sdpLines when a DESCRIBE request is done on the RTSPServer
• OnDemandServerMediaSubsession::closeStreamSource calls Medium::close on inputsource it obtained from H264ServerMediaSubsession::createNewStreamSource, when TEARDOWN is issued on the RTSPServer.
• .. this basically closes our BufferSource instance (that's using a fifo), so we need to check if BufferSource has been destructed
• .. this is done using a reference variable that's given to BufferSource (ValkkaServerMediaSubsession::source_alive). That variable is checked before calling BufferSource::handleFrame.
• There is a tricky inner event loop that generates sps/pps info into the sdp string at H264ServerMediaSubsession::getAuxSDPLine .. (as this has been hacked from H264VideoFileServerMediaSubsession). That can get stuck sometimes (!)
• Media sessions can be removed from the server with RTSPServer::removeServerMediaSession(media_session)

To get this into one's head, let's take a look at this diagram. "{}" means enclosing object:

RTSPServer {

   H264ServerMediaSubsession {
       - member "fifo" is a reference to FrameFifo
   
       - method : createNewStreamSource
           - creates buffer_source = BufferSource (.., fifo)
           - creates H264VideoStreamDiscreteFramer(buffer_source)
           => returns to RTPServer: H264VideoStreamDiscreteFramer(buffer_source)
           
       - inherited method sdpLines
       - inherited method closeStreamSource
   }
}