|
Valkka
1.6.1
OpenSource Video Management
|
H264 decoding is done on the CPU, using the FFmpeg library.
The final operation of interpolation from YUV bitmap into a RGB bitmap of correct (window) size is done on the GPU, using the openGL shading language.
This approach is a nice compromise as it takes some advantage of the GPU by offloading the (heavy) interpolation of (large) bitmaps.
One might think that it would be fancier to do the H264 decoding on the GPU as well, but this is a road to hell - forget about it.
Nv$dia for example, offers H264/5 decoding directly on their GPUs, but then you are dependent on their proprietary implementation, which means that:
So, if you're decoding only one H264 stream, then it might be ok to use proprietary H264 decoding on the GPU (but on the other hand, what's the point if it's only one stream..). If you want to do some serious parallel streaming (like here), invest on CPUs instead.
Other possibilities to transfer the H264 decoding completely to the GPU are (not implemented in Valkka at the moment):
In Valkka, concurrency in decoding and presenting various streams simultaneously is achieved using multithreading and mutex-protected fifos. This works roughly as follows:
Live555 thread (LiveThread) FrameFifo Decoding threads OpenGLFrameFifo OpenGLThread
+-------------+
| |
+---------------------------+ |interpolation|
| rtsp negotiation | -> [FIFO] -> [AVThread] -> |timing |
| frame composition | -> [FIFO] -> [AVthread] -> [Global FIFO] -> |presentation |
| | -> [FIFO] -> [AVthread] -> | |
+---------------------------+ | |
+-------------+
A general purpose "mother" class Thread has been implemented (see multithreading) for multithreading schemes and is inherited by:
To get a rough idea how Live555 works, please see Live555 primer and live555 bridge. The livethread produces frames (class Frame), that are passed to mutex-protected fifos (see queues and fifos).
Between the threads, frames are passed through series of "filters" (see available framefilters). Filters can be used to modify the media packets (say, their timestamps for example) and to produce copying and redirection of the stream. Valkka library filters should not be confused with Live555 sink/source/filters nor with FFmpeg filters - which are completely different things.
For visualization of the media stream plumbings / graphs, we adopt the following notation which you should always use when commenting your python or cpp code:
() == Thread
{} == FrameFilter
[] == FrameFifo queue
To be more informative, we use:
(N. Thread class name: variable name)
{N. FrameFilter class name: variable name)
[N. FrameFifo class name: variable name]
A typical thread / framefilter graph would then look like this:
(1.LiveThread:livethread) --> {2.TimestampFrameFilter:myfilter} --> {3.FifoFrameFilter:fifofilter} --> [4.FrameFifo:framefifo] -->> (5.AVThread:avthread) --> ...
Which means that ..
The whole filter chain from (1) to (4) is simply a callback cascade. Because of this, the execution of LiveThread (1) is blocked, until the callback chain has been completed. The callback chain ends to the "thread border", marked with "-->>". On the "other side" of the thread border, another thread is running independently.
Also, keep in mind the following rule:
In practice, Thread classes manage their own internal FrameFifo and FifoFrameFilter instances, and things become simpler:
* (1.LiveThread:livethread) --> {2.TimestampFrameFilter:myfilter} -->> (3.AVThread:avthread) --> ...
An input framefilter can be requested with AVThread::getFrameFifo()
LiveThread, AVThread and OpenGLThread constructors take a parameter that defines the stack/fifo combination (FrameFifoContext, OpenGLFrameFifoContext).
In the case of LiveThread, the API user passes a separate FrameFilter per each requested stream to LiveThread. That FrameFilter then serves as a starting point for the filter chain. The last filter in the chain is typically FifoFrameFilter, e.g. a filter that feeds the (modified/filtered) decoded frame to a fifo that is then being consumed by AVThread.
For more details, refer to examples, doxygen documentation and the source code itself.
Remember that example we sketched in the github readme page? Using our notation, it would look like this:
(1.LiveThread:livethread) --> {2.TimestampFrameFilter:myfilter}
|
+--> {3.ForkFrameFilter:forkfilter}
| |
| |
through filters, to filesystem <--+ +--->> (6.AVThread:avthread) ---------------------+
|
+--> {7.ForkFrameFilter:forkfilter2}
| |
| |
(10.OpenGLThread:openglthread) <<----------------------------------------------+ +--> .. finally, to analyzing process
feeds the video into various X-windosesSome more miscellaneous details about the architecture:
1.9.1