A RTSP Server Based on Qt Framework and LIVE555

※ The code corresponding to this post is here, the "Server" folder.

    Recent years, the streaming technologies have been more and more widely used : the drone remote control,  the live broadcasting and the video conference.. and so forth. There are several protocols, like HTTP Live Streaming (HLS), WebRTC..etc. But almost all the protocols are based on Real Time Streaming Protocol(RTSP). Thus, RTSP is the zeroth choice for the applications just integrated with the streaming function. `

   One of the famous RTSP libraries is LIVE555. In this post, I  give the key steps for how to set a camera RTSP server on Qt Framework. Why I use Qt ? It is Qt able to reduce the work of Camera grabbing and window drawing. We could pay more attention on LIVE555 without the disturbing from the other issues.

  The video format in here is H264, for its low data-rate, acceptable image-quality and the mature codec implementations, H264 is widely used in streaming. The most famous software implementation of H264 is x264, so I adopt it for this post.


零. Build the x264 library

    ※ I have prepared this library in the git-repository.
    My environment is Windows, Thus I use Mingw-w64 to build x264. You could refer to here to know to install it. After it has been done, download, untar x264 source code and put it in the Mingw-w64 user's folder.
My configuration arguments are :

 ./configure --host=mingw64 --prefix=$PWD/build --enable-shared

 Them make & make install, the x264 dll library would be in the folder build. We need that as the h264 encoder.


一. The Camera Frame Grabber

   We write a class named CameraFramGabber to fetch the image from the camera. That inherits from the class QAbstractVideoSurface, which requires its subclass implemented the 2 pure-virtual functions, present and supportedPixelFormats. While the frame has been fetched, the signal UpdateFrame , which contains the image, would be sent from the present function.

class CameraFrameGrabber : public QAbstractVideoSurface{
:
public:
 virtual bool present(const QVideoFrame &frame) Q_DECL_OVERRIDE;

 virtual QList<QVideoFrame::PixelFormat>
  supportedPixelFormats(QAbstractVideoBuffer::HandleType type) const 
 Q_DECL_OVERRIDE;
:
public slots:
:
 void UpdateFrame(const QImage image);
:
};

bool CameraFrameGrabber::present(const QVideoFrame &frame)
{
 //qDebug() << "grabbed pixelFormat is " << cloneFrame.pixelFormat();
 QVideoFrame cloneFrame(frame);

 cloneFrame.map(QAbstractVideoBuffer::ReadOnly);

 QImage grabbed_image;

 if(QVideoFrame::Format_YUYV == cloneFrame.pixelFormat())
 {
  QByteArray temp;
  temp.resize(cloneFrame.width() * cloneFrame.height() * 3);

  YUYVtoRGB(cloneFrame.width(), cloneFrame.height(),
      cloneFrame.bits(), (unsigned char*)temp.data());

  grabbed_image =  QImage((unsigned char*)temp.data(), cloneFrame.width(),

        cloneFrame.height(), cloneFrame.width() * 3, QImage::Format_RGB888).copy();
 }
#if(0)
 else if(QVideoFrame::Format_RGB32 == cloneFrame.pixelFormat())
 {
  grabbed_image = QImage(cloneFrame.bits(), cloneFrame.width(),
          cloneFrame.height(), cloneFrame.width() * 4, QImage::Format_RGB32).copy();
 }
#endif
 cloneFrame.unmap();

 emit FrameUpdated(grabbed_image);

 return true;
}

  The VideoFrame format at present, would be one of the formats returned from supportedPixelFormats :

QList<QVideoFrame::PixelFormat>
 CameraFrameGrabber::supportedPixelFormats(
  QAbstractVideoBuffer::HandleType type) const
{
 if (type != QAbstractVideoBuffer::NoHandle)
  return QList<QVideoFrame::PixelFormat>();

 QList<QVideoFrame::PixelFormat> pixel_fmt;
#if(1)
 pixel_fmt.append(QVideoFrame::Format_RGB24);
 pixel_fmt.append(QVideoFrame::Format_RGB32);
#endif
 pixel_fmt.append(QVideoFrame::Format_YUYV);

 return pixel_fmt;
}

  We set the output format as  QVideoFrame::Format_YUYV in the function ChangeCameraSetting. it brings we only need to prepare one set of format conversion(YUYV to RGB24).

void CameraFrameGrabber::ChangeCameraSetting(QCamera::State state)
{
	:
	QCameraViewfinderSettings viewfinderSettings;
	viewfinderSettings = m_p_camera->viewfinderSettings();
	:
	qDebug() << m_p_camera->supportedViewfinderPixelFormats();
	viewfinderSettings.setPixelFormat(QVideoFrame::Format_YUYV);
	:

    However,  supporting with only one conversion may make the fetching inefficient : if the camera supports with RGB32, the YUYV to RGB24 conversion consumes more CPU resource than RGB32 to RGB24. The reason why we choose YUYV format is, the format is the most compatible : almost every USB-camera supports with this format.

※ My YUYVtoRGB function assumes the camera resolution is a multiple of four. If it is not, it may lead the application crashed or the image slanted .


二. The H264 Encoder and H264 NAL Factory

 Though currently there is only one H264 encoder, but it may change to the other encoders( like vEnc, OpenH264, DirectShow..etc) in the future. Thus we prepare a H264 virtual class to unify the calling interfaces. That is :

class H264Encoder
{
public:
 H264Encoder(void){}
 virtual ~H264Encoder(void){}

public :
 virtual int Init(int width, int height) = 0;
 virtual void Close(void) = 0;

 virtual QQueue<int> Encode(unsigned char *p_frame_data, 
      QQueue<QByteArray> &ref_queue) = 0;

};

    Notice that the H264 property : one frame may be encoded into multi-NAL units, so we use a queue to store the HAL data and data length.
  The class X264Encoder, using the interfaces of H264Encoder, is our H264 Encoder class directly calling x264 API. I leave out the explanation of X264Encoder, for the x264 calling examples are very easy to be found.
    There is a class H264NalFactory, containing X264Encoder. It makes the encoder is easily to be replaced. Beside, because the image producer(the frame grabber) and the consumer (the RTSP Server) are in the different thread, thus we need a synchronizer to control the race condition.

class H264NalFactory : public QObject
{
 Q_OBJECT
public:
 H264NalFactory(QObject *p_frame_grabber);
 ~H264NalFactory(void) Q_DECL_OVERRIDE;

public:
 int GetH264Nal(unsigned char *p_data, int buffer_size);

public slots:
 void SetResolution(QSize resolution);
 int Encode(QImage image);
 
:
 H264Encoder *m_p_h264_encoder;
 QQueue<QByteArray> m_nal_queue;
};

Here we use a queue to contain the NAL data.

The two key functions are as below, as the producer and consumer interfaces, and we use a semaphore for the synchronization.

int H264NalFactory::Encode(QImage image)
{
 :
 QQueue<int> nal_size_queue;
 
 nal_size_queue = m_p_h264_encoder->Encode(image.bits(), m_nal_queue);
 m_semaphore.release(nal_size_queue.size());
 :
}

int H264NalFactory::GetH264Nal(unsigned char *p_data, int buffer_size)
{
 :
 m_semaphore.tryAcquire(1, MAX_WAIT_TIME_IN_MS);

 if(0 == m_nal_queue.size())
   return 0;
 :
}

三. Prepare the LIVE555 libraries.

    The LIVE555 libaries could be download here. Despite the libraries (ver. 1.00) have been prepared in my repository, I still suggest you to download and build it by yourself.
    We need to create four Qt projects as static library : BasicUsageEnvironment, groupsock, liveMedia and UsageEnvironment, those are corresponding to the LIVE555 folder names, respectively. If your  LIVE555 is later than ver.0.98, It is necesary to disable the OPENSSL function in the liveMedia.

in liveMedia.pro :

:
CONFIG += c++11

DEFINES += NO_OPENSSL
:

四. Implement the RTSP Server.

    Create a project file, we name QtRTSPServer, to contain the build information. It is a very typical QtCreator .pro file.  I omit the most explanation. One thing I need to mention is as the below lines :

LIBS += $$LIVE555_PATH/groupsock/$$BUILD_MODE/groupsock.lib
LIBS += $$LIVE555_PATH/liveMedia/$$BUILD_MODE/liveMedia.lib

PRE_TARGETDEPS += $$LIVE555_PATH/BasicUsageEnvironment/$$BUILD_MODE/BasicUsageEnvironment.lib
PRE_TARGETDEPS += $$LIVE555_PATH/UsageEnvironment/$$BUILD_MODE/UsageEnvironment.lib
PRE_TARGETDEPS += $$LIVE555_PATH/groupsock/$$BUILD_MODE/groupsock.lib
PRE_TARGETDEPS += $$LIVE555_PATH/liveMedia/$$BUILD_MODE/liveMedia.lib

LIBS += Ws2_32.lib

After adding those lines, run qmake, (with make argument install) it will let QtCreator build the dependencies libraries first.


   Below heavily refer to testOnDemandRTSPServer.cpp, which is our RSTP server's prototype.

   As testOnDemandRTSPServer, our server on a thread, thus it is a subclass of QThread with some members related to LIVE555, is declaration is :

class CameraRTSPServer : public QThread
{
 Q_OBJECT
public:
 explicit CameraRTSPServer(QObject *p_grabber, 
  unsigned short port = 554);

 ~CameraRTSPServer(void) Q_DECL_OVERRIDE;

 void Stop(void);

protected:
 virtual void run(void) Q_DECL_OVERRIDE;
private:
 int CreateLive555Objects(void);
 void DestroyLive555Objects(void);
 void AnnounceStream(RTSPServer* p_rtsp_server, 

          ServerMediaSession *m_p_media_session);

private:
 H264NalFactory *m_p_h264_nal_factory;

 unsigned short m_port;
 UsageEnvironment *m_p_env;

 TaskScheduler *m_p_scheduler;
 RTSPServer *m_p_rtsp_server;
 ServerMediaSession *m_p_server_media_session;
 volatile char m_is_to_stop_scheduler;

};

    The LIVE555 members initialization/destroy have been organized as the function CreateLive555Objects and DestoryLive555Objects, brings the run function simple:

void CameraRTSPServer::run(void)
{
 if(0 > CreateLive555Objects())
  return ;

 AnnounceStream(m_p_rtsp_server, 
  m_p_server_media_session);

 m_is_to_stop_scheduler = 0;
 m_p_scheduler->doEventLoop(&m_is_to_stop_scheduler);

 DestroyLive555Objects();
}

    And we need a bridge connecting the RTSP server with the H264NalFactory. Refer to testOnDemandRTSPServer.cpp, about line 101.

// A H.264 video elementary strea
  {
    char const* streamName = "h264ESVideoTest";
    char const* inputFileName = "test.264";
    ServerMediaSession* sms = ServerMediaSession::createNew(

          *env, streamName, streamName, descriptionString);
    sms->addSubsession(H264VideoFileServerMediaSubsession::createNew(

          *env, inputFileName, reuseFirstSource));
    rtspServer->addServerMediaSession(sms);

    announceStream(rtspServer, sms, streamName, inputFileName);
  }

The class H264VideoFileServerMediaSubsession :

class H264VideoFileServerMediaSubsession: public FileServerMediaSubsession {
public:
  static H264VideoFileServerMediaSubsession* createNew(UsageEnvironment& env,

           char const* fileName, Boolean reuseFirstSource);

  // Used to implement "getAuxSDPLine()":
  void checkForAuxSDPLine1();
  void afterPlayingDummy1();

protected:
  H264VideoFileServerMediaSubsession(UsageEnvironment& env,
          char const* fileName, Boolean reuseFirstSource);
      // called only by createNew();
  virtual ~H264VideoFileServerMediaSubsession();

  void setDoneFlag() { fDoneFlag = ~0; }

protected: // redefined virtual functions
  virtual char const* getAuxSDPLine(RTPSink* rtpSink, FramedSource* inputSource);
  virtual FramedSource* createNewStreamSource(unsigned clientSessionId,
           unsigned& estBitrate);
  virtual RTPSink* createNewRTPSink(Groupsock* rtpGroupsock,
           unsigned char rtpPayloadTypeIfDynamic, FramedSource* inputSource);

private:
  char* fAuxSDPLine;
  char fDoneFlag; // used when setting up "fAuxSDPLine"
  RTPSink* fDummyRTPSink; // ditto
};

※ the class FileServerMediaSubsession inherits from the class OnDemandServerMediaSubsession.

    It is : there is a class, named H264VideoFileServerMediaSubsession derived from the class OnDemandServerMediaSubsession, set into the RTSP server.

    For our class to be set into the RTSP Server, is the class H264MediaSubsession, which is pretty like class H264VideoFileServerMediaSubsession :

class H264MediaSubsession : public OnDemandServerMediaSubsession
{
public:
 H264MediaSubsession(UsageEnvironment& ref_env, 
 bool is_reuse_first_source, void *p_grabber);
 virtual ~H264MediaSubsession(void);
 void setDoneFlag() { m_done_flag = 1; }

protected:
 virtual char const* getAuxSDPLine(RTPSink* p_rtp_sink, 

       FramedSource *p_input_source);
 
 virtual FramedSource* createNewStreamSource(unsigned client_session_id,
       unsigned &ref_estimate_bitrate);

 virtual RTPSink* createNewRTPSink(Groupsock* rtpGroupsock, 

       unsigned char rtpPayloadTypeIfDynamic, FramedSource* inputSource);

public:
 void checkForAuxSDPLine1(void);
 void afterPlayingDummy1(void);

private:
 void *m_p_grabber;
 char* m_p_AuxSDPLine;
 char m_done_flag;
 RTPSink *m_p_dummy_Sink;

};

    The function createNewStreamSource is the major difference between our H264MediaSubsession and the H264VideoFileServerMediaSubsession. Here we use the H264VideoStreamDiscreteFramer as the framer, instread of the H264VideoStreamFramer :

FramedSource* H264MediaSubsession::createNewStreamSource(unsigned client_session_id,
               unsigned &ref_estimate_bitrate)
{
 ref_estimate_bitrate = 2 * 1024 * 1024;

 CameraH264Source *p_source;
 p_source = CameraH264Source::createNew(envir(),
  client_session_id, m_p_grabber);

 return H264VideoStreamDiscreteFramer::createNew(envir(), p_source);
}

※ How H264VideoFileServerMediaSubssesion works, your could refer to here (in 漢文).

    Now,  refer to ByteStreamFileSource.cpp, which is used in the H264VideoFileServerMediaSubssesion. We need to implement our own source class, named CameraH264Source.

    As the H264VideoFileServerMediaSubssesion declaration indicates, the source must be a subclass of the FramedSource. In our case, the source class is named CameraH264Source, its declaration as below.

#ifndef CameraH264Source_H
#define CameraH264Source_H

#include "FramedSource.hh"
#include "H264NalFactory.h"

class CameraH264Source : public FramedSource
{
public:
 static CameraH264Source *createNew(UsageEnvironment &ref_env, 

       unsigned client_session_id, void *p_h264_nal_factory);

protected:
 CameraH264Source(UsageEnvironment &ref_env, 

       H264NalFactory *m_p_h264_nal_factory);
 ~CameraH264Source(void);

public:
 virtual void doGetNextFrame();
 virtual unsigned int maxFrameSize() const;

public:
 virtual Boolean isH264VideoStreamFramer(void) const {
  return true; // default implementation
 }


protected:
 void DeliverFrame(void);

private:
 H264NalFactory *m_p_h264_nal_factory;
 unsigned char *m_p_frame_buffer;
};

#endif // CameraH264Source_H

※ VERY important : the isH264VideoStreamFramer function (declared in class MediaSource, the parent of the class FramedSource ) must be overridden, to denote the data is H264 data.

    The most importance of the class CameraH264Source is the interface (virtual) function doGetNextFrame, where the H264 NAL data would be copied to the FramedSource buffer. The calling of the function FramedSource::afterGetting must be in the end of doGetNextFrame, it will schedule when is the next time doGetNextFrame will be called.
    Notice that the H264 NAL head (0x000001 or 0x00000001) must be removed, or the H264VideoStreamDiscreteFramer will keep complaining and bring the streaming fail.

    ※The function DeliverFrame is the implementation of doGetNextFrame.

void CameraH264Source::DeliverFrame(void)
{
 FramedSource::fFrameSize = 0;
 FramedSource::fNumTruncatedBytes = 0;

 unsigned int h264_data_length;

 h264_data_length = (unsigned int)m_p_h264_nal_factory->GetH264Nal(

       m_p_frame_buffer, ONE_FRAME_BUFFER_SIZE);

 if(0 >= h264_data_length)
  return ;


 int trancate;
 trancate = 0;

 {
  char trancate_4[4] = {0x00, 0x00, 0x00, 0x01};
  char trancate_3[3] = {0x00, 0x00, 0x01};

  if(0 == memcmp(&m_p_frame_buffer[0], &trancate_4[0], sizeof(trancate_4)))
  {
   trancate = 4;
  }
  else if(0 == memcmp(&m_p_frame_buffer[0], &trancate_3[0], sizeof(trancate_3)))
  {
   trancate = 3;
  }
 }

 FramedSource::fFrameSize = h264_data_length;
 FramedSource::fNumTruncatedBytes = 0;

 if(h264_data_length > FramedSource::fMaxSize)
 {
  FramedSource::fFrameSize = FramedSource::fMaxSize;
  FramedSource::fNumTruncatedBytes =
    h264_data_length - FramedSource::fMaxSize;
 }/*if */

 gettimeofday(&(FramedSource::fPresentationTime), nullptr);

 memmove(FramedSource::fTo, m_p_frame_buffer + trancate,
  FramedSource::fFrameSize - trancate);

 FramedSource::afterGetting(this);
}

To here, the implementation of the RTSP server has completed.


Now, run the QtSTPServer we done, it will show the RTSP URL in the console.

:
"h264" stream
Play this stream using the URL "rtsp://192.168.13.183:8554/h264"

※In Creator, you had better to add install in the make arguments, in Project -> Build & Run->Build Settings -> Build Steps -> Make -> Make arguments. It will automatically build the dependencies(LIVE555), and copy the necessary dll (x264) to the working folder.


五. VLC player as the RTSP Client.

    There are some players support with RTSP client, VLC is one. We run the VLC in the same computer, and input the streaming URL in that ( VLC menu -> Open Network Stream).

※Beware : if your camera FPS is low( < 10), you need to increase the caching value.

Theoretically, the QtRTSPServer will be streaming to the VLC player, as the below figures show.

Apparently, there is color shift.

My next post would discuss how to replaces the x264 encoder as the Intel QSV encoder. 

Reference.
https://stackoverflow.com/questions/19427576/live555-x264-stream-live-source-based-on-testondemandrtspserver