GStreamer

文件

Pipeline

基本介紹

基本流程:

Source -> 中間處理 -> Sink

範例:

# 以 local 的 sample.mp4 當作媒體來源
# 把影像轉成 H264 並包在 RTP 封包裡面
# 以 UDP 送出到指定的位置
$ gst-launch-1.0 filesrc location=sample.mp4 ! qtdemux ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5566

產生 Pipeline 流程圖

使用 GST_DEBUG_DUMP_DOT_DIR 來指定 DOT 描述語言的輸出資料夾。

$ env GST_DEBUG_DUMP_DOT_DIR=/tmp/dot/ gst-launch-1.0 filesrc location=sample.mp4 ! qtdemux ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5566
$ sudo pacman -S xdot
$ xdot /tmp/dot/0.00.00.023250085-gst-launch.PAUSED_PLAYING.dot

檢查目前系統的支援

$ gst-inspect-1.0
...

$ gst-inspect-1.0 x264
Plugin Details:
  Name                     x264
  Description              libx264-based H264 plugins
  Filename                 /usr/lib/gstreamer-1.0/libgstx264.so
  Version                  1.12.1
  License                  GPL
  Source module            gst-plugins-ugly
  Source release date      2017-06-20
  Binary package           GStreamer Ugly Plugins (Arch Linux)
  Origin URL               http://www.archlinux.org/

  x264enc: x264enc

  1 features:
  +-- 1 elements

除錯

可以利用 GST_DEBUG 設定全域的資訊層級或是特定元件的資訊層級。

$ env GST_DEBUG=4 gst-launch-1.0 filesrc location=sample.mp4 ! qtdemux ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5566

Source

稍微列出一些常見的 Source:

  • 第三方應用程式(用於利用 GStreamer 撰寫自己的程式): appsrc

  • RTSP: rtspsrc

  • UDP: udpsrc

自己系統目前支援的 Source 的完整清單:

$ gst-inspect-1.0 | rg src

Sink

稍微列出一些常見的 Source:

  • 第三方應用程式(用於利用 GStreamer 撰寫自己的程式): appsink

  • 檔案: filesink

  • UDP: udpsink

自己系統目前支援的 Sink 的完整清單:

$ gst-inspect-1.0 | rg src

Binding

GObject Introspection

Repo:

https://github.com/GNOME/gobject-introspection

  • GIR: XML 格式,用於 GObject Introspection,提供共同的結構來存取 API,這格式的設計適用於各種程式語言,利用 namespace 來區分 core、language、library 的功能

  • Typelib: binary 格式的 GIR,設計原則是快、使用少量記憶體、足夠讓各個程式語言沒有原始碼的狀況下做出 binding

  • 工具:
    • g-ir-scanner: 用於產生 GIR,可以爬 headers/GTK-Doc/libraries

    • g-ir-compiler: GIT 轉 Typelib

    • g-ir-generate: Typelib 轉 GIR

    • g-ir-annotation-tool: 從程式碼截出 annotation

    • g-ir-doc-tool: 從 GIR 產生 API 文件

Python

Repo:

https://github.com/GStreamer/gst-python

利用 PyGObject 做成的 binding, 涵蓋大部分 GStreamer 的功能。 PyGObject 文件

安裝:

# 系統安裝
$ sudo pacman -S gst-python

# 用 pip 安裝
$ pip install "git+https://git.gnome.org/browse/pygobject@3.24.1"

使用:

import gi

# 在 import Gst 之前要先指定版本限制。
gi.require_version('Gst', '1.0')

from gi.repository import Gst

Gst.init(None)

# 可以使用 GST_DEBUG 環境變數來開啟除錯訊息,
# 或是使用下面兩行:
# Gst.debug_set_active(True)
# Gst.debug_set_default_threshold(5)

Rust

Repo:

https://github.com/sdroege/gstreamer-rs

Docs:

https://sdroege.github.io/rustdoc/gstreamer/gstreamer/

利用 GIR 從 GStreamer 專案 提供的 GObject-Introspection API metadata 來自動產生 binding, 已經涵蓋大部分 GStreamer 的功能。

GTK 系列 Rust binding 網站

核心元件

core elements

Pipeline Branch - tee

tee 會從原本的 Pipeline 切出 Branch, 範例:

$ gst-launch-1.0 filesrc location=song.ogg ! decodebin ! \
    tee name=t ! queue ! audioconvert ! audioresample ! autoaudiosink \
            t. ! queue ! audioconvert ! goom ! videoconvert ! autovideosink

寫到指定的 File Descriptor - fdsink

$ gst-launch-1.0 -q videotestsrc pattern=ball ! video/x-raw,height=320,width=240,framerate=30/1,format=RGB ! fdsink | ffplay -f rawvideo -pixel_format rgb24 -video_size 240x320 -i -

從指定的 File Descriptor 讀入 - fdsrc

$ echo "Hello GStreamer" | gst-launch-1.0 -v fdsrc ! fakesink dump=true
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
00000000 (0x7f9fd8004890): 48 65 6c 6c 6f 20 47 53 74 72 65 61 6d 65 72 0a  Hello GStreamer.
Got EOS from element "pipeline0".
Execution ended after 0:00:00.000123950
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

input-selector

identity

funnel

其他元件

TCP

  • tcpserversrc

  • tcpserversink

  • tcpclientsrc

  • tcpclientsink

ipcpipeline

可以把 pipeline 拆成數個小 pipeline 跑在不同的 process 裡, 彼此間仍能溝通(基於 fd), 藉此能處理使用特定元件需要較高權限的狀況, 提昇安全性, 架構像是 master/slave, pipeline 狀態可以傳遞。

顯示 plugin 被放入 blacklist 的原因

Pipeline 除錯

$ git clone https://github.com/GStreamer/gst-devtools
$ cd gst-devtools
$ meson . build
$ cd build
$ ninja

$ validate/tools/gst-validate-1.0 playbin file:///path/to/a/video.mkv
Starting pipeline
Pipeline started
     issue : EOS events that are part of the same pipeline 'operation' should have the same seqnum
             Detected on <matroskademux0:video_0>
             Detected on <multiqueue0:sink_0>
             Detected on <matroskademux0:audio_0>
             Detected on <multiqueue0:sink_1>
             Detected on <multiqueue0:src_1>
             Detected on <ac3parse0:sink>
             Detected on <ac3parse0:src>
             Detected on <capsfilter2:sink>
             Detected on <capsfilter2:src>
             Detected on <a52dec0:sink>
             Detected on <a52dec0:src>
             Detected on <inputselector1:sink_0>
             Detected on <inputselector1:src>
             Detected on <audiotee:sink>
             Detected on <audiotee:src_0>
             Detected on <streamsynchronizer0:sink_1>
             Detected on <multiqueue0:src_0>
             Detected on <h264parse0:sink>
             Detected on <h264parse0:src>
             Detected on <capsfilter0:sink>
             Detected on <capsfilter0:src>
             Detected on <vaapidecode0:sink>
             Detected on <vaapidecode0:src>
             Detected on <vaapi-queue:sink>
             Detected on <vaapi-queue:src>
             Detected on <capsfilter1:sink>
             Detected on <capsfilter1:src>
             Detected on <vaapipostproc0:sink>
             Detected on <vaapipostproc0:src>
             Detected on <inputselector0:sink_0>
             Detected on <inputselector0:src>
             Detected on <streamsynchronizer0:sink_0>
             Detected on <streamsynchronizer0:src_0>
             Detected on <vdconv:sink>
             Detected on <vdconv:src>
             Detected on <deinterlace:sink>
             Detected on <streamsynchronizer0:src_1>
             Detected on <deinterlace:src>
             Detected on <aqueue:sink>
             Detected on <vqueue:sink>
             Detected on <aqueue:src>
             Detected on <conv:sink>
             Detected on <conv:src>
             Detected on <resample:sink>
             Detected on <resample:src>
             Detected on <pulsesink2:sink>
             Detected on <vqueue:src>
             Detected on <conv:sink>
             Detected on <conv:src>
             Detected on <scale:sink>
             Detected on <scale:src>
             Detected on <vaapisink0:sink>
             Description : when events/messages are created from another event/message, they should have their seqnums set to the original event/message seqnum

   warning : received the same caps twice
             Detected on <h264parse0:sink>
             Detected on <ac3parse0:sink>
             Detected on <a52dec0:sink>

Issues found: 2

=======> Test PASSED (Return value: 0)

$ validate/tools/gst-validate-1.0 audiotestsrc ! input-selector ! capsfilter name=capsfilter caps=video/x-raw ! fakesink
Starting pipeline
Pipeline started
   warning : EOS received without segment event before
             Detected on <audiotestsrc0:src>
             Detected on <inputselector0:sink_0>
             Detected on <inputselector0:src>
             Detected on <capsfilter:sink>
             Detected on <capsfilter:src>
             Description : A segment event should always be sent before data flow EOS being some kind of data flow, there is no exception in that regard

  critical : a NOT NEGOTIATED message has been posted on the bus.
             Detected on <pipeline0>
             Details : Error message posted by: audiotestsrc0
                      Caps negotiation failed starting from pad 'capsfilter:sink' as the QUERY_CAPS returned EMPTY caps for the following possible reasons:
                         -> Downstream caps struct 0 name 'video/x-raw' differs from filter caps struct 0 name 'audio/x-raw'. The exact reason could not be determined but here is the gathered information:
                      - capsfilter:sink last query caps filter: audio/x-raw, format=(string){ S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE, S8, U8 }, layout=(string)interleaved, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 2147483647 ]
                      - capsfilter:sink possible caps (as returned by a query on it without filter): video/x-raw

             dotfile : no dotfile produced as GST_DEBUG_DUMP_DOT_DIR is not set.
             backtrace :
               gst_debug_get_stack_trace (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b56683c05)
               gst_validate_report_new (/home/dv/zone/gst-devtools/build/validate/gst/validate/libgstvalidate-1.0.so.0.1202.0:0x7f3b56974df0)
               gst_validate_report_valist (/home/dv/zone/gst-devtools/build/validate/gst/validate/libgstvalidate-1.0.so.0.1202.0:0x7f3b5696559d)
               gst_validate_report (/home/dv/zone/gst-devtools/build/validate/gst/validate/libgstvalidate-1.0.so.0.1202.0:0x7f3b56965ab6)
               _bus_handler (/home/dv/zone/gst-devtools/build/validate/gst/validate/libgstvalidate-1.0.so.0.1202.0:0x7f3b56969021)
               ffi_call_unix64 (/usr/lib/libffi.so.6.0.4:0x7f3b543711c4)
               ffi_call (/usr/lib/libffi.so.6.0.4:0x7f3b54370c26)
               g_cclosure_marshal_generic (/usr/lib/libgobject-2.0.so.0.5200.3:0x7f3b563d86a5)
               g_closure_invoke (/usr/lib/libgobject-2.0.so.0.5200.3:0x7f3b563d7ea9)
               ?? (/usr/lib/libgobject-2.0.so.0.5200.3:0x7f3b563ea4aa)
               g_signal_emit_valist (/usr/lib/libgobject-2.0.so.0.5200.3:0x7f3b563f2c81)
               g_signal_emit (/usr/lib/libgobject-2.0.so.0.5200.3:0x7f3b563f369b)
               gst_bus_sync_signal_handler (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5665c964)
               gst_bus_post (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5665cc4a)
               ?? (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b56672bd8)
               ?? (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5664e6da)
               gst_element_post_message (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b56675a2c)
               ?? (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5664ea27)
               ?? (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5669c4ee)
               ?? (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5664c094)
               gst_bus_post (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5665cc20)
               ?? (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b56672bd8)
               gst_element_post_message (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b56675a2c)
               gst_element_message_full_with_details (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b5667610c)
               ?? (/usr/lib/libgstbase-1.0.so.0.1202.0:0x7f3b5310c047)
               ?? (/usr/lib/libgstreamer-1.0.so.0.1202.0:0x7f3b566c407f)
               ?? (/usr/lib/libglib-2.0.so.0.5200.3:0x7f3b561274ac)
               ?? (/usr/lib/libglib-2.0.so.0.5200.3:0x7f3b56126ae1)
               start_thread (/usr/lib/libpthread-2.25.so:0x7f3b54de9045)
               __clone (/usr/lib/libc-2.25.so:0x7f3b55a61f0b)




==== Got criticals. Return value set to 18 ====
     Critical error Error message posted by: audiotestsrc0
 Caps negotiation failed starting from pad 'capsfilter:sink' as the QUERY_CAPS returned EMPTY caps for the following possible reasons:
    -> Downstream caps struct 0 name 'video/x-raw' differs from filter caps struct 0 name 'audio/x-raw'. The exact reason could not be determined but here is the gathered information:
 - capsfilter:sink last query caps filter: audio/x-raw, format=(string){ S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE, S8, U8 }, layout=(string)interleaved, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 2147483647 ]
 - capsfilter:sink possible caps (as returned by a query on it without filter): video/x-raw


Issues found: 2
Returning 18 as errors were found

=======> Test FAILED (Return value: 18)

硬體加速 (Hardware Acceleration)

VA-API

VA-API 是一套開源的 Library 和 API 定義(由 Intel 主推的), 提供一個抽象層來使用硬體處理影像, 硬體廠商可以為這套 API 實做自己的硬體支援。

$ sudo pacman -S libva libva-utils
$ sudo pacman -S libva-intel-driver   # for Intel
$ vainfo
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.40 (libva )
vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 1.8.2
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :     VAEntrypointVLD
      VAProfileMPEG2Simple            :     VAEntrypointEncSlice
      VAProfileMPEG2Main              :     VAEntrypointVLD
      VAProfileMPEG2Main              :     VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:     VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:     VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:     VAEntrypointEncSliceLP
      VAProfileH264Main               :     VAEntrypointVLD
      VAProfileH264Main               :     VAEntrypointEncSlice
      VAProfileH264Main               :     VAEntrypointEncSliceLP
      VAProfileH264High               :     VAEntrypointVLD
      VAProfileH264High               :     VAEntrypointEncSlice
      VAProfileH264High               :     VAEntrypointEncSliceLP
      VAProfileH264MultiviewHigh      :     VAEntrypointVLD
      VAProfileH264MultiviewHigh      :     VAEntrypointEncSlice
      VAProfileH264StereoHigh         :     VAEntrypointVLD
      VAProfileH264StereoHigh         :     VAEntrypointEncSlice
      VAProfileVC1Simple              :     VAEntrypointVLD
      VAProfileVC1Main                :     VAEntrypointVLD
      VAProfileVC1Advanced            :     VAEntrypointVLD
      VAProfileNone                   :     VAEntrypointVideoProc
      VAProfileJPEGBaseline           :     VAEntrypointVLD
      VAProfileJPEGBaseline           :     VAEntrypointEncPicture
      VAProfileVP8Version0_3          :     VAEntrypointVLD
      VAProfileVP8Version0_3          :     VAEntrypointEncSlice
      VAProfileHEVCMain               :     VAEntrypointVLD
      VAProfileHEVCMain               :     VAEntrypointEncSlice

VDPAU

Intel Media SDK

OMX

Repo:

https://github.com/GStreamer/gst-omx

開發小技巧

檢查是否可以收到串流

gst-launch-1.0 udpsrc port=9327 ! fakesink dump=true

檢查串流內的格式

gst-launch-1.0 -v rtspsrc location=rtsp://.../ ! fakesink

檢查 log,尋找 “media=(string)audio” 和 “media=(string)video” 的 caps, 套用到另外要用的 pipeline 裡:

gst-launch-1.0 udpsrc port=9327 ! capsfilter caps="application/x-rtp, media=(string)audio, payload=(int)96, clock-rate=(int)12000, encoding-name=(string)MPEG4-GENERIC, encoding-params=(string)2, profile-level-id=(string)1, mode=(string)AAC-hbr, sizelength=(string)13, indexlength=(string)3, indexdeltalength=(string)3, config=(string)1490, a-sdplang=(string)en, ssrc=(uint)430961011, clock-base=(uint)0, seqnum-base=(uint)1, npt-start=(guint64)0, npt-stop=(guint64)596480000000, play-speed=(double)1, play-scale=(double)1" ! rtpmp4gdepay ! decodebin ! autoaudiosink
gst-launch-1.0 udpsrc port=9327 ! capsfilter caps="application/x-rtp, media=(string)audio, payload=(int)96, clock-rate=(int)12000, encoding-name=(string)MPEG4-GENERIC, encoding-params=(string)2, profile-level-id=(string)1, mode=(string)AAC-hbr, sizelength=(string)13, indexlength=(string)3, indexdeltalength=(string)3, config=(string)1490, a-sdplang=(string)en, ssrc=(uint)430961011, clock-base=(uint)0, seqnum-base=(uint)1, npt-start=(guint64)0, npt-stop=(guint64)596480000000, play-speed=(double)1, play-scale=(double)1" ! rtpmp4gdepay ! aacparse ! faad ! autoaudiosink
gst-launch-1.0 udpsrc port=9527 ! capsfilter caps='application/x-rtp, media=(string)video, payload=(int)97, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, profile-level-id=(string)42C01E, sprop-parameter-sets=(string)"Z0LAHtkDxWhAAAADAEAAAAwDxYuS\,aMuMsg\=\=", a-framesize=(string)240-160, a-sdplang=(string)en, a-cliprect=(string)"0\,0\,160\,240", a-framerate=(string)24.0, ssrc=(uint)1423328209, clock-base=(uint)0, seqnum-base=(uint)1, npt-start=(guint64)0, npt-stop=(guint64)596480000000, play-speed=(double)1, play-scale=(double)1' ! rtph264depay ! h264parse ! avdec_h264 ! autovideosink

範例

RTSP to WebM with HTML video tag

gst-launch-1.0 \
    webmmux name=webm ! tcpserversink host=127.0.0.1 port=8080 \
    rtspsrc location=rtsp://.../ ! \
    decodebin name=dmux \
    dmux. ! videoconvert ! vp8enc ! webm.video_0
gst-launch-1.0 \
    webmmux name=webm ! tcpserversink host=127.0.0.1 port=8080 \
    rtspsrc location=rtsp://.../ ! \
    decodebin name=dmux \
    dmux. ! audioconvert ! vorbisenc ! webm.audio_0
gst-launch-1.0 \
    webmmux name=webm ! tcpserversink host=127.0.0.1 port=8081 \
    uridecodebin uri=rtsp://.../ name=dmux \
    dmux. ! videoconvert ! vp8enc ! queue ! webm.video_0 \
    dmux. ! audioconvert ! audiorate ! vorbisenc ! queue ! webm.audio_0

HTML 範例:

<video width=640 height=480 autoplay>
 <source src="http://localhost:8081">
</video>

Benchmark & Profiling

Term

  • source

  • sink

  • pad

  • bin

  • pipeline

設計重點

Pipeline

Write Plugin

Real-Time Streaming

Cheatsheet

Check Receiving

fakesink dump=true 會把收到的資料以十六進位的方式輸出

# 從 RTSP
gst-launch-1.0 rtspsrc location=rtsp://... ! fakesink dump=true

Play Directly

# 從攝影機
gst-launch-1.0 v4l2src ! autovideosink

# 從 RTSP
gst-launch-1.0 rtspsrc location=rtsp://... ! decodebin ! autovideosink


# 透過 playbin
gst-launch-1.0 playbin uri=file:///path/to/video
gst-launch-1.0 playbin uri=rtsp://...

Record

注意 -e

# 從攝影機
gst-launch-1.0 -e v4l2src ! x264enc ! mp4mux ! filesink location=/tmp/sample.mp4

# 從 RTSP
gst-launch-1.0 -e rtspsrc location=rtsp://... ! decodebin ! x264enc ! mp4mux ! filesink location=/tmp/sample.mp4

# 從 RTP
gst-launch-1.0 -e udpsrc address=0.0.0.0 port=5566 ! application/x-rtp,media=video,payload=96,encoding-name=H264 ! rtph264depay ! decodebin ! x264enc ! mp4mux ! filesink location=/tmp/sample.mp4

Restart Pipeline in Application

先把 pipeline 的狀態設成 NULL , 接著再設成 PLAYING , 這樣就能把整個 pipeline 重啟。

pipeline.set_state(Gst.State.NULL)
pipeline.set_state(Gst.State.PLAYING)

分析用 Plugins

  • audiolatency

License

LGPL

GStreamer 0.10 -> 1.x

https://gstreamer.freedesktop.org/documentation/application-development/appendix/porting-1-0.html

GStreamer 1.14

  • [2018][FOSDEM] What’s new in GStreamer
    • AV1 支援,AV1 目前還在發展階段,效能還不夠好,但是已經可以嘗試

    • IPC pipeline,讓 pipeline 可以開在多個 process,且可以互相溝通

    • 支援使用 ring buffer 做 debug log

    • leak tracer 改進

    • latency tracer 改進

    • hlssink2

    • RTSP 2.0 支援

    • mp3 相關功能因為專利過期而將從 gst-plugins-ugly 移到 gst-plugins-good

    • OpenGL 支援從 gst-plugins-bad 移到 gst-plugins-base

    • WebRTC 支援

    • Meson build system 支援

參考

Talks:

Articles: