[Bug 781069] New: GPU resources and VAAPI

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Bug ID: 781069
Summary: GPU resources and VAAPI
Classification: Platform
Product: GStreamer
Version: 1.11.x
OS: Linux
Status: NEW
Severity: normal
Priority: Normal
Component: gstreamer-vaapi
Assignee: gstreamer-***@lists.freedesktop.org
Reporter: ***@setplex.com
QA Contact: gstreamer-***@lists.freedesktop.org
CC: ***@gmail.com, ***@igalia.com
GNOME version: ---

Created attachment 349532
--> https://bugzilla.gnome.org/attachment.cgi?id=349532&action=edit
Gstreamer-VAAPI vs Gstreamer-MSDK

When transcoding with Gstreamer-VAAPI a lot more of GPU resources are being
used then Gstreamer-MSDK.

Live Transcode VAAPI: GPU 27%
Live Transcode MSDK: GPU 6%

VAAPI seems to be using 4 times more GPU then MSDK.

Please see the attached image I took screenshot of both servers.

Environment:

2x identical servers with Xeon(R) CPU E3-1245 v5 SKYLAKE

Server 1 with VAAPI: Ubuntu 16.04
vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 1.8.2.pre1
(1.7.3-372-g2f0a844)

Gstreamer-VAAPI Pipeline:
gst-launch-1.0 souphttpsrc location="http://localhost:80/oranews_HD/mpegts"
is-live=true ! tsdemux name=demux ! queue max-size-buffers=1200
max-size-buffers=0 max-size-time=0 ! \
h264parse ! vaapih264dec ! vaapipostproc width=1280 height=720 ! vaapih264enc
rate-control=2 bitrate=1700 ! h264parse ! \
flvmux streamable=true name=mux ! rtmpsink
location="rtmp://localhost:1935/pushrtmp/vappi_outs live=1" demux. ! queue
max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \
mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue !
voaacenc bitrate=128000 ! mux.



Server 2 with MSDK: Centos 7.2
Media Server Studio 2017
vainfo: VA-API version: 0.99 (libva 1.67.0.pre1)

Gstreamer-MSDK Pipeline:
gst-launch-1.0 souphttpsrc location="http://localhost:80/oranews_HD/mpegts"
is-live=true ! tsdemux name=demux ! queue max-size-buffers=1200
max-size-buffers=0 max-size-time=0 \
! h264parse ! mfxh264dec ! mfxvpp width=1280 height=720 ! mfxh264enc
rate-control=1 bitrate=1700 ! flvmux streamable=true name=mux ! rtmpsink
location="rtmp://localhost:1935/pushrtmp/mfx_out live=1" demux. \
! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 !
mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue !
voaacenc bitrate=128000 ! mux.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Jan Schmidt <***@noraisin.net> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@noraisin.net

--- Comment #1 from Jan Schmidt <***@noraisin.net> ---
Interesting, but I'm not sure there's much GStreamer can do about 2 internal
driver implementations of video codec processing.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Víctor Manuel Jáquez Leal <***@igalia.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |NOTABUG

--- Comment #2 from Víctor Manuel Jáquez Leal <***@igalia.com> ---
Thanks for reporting this issue.

As Jan said, I don't know what we can do in GStreamer since this is related
with the driver itself.

I would recommend to start a discussion in the intel-vaapi-driver mailing list:

https://lists.01.org/mailman/listinfo/intel-vaapi-media

I'm closing it, but if you see that we can improve something in the gstreamer
side, please reopen it.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #3 from Benjamin <***@setplex.com> ---
Created attachment 349801
--> https://bugzilla.gnome.org/attachment.cgi?id=349801&action=edit
VAAPI verbose pipeline

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Benjamin <***@setplex.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|NOTABUG |---

--- Comment #4 from Benjamin <***@setplex.com> ---
Hi,
I have reported the issue with Intel-Vaapi-Driver they found out it's a
GStreamer issue and told me to report back here.

The findings were:
Gstreamer is making the interlaced content double the framereate after
inserting the deinterlacer. My source is intelaced at 25FPS the output comes
out 50 FPS and that's what is making the load double in GPU.

We tried disabling the deinterlace in vappipostproc and then load was normal
and FPS was the same as source.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #5 from Víctor Manuel Jáquez Leal <***@igalia.com> ---
(In reply to Benjamin from comment #4)

Post by "GStreamer" (GNOME Bugzilla)
Hi,
I have reported the issue with Intel-Vaapi-Driver they found out it's a
GStreamer issue and told me to report back here.
Gstreamer is making the interlaced content double the framereate after
inserting the deinterlacer. My source is intelaced at 25FPS the output comes
out 50 FPS and that's what is making the load double in GPU.
We tried disabling the deinterlace in vappipostproc and then load was normal
and FPS was the same as source.

As far as I know, that's what interlaced media does[1], captures in a single
frame, two different times. So, when doing the opposite, the framerate is
duplicated. Am I losing something?

1. https://en.wikipedia.org/wiki/Interlaced

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #6 from Benjamin <***@setplex.com> ---
Frames are split, but it doesn't mean that if you deinterlace it the output to
should be 50
FPS. I also tried to use deinterlce element and x264enc the output is the same
as source stays at 25 FPS.

same with mfxvpp and mfx264enc output stays at 25 FPS.


but when using vaapipostproc and vaapih264enc output goes to 50 fps.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #7 from sreerenj <***@gmail.com> ---
WE do the bob deinterlacing by default and the frame rate should be identical
to field rate.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #8 from Benjamin <***@setplex.com> ---
even Gstreamer-MSDK uses BOB but it's not outputing 50 FPS like VAAPI does.

! mfxh264dec ! mfxvpp width=1280 height=720 deinterlace-mode=1 ! mfxh264enc
rate-control=1 bitrate=1700 !
when using MSDK output is normal FPS remains at 25 FPS with Deinterlacing.


Gstreamer-VAAPI
! vaapih264dec ! vaapipostproc width=1280 height=720 deinterlace-method=1 !
vaapih264enc rate-control=2 bitrate=1700 !
when using VAAPI output goes to 50 FPS so the issue remains in Gstreamer-VAAPI

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #9 from sreerenj <***@gmail.com> ---
(In reply to sreerenj from comment #7)

Post by "GStreamer" (GNOME Bugzilla)
WE do the bob deinterlacing by default and the frame rate should be
identical to field rate.

I mean *It shouldn't double the framerate* :)

If field rate is 30 , then framerate should also be 30. Because bob
deinterlacing only do the line doubling(in each field picture) and we should
get same number of frames

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #10 from Víctor Manuel Jáquez Leal <***@igalia.com> ---
(In reply to sreerenj from comment #9)

Post by "GStreamer" (GNOME Bugzilla)
(In reply to sreerenj from comment #7)

Post by "GStreamer" (GNOME Bugzilla)
WE do the bob deinterlacing by default and the frame rate should be
identical to field rate.

I mean *It shouldn't double the framerate* :)
If field rate is 30 , then framerate should also be 30. Because bob
deinterlacing only do the line doubling(in each field picture) and we should
get same number of frames

Hi Sree! Can you address me any documentation explaining that? Because all what
a I've found [1][2] mention that (if I am reading correctly) bob, among other
deinterlacing methods, is enabled, the frame rate is twice.

Where the field rate is determined?

1. http://www.100fps.com/
2. https://wiki.videolan.org/Deinterlacing

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Víctor Manuel Jáquez Leal <***@igalia.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|GPU resources and VAAPI |postproc: framerate should
| |not be always duplicated

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #11 from Benjamin <***@setplex.com> ---
@Victor, to explain it in very simple way.

In Interlacing the fields are split in single frame for speed in traditional
broadcasting, it was done specially for sporting events to look smooth.

Now when Deinterlacing the fields are doubled in the same frame and not
doubling the same frame.

here is video in reference to that:

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Julien Isorce <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #12 from sreerenj <***@gmail.com> ---
(In reply to sreerenj from comment #9)

Post by "GStreamer" (GNOME Bugzilla)
(In reply to sreerenj from comment #7)

Post by "GStreamer" (GNOME Bugzilla)
WE do the bob deinterlacing by default and the frame rate should be
identical to field rate.

I mean *It shouldn't double the framerate* :)
If field rate is 30 , then framerate should also be 30. Because bob
deinterlacing only do the line doubling(in each field picture) and we should
get same number of frames

Hm Wrong! I was thinking that it is "field encoded (not the two fields per
frame scenario)" and framerate considered as field rate. But I don't think it
is the case.

IF the frame is encoded as two fields per frame(eg: 50 fields per second), and
if the pipeline before deinterlacer report the frame rate as 25, then the
output of deinterlacer should be 50fps.
For eg: Try an interlaced content decode + deinterlace:

vaapidecode ! vaapipostproc deintelace-method=1
avdec_h264 ! deinterlace method=6 !

Both should give same *doubled fps* after deinterlacer element.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #13 from Benjamin <***@setplex.com> ---
I have used this as source for all my tests
http://albcast.tv/vaapi/source_raw.ts
I noticed that frame doubling happens only with Gstreamer's deinterlace element
and VAAPI and the deinterlace is bad shaky video.
then I tried avdeinterlace element from gst-libav and Gstreamer-MSDK and the
video was good and framerate stayed the same as source please see pipelines
below with downloadable output file.

gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue
max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse !
avdec_h264 ! deinterlace method=6 ! x264enc pass=0 bitrate=1700 ! mpegtsmux !
filesink location="avdec264_deintelace_6.ts" -v
Output 50 FPS (not deinterlacing properly shaky video)
http://albcast.tv/vaapi/avdec264_deintelace_6.ts

gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue
max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse !
avdec_h264 ! avdeinterlace ! x264enc pass=0 bitrate=1700 ! mpegtsmux !
filesink location="avdech264_avdeintelace.ts" -v
! avdec_h264 ! avdeinterlace ! x264enc pass=0 bitrate=1700 !
Output 25 FPS (GOOD video results)
http://albcast.tv/vaapi/avdech264_avdeintelace.ts

gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue
max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse !
vaapih264dec ! vaapipostproc deinterlace-method=1 ! vaapih264enc
rate-control=2 bitrate=1700 ! mpegtsmux ! filesink
location="vaapih264dec_deintelace_1.ts"
Output 50 FPS (not deinterlacing properly shaky video)
http://albcast.tv/vaapi/vaapih264dec_deintelace_1.ts


gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue
max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse !
mfxh264dec ! mfxvpp deinterlace-mode=1 ! mfxh264enc rate-control=1 bitrate=1700
! mpegtsmux ! filesink location="mfxh264dec_deinterlace_1.ts"
Output 25 FPS (GOOD Video Results)
http://albcast.tv/vaapi/mfxh264dec_deinterlace_1.ts




Also I tried FFmpeg with deinterlacing it didn't double the framerate and no
shaky video.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #14 from Víctor Manuel Jáquez Leal <***@igalia.com> ---
@Benjamin, to be in the same page, as far as I know there are several methods
to deinterlace, but we can group them in two: 1) those which generate two
frames with both fields, and 2) those which extract one field, and ditch the
other.

In the first group of methods (such as bob) from one original frame, we get
two, thus the framerate is duplicated. When using the second group (such as
weave) the framerate remains as original because there is one output frame from
one input frame.

Hence, we ought to ask which method is used in each element. In the particular
case of vaapipostproc, by default, bob is used, which duplicates the framerate,
just as the deinterlace element configured with the same method.

Now, I'm not saying that vaapipostproc is bug free. I feel that we are not
handling well the case of the second group of methods (discarding one frame).

By the way, if you still want two frames (double framerate), but without
shaking, use a better deinterlacing method such as motion-adaptive, though it
is more expensive in terms of GPU.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #15 from Benjamin <***@setplex.com> ---
@Victor, I tried every deinterlace option in vaapipostproc

deinterlace-method : Deinterlace method to use flags: readable, writable
Enum "GstVaapiDeinterlaceMethod" Default: 1, "bob"
(0): none - Disable deinterlacing
(1): bob - Bob deinterlacing
(2): weave - Weave deinterlacing
(3): motion-adaptive - Motion adaptive
(4): motion-compensated - Motion compensated

and they all doubling the framerate and GPU usage goes 20% just for
deinterlacing alone.


In the other hand i tried with https://github.com/01org/gstreamer-media-SDK
gst-inspect-1.0 mfxvpp:
deinterlacing with MSDK also uses bob by default

deinterlace-mode : Deinterlace mode to use
flags: readable, writable
Enum "GstMfxDeinterlaceMode" Default: 1, "bob"
(1): bob - Bob deinterlacing
(2): adi - Advanced deinterlacing
(11): adi-noref - Advanced deinterlacing
(12): adi-scd - Advanced deinterlacing
(13): weave - Field weaving

Frame rate is not doubled with bob and video is not shaking.
Also with MSDK deinterlcing has very low GPU footprint like 2% max.



As of right now with VAAPI I can't do more then 3x ***@25 leaner
transcodes GPU goes to 100% and it's Skylake HD Graphics 530

vs MSDK 12x ***@25

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #16 from sreerenj <***@gmail.com> ---
(In reply to Benjamin from comment #15)

Post by "GStreamer" (GNOME Bugzilla)
@Victor, I tried every deinterlace option in vaapipostproc
deinterlace-method : Deinterlace method to use flags: readable,
writable
Enum "GstVaapiDeinterlaceMethod" Default: 1, "bob"
(0): none - Disable deinterlacing
(1): bob - Bob deinterlacing
(2): weave - Weave deinterlacing
(3): motion-adaptive - Motion adaptive
(4): motion-compensated - Motion compensated
and they all doubling the framerate and GPU usage goes 20% just for
deinterlacing alone.
In the other hand i tried with https://github.com/01org/gstreamer-media-SDK
deinterlacing with MSDK also uses bob by default
deinterlace-mode : Deinterlace mode to use
flags: readable, writable
Enum "GstMfxDeinterlaceMode" Default: 1, "bob"
(1): bob - Bob deinterlacing
(2): adi - Advanced deinterlacing
(11): adi-noref - Advanced deinterlacing
(12): adi-scd - Advanced deinterlacing
(13): weave - Field weaving
Frame rate is not doubled with bob and video is not shaking.
Also with MSDK deinterlcing has very low GPU footprint like 2% max.

Can you check the performance with out the encoder in pipeline ? eg: a decdoer
! deinterlacer deinterlace-method=2 !

Just to check whether the driver is actually doing the deinterlacing and not
doing the interlaced encoding over interlaced content.

But most probably performance impact should be investigated from driver side.

Post by "GStreamer" (GNOME Bugzilla)
transcodes GPU goes to 100% and it's Skylake HD Graphics 530

How many streams you can encode with gstreamer-vaapi, but with no
de-interlacing?

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Hyunjun Ko <***@igalia.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@igalia.com

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #17 from Benjamin <***@setplex.com> ---
deinterlacer without encoder:
! vaapih264dec ! vaapipostproc deinterlace-method=2 ! filesink
location=capture1.raw
Load: 24%, all this load just from deinterlacing.

I asked the people in the driver side, they said the frame doubling is coming
from Gstreamer-VAAPI not the driver and that's why there is high load.



If I disable the deinterlacing I can do a lot more encoding.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Víctor Manuel Jáquez Leal <***@igalia.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Whiteboard| |P1

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #18 from sreerenj <***@gmail.com> ---
I think there is a confusion in the interpretation of the field "framerate"
here:

We need a clarification on the interpretation of fps:

If "fps" field in a decoder OR parser output caps supposed to represent:

A)"actual output framerate in the display", then a fix needed in
GStreamer-vaapi. Don't double the framerate.

B)"output framerate(not field-rate) producing by the parser/decoder", then a
fix needed in parser/decoder to make the fps as half of field-rate.


C)"output field-rate producing by the parser/decoder", then a fix needed in
GStreamer-vaapi, Don't double the framerate. But we need a documentation in
GStreamer explaining "field-rate represents the framerate".


Note: Upstream gst-MSDK won't duplicate the fps, because it requires the field
rate as input framerate(which is what the parser is providing and inherited to
decoder too) and actual frame rate as output framerate (which is equal to
field-rate).

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

Nicolas Dufresne (ndufresne) <***@ndufresne.ca> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@ndufresne.ca

--- Comment #19 from Nicolas Dufresne (ndufresne) <***@ndufresne.ca> ---
In gstreamer caps, the framerate field is always frames. So the parser helper
gst_h264_video_calculate_framerate() returns a framerate (it will half the SPS
framerate).

Though, if I understood correctly, for scalerbob technique, if you don't double
the rate, it means we will likely scale up all top fields, or all bottom
fields, dropping the other fields. Is that what MSDK VPP implementation do ?
This would obviously remove shakyness.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

--- Comment #20 from sreerenj <***@gmail.com> ---
(In reply to Nicolas Dufresne (ndufresne) from comment #19)

Post by "GStreamer" (GNOME Bugzilla)
In gstreamer caps, the framerate field is always frames. So the parser
helper gst_h264_video_calculate_framerate() returns a framerate (it will
half the SPS framerate).

So the parser produces fps which is the actual number of frames outputting from
the parser, it won't send a field in a buffer I guess.
Then we all are good, gstreamer-vaapi should double the framerate IMO.

Post by "GStreamer" (GNOME Bugzilla)
Though, if I understood correctly, for scalerbob technique, if you don't
double the rate, it means we will likely scale up all top fields, or all
bottom fields, dropping the other fields. Is that what MSDK VPP
implementation do ? This would obviously remove shakyness.

I'm not sure what msdk internally do, but it provides two modes for BOB. Either
you can output half-framerate or double-framerate. It is up to the user to
chose what he/she wants.

What I am curious is, whether the driver always doubles the fps or not. Is it
only msdk plays in the middle to drop or keep frames?

We are dealing with two drivers here. msdk uses iHD and gst-vaapi uses i965.
It is worth to investigate whether one driver has a mode to set the number of
frames required for deinterlacing based on user-provided fps.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=781069

GStreamer system administrator <***@gstreamer.net> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution|--- |OBSOLETE

--- Comment #21 from GStreamer system administrator <***@gstreamer.net> ---
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been
closed from further activity.

You can subscribe and participate further through the new bug through this link
to our GitLab instance:
https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/issues/54.