[Bug 777213] New: hlsdemux: Re-expose only variant stream with bitrate switching

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

Bug ID: 777213
Summary: hlsdemux: Re-expose only variant stream with bitrate
switching
Classification: Platform
Product: GStreamer
Version: git master
OS: Linux
Status: NEW
Severity: normal
Priority: Normal
Component: gst-plugins-bad
Assignee: gstreamer-***@lists.freedesktop.org
Reporter: ***@lge.com
QA Contact: gstreamer-***@lists.freedesktop.org
GNOME version: ---

We don't need to re-expose streams in EXT-X-MEDIA, since only variant streams
can be changed with bitrate switching

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #1 from Seungha Yang <***@lge.com> ---
Created attachment 343424
--> https://bugzilla.gnome.org/attachment.cgi?id=343424&action=edit
adaptivedemux: Do not expose/remove static streams

Depending on streaming protocol (i.e., hls) there might be streams
which don't need to be re-exposed/removed during playback

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #2 from Seungha Yang <***@lge.com> ---
Created attachment 343425
--> https://bugzilla.gnome.org/attachment.cgi?id=343425&action=edit
hlsdemux: Re-expose only variant stream with bitrate switching

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #3 from Seungha Yang <***@lge.com> ---
remove/expose streams on EXT-X-MEDIA tag seems to unnecessary.
Do I missing something?

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

Seungha Yang <***@lge.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #343424|0 |1
is obsolete| |
Attachment #343425|0 |1
is obsolete| |

--- Comment #4 from Seungha Yang <***@lge.com> ---
Created attachment 355514
--> https://bugzilla.gnome.org/attachment.cgi?id=355514&action=edit
hlsdemux: Re-expose only variant stream with bitrate switching

Re-exposing all streams per variant change is very inefficient
operation because:
* EXT-X-MEDIA streams will not be changed during playback.
* baseclass should wait the preparation of task for new streams
until all download tasks for old streams are finished.
This might be able to break seamless bitrate switching.

When streams-aware flag is set on parent bin (i.e., playbin3),
since it's allowed that dynamic pad exposing, only the variant streams
will be re-exposed and all the other rendition streams will be reused.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #5 from Seungha Yang <***@lge.com> ---
Created attachment 355515
--> https://bugzilla.gnome.org/attachment.cgi?id=355515&action=edit
urisourcebin: Set streams-aware flag

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #6 from Seungha Yang <***@lge.com> ---
I missed one thing on commit message. In case playbin 3, I allowed hlsdemux to
be able to expose subtitle streams.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

tianyahai <***@qq.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@qq.com

--- Comment #7 from tianyahai <***@qq.com> ---
Hi, how about subtitle streams, now my m3u8 file has eight subtitles, it can be
displayed by playbin, but when i switch to another, the subtitle disappeared,
why?

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #8 from tianyahai <***@qq.com> ---
what's the meaning of GST_BIN_FLAG_STREAMS_AWARE?
I added GST_BIN_FLAG_STREAMS_AWARE to the code refer according to your method,
but the value of demux->streams_aware is FALSE, when the code is running

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #9 from Seungha Yang <***@navercorp.com> ---
(In reply to tianyahai from comment #7)

Post by "GStreamer" (GNOME Bugzilla)
Hi, how about subtitle streams, now my m3u8 file has eight subtitles, it can
be displayed by playbin, but when i switch to another, the subtitle
disappeared, why?

AFAIK, subtitle is disabled in current hlsdemux. Need more context such as your
m3u8 and your playbin usecase (playbin or playbin3).


(In reply to tianyahai from comment #8)

Post by "GStreamer" (GNOME Bugzilla)
what's the meaning of GST_BIN_FLAG_STREAMS_AWARE?
I added GST_BIN_FLAG_STREAMS_AWARE to the code refer according to your
method, but the value of demux->streams_aware is FALSE, when the code is
running

You can refer to its meaning at
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstBin.html
My understanding about it is that, it's used in playbin3 (not playbin) to
indicate that corresponding element/bin can handle GstStreams API and also can
handle stream add/removal during playback without any interruption.

I guess you are now use playbin (not playbin3). The playbin doesn't use
urisourcebin but do uridecodebin. Please not that my patch aims to set the flag
at urisourcebin and read it by adaptivedemux (this is urisourcebin's child bin)

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #10 from tianyahai <***@qq.com> ---
(In reply to Seungha Yang from comment #9)

Post by "GStreamer" (GNOME Bugzilla)
(In reply to tianyahai from comment #7)

Post by "GStreamer" (GNOME Bugzilla)
Hi, how about subtitle streams, now my m3u8 file has eight subtitles, it can
be displayed by playbin, but when i switch to another, the subtitle
disappeared, why?

AFAIK, subtitle is disabled in current hlsdemux. Need more context such as
your m3u8 and your playbin usecase (playbin or playbin3).

Thank you for your reply, in deeded, the subtitles are closed by default, but
in the code of gsthlsdemux.c file i enabled subtitle like this：
select = demux->streams_aware ? TRUE : (media->mtype ==
GST_HLS_MEDIA_TYPE_VIDEO ||
media->mtype == GST_HLS_MEDIA_TYPE_AUDIO || media->mtype ==
GST_HLS_MEDIA_TYPE_SUBTITLES);

In my m3u8 there are eight subtitles, the subtitles can be displayed when the
code is running, but if i switch subtitle to another it will be disappear, i
used playbin or playbin2, not playbin3

Post by "GStreamer" (GNOME Bugzilla)
(In reply to tianyahai from comment #8)

Post by "GStreamer" (GNOME Bugzilla)
what's the meaning of GST_BIN_FLAG_STREAMS_AWARE?
I added GST_BIN_FLAG_STREAMS_AWARE to the code refer according to your
method, but the value of demux->streams_aware is FALSE, when the code is
running

You can refer to its meaning at
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/
GstBin.html
My understanding about it is that, it's used in playbin3 (not playbin) to
indicate that corresponding element/bin can handle GstStreams API and also
can handle stream add/removal during playback without any interruption.
I guess you are now use playbin (not playbin3). The playbin doesn't use
urisourcebin but do uridecodebin. Please not that my patch aims to set the
flag at urisourcebin and read it by adaptivedemux (this is urisourcebin's
child bin)

Yes,as you say,i used playbin,can i add "GST_OBJECT_FLAG_SET (dec,
GST_ELEMENT_FLAG_SOURCE | GST_BIN_FLAG_STREAMS_AWARE)" to gsturidecodebin.c
file according to your urisourcebin file?
How can hlsdemux achieve switch subtitles...

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #11 from tianyahai <***@qq.com> ---
I submitted it in this: https://bugzilla.gnome.org/show_bug.cgi?id=788869

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #12 from tianyahai <***@qq.com> ---
I try playbin3 to play my m3u8, if it only contain one stream as MediaPlaylist,
it's ok, if contains variant streams, it failed like below:
...
PREROLLING ...
... 100%
(gst-launch-1.0:22408): GStreamer-CRITICAL **:
gst_stream_collection_get_stream: assertion 'GST_IS_STREAM_COLLECTION
(collection)' failed
Caught SIGSEGV
#0 0xb7765c31 in __kernel_vsyscall ()
#1 0xb736af8f in poll () at ../sysdeps/unix/syscall-template.S:84
#2 0xb74b6b20 in g_poll () from /lib/i386-linux-gnu/libglib-2.0.so.0
#3 0xb74a811c in ?? () from /lib/i386-linux-gnu/libglib-2.0.so.0
#4 0xb74a8539 in g_main_loop_run () from /lib/i386-linux-gnu/libglib-2.0.so.0
#5 0xb763fd81 in gst_bus_poll (bus=0x8648e20, events=GST_MESSAGE_ANY,
#6 0x0804ba1a in ?? ()
#7 0x0804a964 in ?? ()
#8 0xb72a6637 in __libc_start_main (main=0x804a3f0, argc=3, argv=0xbfac64b4,
#9 0x0804b01c in ?? ()
Spinning. Please run 'gdb gst-launch-1.0 22408' to continue debugging, Ctrl-C
to quit, or Ctrl-\ to dump core.

Why it's not ok, does playbin3 not support multi-rate m3u8 file?

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #13 from Seungha Yang <***@navercorp.com> ---
(In reply to tianyahai from comment #10)

Post by "GStreamer" (GNOME Bugzilla)
Thank you for your reply, in deeded, the subtitles are closed by default,
but in the code of gsthlsdemux.c file i enabled subtitle like this：
select = demux->streams_aware ? TRUE : (media->mtype ==
GST_HLS_MEDIA_TYPE_VIDEO ||
media->mtype == GST_HLS_MEDIA_TYPE_AUDIO || media->mtype ==
GST_HLS_MEDIA_TYPE_SUBTITLES);
In my m3u8 there are eight subtitles, the subtitles can be displayed when
the code is running, but if i switch subtitle to another it will be
disappear, i used playbin or playbin2, not playbin3

This comment might be helpful for you :)
https://bugzilla.gnome.org/show_bug.cgi?id=779011#c7

Post by "GStreamer" (GNOME Bugzilla)
Yes,as you say,i used playbin,can i add "GST_OBJECT_FLAG_SET (dec,
GST_ELEMENT_FLAG_SOURCE | GST_BIN_FLAG_STREAMS_AWARE)" to gsturidecodebin.c
file according to your urisourcebin file?

You can try it but it doesn't make sense because uridecodebin doesn't
"STREAMS_AWARE".

Post by "GStreamer" (GNOME Bugzilla)
How can hlsdemux achieve switch subtitles...

It's totally dependent on adaptivdemux's track switch method. Firstly,
adaptivdemux(hlsdemux) will expose all available tracks during preroll state.
Then, stop streaming for non-activated track depending on flow return
"not-linked". It's well described on comment of gstadaptivedemux.c

Then, if you try switch track, it will trigger "reconfigure" event which also
trigger re-streaming for newly activated track in adaptivedemux. Basically,
this is how adaptivedemux switch tracks. But when I saw comment above, there
seems weak point (maybe bug?) in playbin's subtitle track change behavior.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #14 from tianyahai <***@qq.com> ---
(In reply to Seungha Yang from comment #13)

Post by "GStreamer" (GNOME Bugzilla)
(In reply to tianyahai from comment #10)

Post by "GStreamer" (GNOME Bugzilla)
Thank you for your reply, in deeded, the subtitles are closed by default,
but in the code of gsthlsdemux.c file i enabled subtitle like this：
select = demux->streams_aware ? TRUE : (media->mtype ==
GST_HLS_MEDIA_TYPE_VIDEO ||
media->mtype == GST_HLS_MEDIA_TYPE_AUDIO || media->mtype ==
GST_HLS_MEDIA_TYPE_SUBTITLES);
In my m3u8 there are eight subtitles, the subtitles can be displayed when
the code is running, but if i switch subtitle to another it will be
disappear, i used playbin or playbin2, not playbin3

This comment might be helpful for you :)
https://bugzilla.gnome.org/show_bug.cgi?id=779011#c7

Thank you,i will try it follow this comment

Post by "GStreamer" (GNOME Bugzilla)

Post by "GStreamer" (GNOME Bugzilla)
Yes,as you say,i used playbin,can i add "GST_OBJECT_FLAG_SET (dec,
GST_ELEMENT_FLAG_SOURCE | GST_BIN_FLAG_STREAMS_AWARE)" to gsturidecodebin.c
file according to your urisourcebin file?

You can try it but it doesn't make sense because uridecodebin doesn't
"STREAMS_AWARE".

...indeed it doesn't work...

Post by "GStreamer" (GNOME Bugzilla)

Post by "GStreamer" (GNOME Bugzilla)
How can hlsdemux achieve switch subtitles...

It's totally dependent on adaptivdemux's track switch method. Firstly,
adaptivdemux(hlsdemux) will expose all available tracks during preroll
state. Then, stop streaming for non-activated track depending on flow return
"not-linked". It's well described on comment of gstadaptivedemux.c
Then, if you try switch track, it will trigger "reconfigure" event which
also trigger re-streaming for newly activated track in adaptivedemux.
Basically, this is how adaptivedemux switch tracks. But when I saw comment
above, there seems weak point (maybe bug?) in playbin's subtitle track
change behavior.

Have you tested m3u8 file that contains mulit bitrate streams and subtitles,
and can you switch subtile seccessfully? thank you again, have a nice day^_^

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #15 from Seungha Yang <***@navercorp.com> ---
(In reply to tianyahai from comment #14)

Post by "GStreamer" (GNOME Bugzilla)
Have you tested m3u8 file that contains mulit bitrate streams and subtitles,
and can you switch subtile seccessfully? thank you again, have a nice day^_^

I think current gstreamer is not good enough for hls subtitle track change....
DASH + multi TTML case might have similar issue I guess...

In my opinion, current adaptivedemux structure has problem from subtitle track
change point of view.
First, at adaptivedemux side, streaming progress of subtitle is not fully
synchornized with A/V. (since subtitle data is too small, downloading the
subtitle is too faster than A/V).
Adaptivedemux's track change method is that, it tries to download new stream
from 1) last matching streams time position or 2) depending on query position.
This can cause mismatch between old/new subtitle stream's time position.
Second, the fact that multiqueue element doesn't fully consider synchronization
of sparse stream (subtitle stream).

Since I'm not expert... above comment might be incorrect. It's just my opinion.

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #16 from tianyahai <***@qq.com> ---
(In reply to Seungha Yang from comment #15)

Post by "GStreamer" (GNOME Bugzilla)
(In reply to tianyahai from comment #14)

Post by "GStreamer" (GNOME Bugzilla)
Have you tested m3u8 file that contains mulit bitrate streams and subtitles,
and can you switch subtile seccessfully? thank you again, have a nice day^_^

I think current gstreamer is not good enough for hls subtitle track
change.... DASH + multi TTML case might have similar issue I guess...
In my opinion, current adaptivedemux structure has problem from subtitle
track change point of view.
First, at adaptivedemux side, streaming progress of subtitle is not fully
synchornized with A/V. (since subtitle data is too small, downloading the
subtitle is too faster than A/V).
Adaptivedemux's track change method is that, it tries to download new stream
from 1) last matching streams time position or 2) depending on query
position.
This can cause mismatch between old/new subtitle stream's time position.
Second, the fact that multiqueue element doesn't fully consider
synchronization of sparse stream (subtitle stream).
Since I'm not expert... above comment might be incorrect. It's just my
opinion.

Ok,thank you for your help,next i will try playbin3 play m3u8...
but as you said "subtitle is not fully synchornized with A/V.."
even if it switch subtitles successful by playbin3,subtitle display maybe have
some problem..

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

--- Comment #18 from Edward Hervey <***@bilboed.com> ---
*** Bug 796000 has been marked as a duplicate of this bug. ***

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

Seungha Yang <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #355514|0 |1
is obsolete| |

--- Comment #17 from Seungha Yang <***@gmail.com> ---
Comment on attachment 355514
--> https://bugzilla.gnome.org/attachment.cgi?id=355514
hlsdemux: Re-expose only variant stream with bitrate switching

Will be updated again

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

Seungha Yang <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Depends on| |775132
Summary|hlsdemux: Re-expose only |adaptivedemux: Enable reuse
|variant stream with bitrate |pad for "streams-aware"
|switching |usecase

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

Seungha Yang <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #355515|0 |1
is obsolete| |

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213
Bug 777213 depends on bug 775132, which changed state.

Bug 775132 Summary: adaptivedemux: Download fragment only for activated stream
https://bugzilla.gnome.org/show_bug.cgi?id=775132

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |OBSOLETE

["GStreamer" (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777213

GStreamer system administrator <***@gstreamer.net> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |OBSOLETE

--- Comment #19 from GStreamer system administrator <***@gstreamer.net> ---
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been
closed from further activity.

You can subscribe and participate further through the new bug through this link
to our GitLab instance:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/508.