Lessons learnt in new playback elements

Edward Hervey

The past 2 years

  • New adaptive demuxers
  • In-depth refactoring
  • New features and lessons learnt

Recap : The new playback elements

Goal

  • "Figure out the most efficent combination of elements …
  • … to handle various playback-related tasks"
  • Avoid what doesn't need processing
    • Lower memory/cpu usage
    • Lower latency for actions
  • Re-usable elements

How ?

  • New elements
  • Improving existing multiqueue
  • New "Streams Aware" API

playbin3

  • A pipeline
  • "Play this URI"
  • Made of uridecodebin3 and playsink
  • Handles gapless playback (more on this later)

uridecodebin3

  • "Give me the raw audio/video/text for this URI"
  • Made of urisourcebin and decodebin3

decodebin3

  • "Decode these inputs"
  • One (or more) sink pad
  • Made of parsebin, multiqueue and decoders
  • No "buffering" !
  • Handle fast stream selection (just before decoders)

Improvements to multiqueue

  • Dynamically figure out interleave of incoming streams
  • "buffer" enough to be able cope with that interleave
    • i.e. to be able to push downstream at the same "time"
    • Allows "instantaneous" stream-switching

parsebin

  • One sink pad
  • Identifies the input and figures out what demuxer, parser, depayloader, is needed
  • Provides you with "parsed timed elementary streams"
  • No extra threads or buffering
  • Now also used in new adaptive demuxers

urisourcebin

  • Source bin
  • "Provide me the content of this URI"
  • Can handle buffering

new API : Streams and Collection

  • GstStream and GstStreamCollection
  • GST_EVENT_SELECT_STREAMS
  • Can be handled by any element (instead of just playbin)
  • "Streams Aware"
    • add-pad, EOS, remove pad
    • Re-use pad for changing stream
  • Fallback support in parsebin and decodebin3

mission accomplished ?

Handling upstream selection

Previously : Only handled by decodebin3

  • Need to wait for data to be present in decodebin3
    • Selection done on streams "present"
  • No way to select streams "as early as possible"
    • i.e. not present yet within decodebin3

Need a mechanism to decide who handles "stream selection"

  • There can only be a single "stream selection" handler
  • How to figure that out ?

GST_QUERY_SELECTABLE

  • If TRUE, that element handles selection
  • Downstream MUST handle all incoming streams
  • Ex : decodebin3 can query upstream on any new streams

Result

  • New adaptive demuxers provide collection as early as possible
  • decodebin3 just processes all incoming streams
  • Future : Add support to other sources (rtspsrc, …)

Improve buffering

Previously

  • Done on "wave-hand" algorithm by queue2 in urisourcebin
    • Best-effort guess-timation
  • Need a more efficient system based on time
  • "I want at least XX seconds available for playback"

New adaptive demuxers

  • Do their own buffering
  • Use parsebin
  • Buffering on actual timed data is the most efficient

Let's do buffering in time !

  • New parse-streams property on urisourcebin
    • Use parsebin to provide "timed parsed" streams
    • And multiqueue to handle interleave …
  • … and use multiqueue buffering if needed
    • Buffering done on time

Result

  • Buffering is dynamically adjusted based on actual content
  • Not too much, not too little

Gapless

Previously

  • Done in playbin3
  • Pre-roll a full uridecodebin3 and switch when EOS
    • A whole bunch of decoders, multiqueue, etc…

Goal

  • Most of the time the same format
  • Can we re-use more ?

pre-roll in uridecodebin3

  • Pre-roll urisourcebin and re-use decodebin3
  • First attempt was before parse-streams feature in urisourcebin
    • Dynamic Scheduling change is hell on earth
  • parse-streams makes this easier (push-based only)

Result

  • gapless decoding with lower cpu/mem usage
  • Bonus : Instant URI switching for free
    • TV-style, re-use decoder but switch immediately

Lessons learnt

  • Once you've improved 90% of the code and tasks
  • You only have the remaining 90% to improve :smile:

Buffering only reliable with the actual timed data

Dynamically switching between pull and push scheduling

  • Don't, just don't

The future

playbin3 and decodebin3 as defaults

  • Still some issues to iron out
  • Testing and feedback welcome