[WPE-2.38] missing video in webrtc playback

when entering a game on Blacknut app (limited time demo available, I'll provide URI on priv),
most of the time once phone is paired the game starts streaming without any issues.
but every 1/5 or so game starts first frame is shown, audio is played and reaction to keys
can still be connected with sounds produced. so it seems like proper video is generated somewhere.
webrtc logs show that stream is delivered properly and injected into gstreamer. device logs also show
that video is injected somewhere but it is not rendered anywhere.

here few possibilities that come to mind which I haven't had confirmed:

that some entity just paused the video,
timestamps on video are counted wrong
video texture is not updated
there is something different about the stream type used

compared logs from the situation that video runs to the one that it doesn't show only that frames are
not taken from the decoder. why it happens atm I can't tell

I wasn't able to reproduce this yet. Can you get debug logs please? GST_DEBUG_FILE=/path/to/file.log GST_DEBUG="3,webkit*:9"

Ping?

sorry it took so long, here's log from the situation as it was described

audio can be heard
it can be confirmed that there's reaction to keys on the phone in audio change
visible frame doesn't change and it seems to be from the very start of the stream

The log shows that video frames are apparently pushed by the mediastreamsrc element, I'd like to be sure though, can you add appsrc:5,basesrc:5 in the logs?

At least the decoder is emitting buffers fine. So hopefully the issue is only on the mediastreamsrc and playback pipeline.

wpeframework.log.zip

Is there a way to get gst logs from "westeros"?

The libwebrtc video decoder doesn't actually decode, it just parses the bitstream. Those parsed buffers are pushed to the playback pipeline for decoding. The basesrc logs look OK (though a ~5 seconds latency event sent from downstream is worrying...)

Have you checked with 2.46 ?

Can you try this branch? https://github.com/WebPlatformForEmbedded/WPEWebKit/tree/wpe-2.38-mediastreamsrc-backport

haven't had a chance to check it with 2.46 but definitely will with wpe-2.38-mediastreamsrc-backport today.

here's the log with westeros* enabled wpeframework.log.zip.

with latency I did notice that video seems to be out of place and tried to look at some upstream changes related to pts setting in webrtc but as I didn't have much success there was no reason to bring it up. if there was something related you can think of maybe I could try some cherry-picks to see if it covers the issue.

I pushed another commit to the backport branch, maybe it will help with A/V sync issues.

I tried it and 5/6 times it was fine but the last time I got again black screen and stream flowing. see attached

wpeframework.log.zip

Have you been able to check with 2.46 so far?

I've done a bit of digging by reading source code from westeros sink and analyzing the logs.

In the log from the last message, what is the execution where the problem occurred? From the message I would assume the last part of the log corresponds to the issue, but looking at the log itself it's not clear to me, and the interpretation I would make would vary depending on which it is.

Asuming the issue is at the end of the log

(Note: given what I describe in the next point, I am not conviced about the issue being at the end of the log. Still, I started investigating based on that premise and I am including my conclusions for that possibility just in case.)

Looking at the last log we can see westerossink is indeed getting frames. 361 frames in almost 12 seconds of real time clock to be exact (roughly 30fps).

The problem has to be related therefore to what whappens (or doesn't happen) to those frames inside westeros-sink and beyond. In this case there is a good chance the problem is in westeros itself, and particularly in westeros-gl.

It may be a good idea to obtain additional westeros logs by setting the environment variable WESTEROS_SINK_DEBUG_FRAME=1 since I have noticed that, for instance, westeros will not log a failure of sendmsg() inside wstSendFrameVideoClientConnection() otherwise.

wstSendFrameVideoClientConnection() sends messages with the decoded frames to a UNIX domain socket at $XDG_RUNTIME_DIR/video where westeros-gl is listening (bound in wstInitServiceServer(), accepting connections in wstVideoServerThread() and spawning a per-connection thread that runs wstVideoServerConnectionThread()). It is possible to enable logging in the westeros-gl process with the environment variables WESTEROS_GL_DEBUG=6 and WESTEROS_GL_DEBUG_FRAME=1.

Assuming the issue is NOT at the end of the log

There is a video quality change in the log. The first part of the log (from ~0:12:10.004230156 to 0:13:30.937415545) has 1080p video. After that, a flush occurs and we get 720p video at 0:13:35.193236263.

Notably, there is an 80 seconds interval between these two parts. In the second part we see frames being accepted by westeros-sink, decoded and sent to westeros-gl. For all I know, everything is working fine in this second part and the reported issue occurs only in the first part.

In the first part westeros-sink receives only one frame (which would certainly match with the behavior reported). This is made obvious by the fact that the thread running wstVideoOutputThread() doesn't log anything from the moment it notifies first-video-frame-callback all the way until westeros-sink has its state set to NULL 80 seconds later.

0:12:10.439726404  7897 0xc390d400 DEBUG           westerossink westeros-sink-soc.c:6663:wstVideoOutputThread: wstVideoOutputThread: enter
0:12:10.439899489  7897 0xc390d400 WARN            westerossink westeros-sink-soc.c:6677:wstVideoOutputThread: set decoder thread to policy SCHED_FIFO priority 1
0:12:10.468389261  7897 0xc390d400 DEBUG           westerossink westeros-sink-soc.c:6715:wstVideoOutputThread: output thread: issue input VIDIOC_STREAMON
0:12:10.468600720  7897 0xc390d400 DEBUG           westerossink westeros-sink-soc.c:6744:wstVideoOutputThread: Out compose default: (0, 0, 1920, 1080)
0:12:10.468697554  7897 0xc390d400 DEBUG           westerossink westeros-sink-soc.c:6754:wstVideoOutputThread: Out compose: (0, 0, 1920, 1080)
0:12:10.490513958  7897 0xc390d400 DEBUG           westerossink westeros-sink-soc.c:6806:wstVideoOutputThread: wstVideoOutputThread: emit first frame signal
0:13:30.981736607  7897 0xc390d400 DEBUG           westerossink westeros-sink-soc.c:7229:wstVideoOutputThread: wstVideoOutputThread: exit

By filtering for the gst_westeros_sink_soc_render() (the function that gets called when a buffer arrives at westeros-sink's sinkpad) we can see that only one buffer got there. That would certainly cause only one video frame to show...

$ cat wpeframework_no_nulls.log | grep -Ei 'gst_westeros_sink_soc_render: buffer|gst_westeros_sink_change_state'
...
0:12:10.393809167  7897 0xbcdbe168 DEBUG           westerossink westeros-sink.c:1638:gst_westeros_sink_change_state:<WesterosVideoSink> westeros-sink: sink 0x23151a8 change state from READY to PAUSED
0:12:10.437211891  7897 0xebec8920 LOG             westerossink westeros-sink-soc.c:2286:gst_westeros_sink_soc_render: gst_westeros_sink_soc_render: buffer 0xebee8348, len 8199 timestamp: 395335
0:13:30.937415545  7897  0x1d06f20 DEBUG           westerossink westeros-sink.c:1638:gst_westeros_sink_change_state:<WesterosVideoSink> westeros-sink: sink 0x23151a8 change state from PAUSED to READY
...

If the rest of the frames did not arrive there, the next mystery would be to find where they disappeared.

We can see the streaming thread of the H264_dec_appsrc_0x22443d8 calling the create() function more than once in this period.

0:12:10.311917995  7897 0xdb8982e0 DEBUG                basesrc gstbasesrc.c:2579:gst_base_src_get_range:<H264_dec_appsrc_0x22443d8> calling create offset 0 length 4096, time 0
0:12:10.320617248  7897 0xdb8982e0 DEBUG                basesrc gstbasesrc.c:2579:gst_base_src_get_range:<H264_dec_appsrc_0x22443d8> calling create offset 8199 length 4096, time 0
0:12:10.341789691  7897 0xdb8982e0 DEBUG                basesrc gstbasesrc.c:2579:gst_base_src_get_range:<H264_dec_appsrc_0x22443d8> calling create offset 8217 length 4096, time 0
0:12:10.350500444  7897 0xdb8982e0 DEBUG                basesrc gstbasesrc.c:2579:gst_base_src_get_range:<H264_dec_appsrc_0x22443d8> calling create offset 8236 length 4096, time 0

So far we can assume that WebKit has put those frames in the pipeline and that westeros-sink received the first one but wasn't able to receive further ones.

Given that westeros-sink is a sink element, my main suspicion would be that now westeros-sink has prerolled but something else in the pipeline (e.g. the audio sink) hasn't, preventing it from reaching PLAYING.

I would recommend collecting pipeline dumps and additional logs to confirm or disprove these hypothesis and to further bound the search for the missing frames. It would also be helpful to know the exact versions of GStreamer, westeros-sink and other components, as the line numbers in the log don't seem to match WPEWebKit reference buildroot.

@vivienne-w

A close collaboration of @vivienne-w, @ntrrgc and I to debug this Issue, where we could reproduce it on Sagem, revealed that saving the h264 stream to disk (taking the buffers from different elements, through different stages of debugging) and playing it back using gst-play-1.0 would show the video without any stalls. Analyzing the differences between all scenarios (WebKit failing scenario, the successful standalone playback and an third successful scenario where buffers were stolen from WebKit and successfully played in real time using a parallel temporary pipeline), we found out that the failing WesterosSink had the immediate-output property set to True, while the working WesterosSink hadn't.

Commenting out the code that sets the property in the Westeros quirk got the video streaming working for us.

That property is always set when MediaStream is being used, but and at least in this case is causing trouble. I'm looking into WesterosSink source code to try to find more information that can help me discriminate in which cases it should be set and in which it shouldn't.

Worked on WebRTC key frame request when an error is detected on the playback pipeline, but I couldn't make it work in practice because none of the decoders I've tried (OpenMAX on Raspberry Pi, Broadcom Nexus, at least in the version I have here) trigger actual decoding errors with corrupted videos.

Comcast people have a Nexus version where that error detection works for them, so I've left the work done so far in this branch: https://github.com/WebPlatformForEmbedded/WPEWebKit/commits/eocanha/eocanha-debug-167/

Apart from this, the immediate-output property keeps being a problem on my side.

We've been told to keep this Issue on hold by now.

Navigation Menu

[WPE-2.38] missing video in webrtc playback #1572

Asuming the issue is at the end of the log

Assuming the issue is NOT at the end of the log

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[WPE-2.38] missing video in webrtc playback #1572

Description

Activity

philn commented on Oct 25, 2025

philn commented on Nov 11, 2025

BunioFH commented on Dec 4, 2025

philn commented on Dec 9, 2025

BunioFH commented on Dec 15, 2025

philn commented on Dec 15, 2025

philn commented on Dec 15, 2025

philn commented on Dec 15, 2025

BunioFH commented on Dec 16, 2025

philn commented on Dec 16, 2025

BunioFH commented on Dec 17, 2025

vivienne-w commented on Feb 10, 2026

ntrrgc commented on Feb 19, 2026

Asuming the issue is at the end of the log

Assuming the issue is NOT at the end of the log

eocanha commented on Mar 12, 2026

eocanha commented on Sep 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions