Skip to content

[Bug] Compounded timing inaccuracy on master compared to 3.10.3 (Follow-up to #2420) #2994

Open

Description

@nomakewan

Board / MCU

ESP32-S3

LED Chipset

WS2812B / NeoPixel

FastLED Version

master#27a1583

Arduino / PlatformIO version

PlatformIO 6.1.19

Operating System

Windows

Description

Per the close message in #2420 asking to open a new issue if #2420 was not correctly resolved, I am opening this issue to confirm that #2420 was not correctly resolved and that the timing drift first reported in #2420 is very much still present as of master#27a1583.

Per the request in the closure message asking for serial data timing messages, I have provided the timing messages confirming that the drift is not microseconds as suggested but is rather 63~300 milliseconds per sequence iteration.

Please additionally note that the serial output with the exact same minimal sketch on 3.10.3 looks like this:

2495
2495
2495
2495
2495
2495
2495
2495
2495
2495

Considering the only difference between the data posted below and the data posted above is FastLED 3.10.3 versus master#27a1583, this strongly suggests that the underlying timing issue reported in #2420 is as yet unresolved, and is indeed caused by FastLED itself rather than any bug in the program, the millisDelay library, or the ESP-IDF framework.

Steps to reproduce

  1. Target FastLED master#27a1583.
  2. Compile minimal sketch.
  3. See that timing precision is significantly more unreliable than 3.10.3.

Minimal sketch

#include <Arduino.h>
#include <millisDelay.h>
#include <FastLED.h>

uint32_t i_iteration_time = 0;
uint8_t i_post_fade = 255;
CRGB leds[35];
#define LED_PIN 4
#define FAST_LED_UPDATE_MS 5
#define NUM_LEDS 35 // 35-LED NeoPixel ring
uint8_t i_fast_led_delay = FAST_LED_UPDATE_MS;
millisDelay ms_fast_led;
millisDelay ms_delay_post;

void setup() {
  FastLED.addLeds<NEOPIXEL, LED_PIN>(leds, NUM_LEDS).setCorrection(TypicalLEDStrip);
  FastLED.setMaxRefreshRate(800);
  FastLED.show();

#ifdef ESP32
  setCpuFrequencyMhz(160);
  Serial.begin(115200);
#endif

  ms_delay_post.start(0);
  ms_fast_led.start(i_fast_led_delay);
}

void updateLEDs() {
  if(ms_fast_led.justFinished()) {
    FastLED.show();
    ms_fast_led.start(i_fast_led_delay);
  }
}

void loop() {
  if(i_post_fade > 0 && ms_delay_post.justFinished()) {
    const uint8_t i_ring_start = 0;
    const uint8_t i_ring_end = 34;
    const uint8_t i_ring_divisor = 7;
    uint8_t i_ring_counter = 0;

    i_ring_counter = ((255 - i_post_fade) / i_ring_divisor) + i_ring_start;

    if(i_ring_counter <= i_ring_end) {
      leds[i_ring_counter] = CRGB::Red;
    }

    if(i_ring_counter - 1 >= i_ring_start && i_ring_counter - 1 <= i_ring_end) {
      leds[i_ring_counter - 1] = CRGB::Black;
    }

    i_post_fade--;

    if(i_post_fade == 0) {
      Serial.println(millis() - i_iteration_time);
      i_iteration_time = millis();
      ms_delay_post.stop();

      for(uint8_t i = i_ring_start; i <= i_ring_end; i++) {
        leds[i] = CRGB::Black;
      }

      updateLEDs();
      delay(1000);
      i_post_fade = 255;
      ms_delay_post.start(225);
      return;
    }
    else {
      ms_delay_post.start(5);
    }
  }

  updateLEDs();
}

Serial / debug output

2563
2654
2752
2738
2752
2750
2579
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2592
2638
2566
2745
2582
2575
2750
2747
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563
2564
2563

Checklist

  • I have searched existing issues for duplicates
  • I have tested with the latest FastLED release or git master
  • I can reproduce this with a minimal sketch

Activity

  1. zackees commented on Jun 9, 2026

    @zackees
    Member

    tapped out of credits until thursday.

  2. zackees commented on Jul 29, 2026

    @zackees
    Member

    Root-caused. This is not microsecond-level drift — it is FreeRTOS tick quantization inside FastLED.show()'s refresh-rate throttle, and it explains both the magnitude and why 3.10.3 was rock-steady.

    The regression

    git log -L on the throttle loop in src/FastLED.cpp.hpp gives the exact 3.10.3 → master delta, commit 5c8f7a1e7f ("try and fix wifi slowdown", refs #2193):

    -	while(mNMinMicros && ((fl::micros()-lastshow) < mNMinMicros));
    +	while(mNMinMicros && ((fl::micros()-lastshow) < mNMinMicros)) {
    +#if SKETCH_HAS_LARGE_MEMORY
    +		fl::task::run(250, fl::task::ExecFlags::SYSTEM);
    +#endif
    +	}

    3.10.3 spun in a microsecond-accurate busy-wait. Master sleeps.

    Why the sleep is far longer than 250 µs

    1. fl::task::run(250, SYSTEM) — src/fl/task/executor.cpp.hpp:135-141. Because microseconds > 0, it calls pumpCoroutines(1000). The caller's 250 µs is discarded; 1000 is hardcoded.
    2. pumpCoroutines on ESP32 — src/platforms/esp/32/coroutine_esp32.impl.hpp:82-85. It converts µs→ticks and then forces a floor:
      if (ticks == 0) { ticks = 1; }
      vTaskDelay(ticks);
    3. So every iteration of the throttle loop sleeps at least one full FreeRTOS tick — 1 ms on Arduino-ESP32 (1000 Hz), 10 ms on default ESP-IDF (100 Hz).

    Your sketch sets setMaxRefreshRate(800), so mNMinMicros = 1250 µs. The throttle needs to wait ~1.25 ms but can only wake on 1 ms tick boundaries — so each show() overshoots by a variable 0–1 ms, and it never averages out because it re-quantizes every frame.

    The numbers match

    Your sequence is 2495 ms with show() every 5 ms ≈ 499 shows per sequence. At 1 ms tick granularity:

    jitter per show drift per sequence
    0.05 ms +25 ms
    0.25 ms +125 ms
    0.60 ms +299 ms

    Predicted 25–299 ms. You reported 63–300 ms. That is the whole effect.

    It also explains the "compounded" framing from #2420 — it is not accumulating error, it is ~500 independent quantization events per sequence.

    Why a plain revert is wrong

    The yield was added deliberately: without it, show() busy-waits with interrupts effectively starving lower-priority tasks, which caused the WiFi/websocket starvation in #2193 and #2254. Reverting reintroduces those.

    Suggested fix

    Only take the deep yield when there is at least a full tick left to burn; spin the sub-tick remainder:

    while (mNMinMicros && ((fl::micros() - lastshow) < mNMinMicros)) {
    #if SKETCH_HAS_LARGE_MEMORY
        const fl::u32 remaining = mNMinMicros - (fl::micros() - lastshow);
        if (remaining >= FL_TICK_PERIOD_US) {
            fl::task::run(250, fl::task::ExecFlags::SYSTEM);  // long wait: be a good citizen
        }
        // sub-tick remainder: spin, because vTaskDelay would overshoot past our deadline
    #endif
    }

    Long throttles (low setMaxRefreshRate) keep yielding to WiFi/lwIP exactly as today; short ones stay microsecond-accurate like 3.10.3. Note that at any refresh rate above ~1000 Hz the entire wait is sub-tick, so today it is pure quantization noise.

    A second, independent bug worth fixing regardless: executor.cpp.hpp:138 ignores its microseconds argument and hardcodes pumpCoroutines(1000). Every caller asking for a shorter yield silently gets 1000 µs (then rounded up to a tick).

    Caveats

    Analysis is from source and git history — I do not have an ESP32-S3 on hand, so I have not measured this on your setup. Two things would confirm it in minutes:

    • Set FastLED.setMaxRefreshRate(0) to disable the throttle entirely. If timing snaps back to a steady 2495, this is confirmed.
    • Or log micros() immediately before and after FastLED.show() — you should see it consistently overshoot to ~1 ms boundaries rather than ~1250 µs.

    Happy to put up the fix if the maintainers agree with the approach; it needs someone with hardware to validate that WiFi fairness is preserved.

  3. zackees commented on Jul 29, 2026

    @zackees
    Member

    Filed #3762 with the full analysis and a proposed fix. Short version: show()'s refresh throttle is the last call site still taking an unconditional deep yield — the same defect was already fixed in the channel wait loops by 9d71de01d0 (skip the vTaskDelay(1) floor) and #2817 (NetworkDetector gating). show() was missed.

    #3762 also raises whether the yield is needed at all now that transposition is fast, and proposes an ESP32-C6 bench experiment to settle it by measurement rather than argument.

  4. nomakewan commented on Jul 29, 2026

    @nomakewan
    ContributorAuthor

    Analysis is from source and git history — I do not have an ESP32-S3 on hand, so I have not measured this on your setup. Two things would confirm it in minutes:

    * Set `FastLED.setMaxRefreshRate(0)` to disable the throttle entirely. If timing snaps back to a steady 2495, this is confirmed.
    

    Setting this did not resolve the issue. Please note that in the meantime between filing this issue and now, a new version of the ESP-IDF framework has been released. I have updated to this latest version, and updating did not have any effect whatsoever on the results. Please see the timing below with FastLED.setMaxRefreshRate(0); set in setup().

    ESP-ROM:esp32s3-20210327
    Build:Mar 27 2021
    rst:0x15 (USB_UART_CHIP_RESET),boot:0x8 (SPI_FAST_FLASH_BOOT)
    Saved PC:0x40378dca
    SPIWP:0xee
    mode:DIO, clock div:1
    load:0x3fce2820,len:0x10cc
    load:0x403c8700,len:0xc2c
    load:0x403cb700,len:0x30b0
    entry 0x403c88b8
    
    E (28) cache: esp_cache_msync(113): invalid addr or null pointer
    
    2650
    2633
    2677
    2647
    2629
    2647
    2619
    2613
    2633
    2668
    2640
    2612
    2626
    2592
    2710
    2645
    2645
    2591
    2626
    2584
    2643
    2668
    2613
    2655
    2631
    2621
    2627
    2661
    
  5. moved this from Triage to Done in FastLED Trackeron Jul 29, 2026
  6. nomakewan commented on Jul 29, 2026

    @nomakewan
    ContributorAuthor

    I have tested commit 1e70d60 on real hardware and the problem not only persists, but the timing drift has become measurably worse. Please see the following serial output.

    E (689) cache: esp_cache_msync(113): invalid addr or null pointer
    
    2732
    2743
    2983
    2747
    2752
    2648
    2745
    2654
    2743
    2988
    2752
    2650
    2752
    2738
    2745
    2990
    2745
    2741
    2752
    2743
    

    I request this issue be reopened as it has not been resolved.

  7. zackees commented on Jul 29, 2026

    @zackees
    Member

    Retracting my earlier root-cause analysis, and reopening — this is not fixed.

    I previously concluded the drift came from FreeRTOS tick quantization inside CFastLED::show()'s refresh-rate throttle, and #3764 was merged on that basis, which auto-closed this issue. That conclusion was wrong.

    The throttle never runs in your sketch. show() is gated by ms_fast_led at 5 ms. Subtracting the ~1050 us WS2812 transmit for 35 LEDs, the throttle sees ~3950 us elapsed against mNMinMicros = 1250 us at setMaxRefreshRate(800):

    show() called every                  5000 us
    WS2812 transmit, 35 LEDs             1050 us
    elapsed seen by throttle             3950 us
    mNMinMicros @ setMaxRefreshRate(800) 1250 us
    throttle engages?                    NO   (3950 >= 1250 -> loop exits on first check)
    

    So the code path #3764 changed is never reached by your sketch and cannot explain what you measured. #3764 is a genuine improvement for sketches that do hit the throttle, but it is not a fix for this.

    How I got it wrong, since it is instructive: my supporting arithmetic (predicted 25-299 ms vs your reported 63-300 ms) came from choosing a per-frame jitter fraction freely across an order of magnitude. Three arbitrary multipliers will bracket almost any target. I treated that as confirmation when it was curve-fitting, and I never ran the one check that would have falsified it in seconds — does the throttle condition ever evaluate true here? — despite having your sketch the whole time. Apologies for the noise.

    Where the evidence actually points. Your drift is 68-257 ms over 254 fade steps = 268-1012 us per step. A single WS2812 show() for 35 LEDs is ~1050 us — the same order of magnitude. That implicates show()'s duration, not throttle quantization: your loop polls justFinished() once per iteration, so a longer show() widens the gap between a 5 ms deadline passing and the poll observing it, and that excess compounds across 254 steps.

    The real question is therefore why FastLED.show() got slower between 3.10.3 and master. Candidates, unranked and unmeasured: the RMT5/channels driver rewrite, per-frame transposition/encoding cost, EngineEvents begin/end-frame listener work, or per-frame allocation/logging added since 3.10.3.

    Tracking the investigation in #3765.

    Two things that would help enormously if you have a moment, both about 5 minutes:

    1. Time show() directly on both versions:

      uint32_t t0 = micros();
      FastLED.show();
      Serial.println(micros() - t0);

      If master reports meaningfully more than 3.10.3 for your 35-LED strip, that confirms the direction and gives us a number to bisect against.

    2. Confirm the throttle is irrelevant by setting FastLED.setMaxRefreshRate(0) on master. My analysis says your timing will be unchanged. If it does change, my retraction is itself wrong and I want to know.

  8. moved this from Done to Triage in FastLED Trackeron Jul 29, 2026
  9. nomakewan commented on Jul 29, 2026

    @nomakewan
    ContributorAuthor

    The test I performed when requesting this issue be reopened was already tested using FastLED.setMaxRefreshRate(0); in setup(). The numbers posted are therefore already relevant to your request and fulfill the needs of 2.

    Here is the direct show() timing you have requested for 1 using the code you provided:

    2648
    2638
    2646
    2636
    2636
    2636
    2644
    2636
    2634
    2635
    2643
    2639
    2637
    2636
    2643
    2641
    2636
    2637
    2646
    2639
    2634
    2636
    2641
    2636
    2634
    2637
    2645
    2641
    2633
    2638
    2642
    2637
    2634
    2637
    2636
    2640
    2633
    2640
    2638
    2637
    2634
    2638
    2642
    2637
    2634
    2639
    2636
    2637
    2639
    2635
    2635
    2642
    2633
    2643
    2638
    2642
    2633
    2641
    2637
    2638
    2640
    2642
    2634
    2641
    2635
    2637
    2639
    2639
    2633
    2641
    2636
    2637
    2640
    2642
    2634
    2641
    2635
    2638
    2639
    2642
    2633
    2642
    2639
    2642
    2635
    2640
    2636
    2640
    2638
    2640
    2637
    2639
    2635
    2639
    2639
    2642
    2635
    2642
    2636
    2640
    2636
    2642
    2637
    2640
    2636
    2639
    2636
    2639
    2636
    2640
    2636
    2640
    2638
    2640
    2637
    2640
    2635
    2640
    2637
    2639
    2640
    2638
    2636
    2641
    2638
    2637
    2639
    2640
    2634
    2641
    2639
    2642
    2640
    2637
    2636
    2641
    2633
    2642
    2641
    2642
    2633
    2641
    2639
    2637
    2638
    2642
    2639
    2644
    2633
    2639
    2638
    2637
    2635
    2641
    2639
    2637
    2637
    2639
    2634
    2642
    2638
    2637
    2638
    2639
    2634
    2642
    2636
    2638
    2637
    2639
    2634
    2641
    2638
    2637
    2639
    2640
    2638
    2640
    2636
    2642
    2636
    2642
    2639
    2640
    2638
    2642
    2641
    2637
    2638
    2642
    2635
    2640
    2637
    2637
    2636
    2644
    2633
    2639
    2641
    2638
    2636
    2644
    2635
    2642
    2637
    2638
    2636
    2644
    2638
    2642
    2641
    2642
    2638
    2640
    2639
    2639
    2640
    2642
    2639
    2639
    2635
    2640
    2641
    2637
    2635
    2642
    2634
    2643
    2220
    2643
    2637
    2641
    2638
    2639
    2636
    2639
    2634
    2639
    2636
    2639
    2638
    2636
    2635
    2641
    2634
    2637
    2636
    2639
    2634
    2639
    2636
    2636
    2635
    2639
    2638
    2638
    2634
    2639
    2639
    2638
    2636
    2643
    2640
    2638
    2638
    2640
    2640
    2643
    2639
    2637
    2639
    2642
    2639
    2638
    2638
    2639
    2636
    2639
    2640
    2639
    2639
    2639
    2636
    2642
    2637
    2638
    2638
    2640
    2640
    2642
    2636
    2640
    2639
    2637
    2638
    2642
    2639
    2638
    2637
    2642
    2636
    2639
    2640
    2638
    2636
    2639
    2636
    2639
    2636
    2643
    2640
    2638
    2640
    2639
    2635
    2641
    2637
    2637
    2638
    2640
    2639
    2639
    2638
    2640
    2640
    2638
    2640
    2643
    2634
    2643
    2638
    2642
    2639
    2639
    2638
    2640
    2640
    2638
    2638
    2640
    2635
    2642
    2639
    2638
    2637
    2643
    2640
    2638
    2637
    2642
    2635
    2640
    2637
    2642
    2641
    2637
    2636
    2641
    2635
    2639
    2637
    2637
    2638
    2641
    2635
    2639
    2638
    2642
    2634
    2641
    2637
    2637
    2638
    2640
    2635
    2641
    2637
    2638
    2641
    2639
    2634
    2643
    2640
    2637
    2637
    2639
    2635
    2641
    2636
    2642
    2636
    2639
    2637
    2638
    2637
    2639
    2636
    2640
    2640
    2638
    2636
    2639
    2641
    2642
    2634
    2643
    2637
    2642
    2635
    2639
    2637
    2638
    2639
    2642
    2636
    2642
    2639
    2639
    2640
    2642
    2635
    2640
    2640
    2639
    2636
    2642
    2637
    2642
    2639
    2643
    2640
    2642
    2635
    2640
    2637
    2643
    2639
    2638
    2638
    2639
    2635
    2640
    2640
    2642
    2640
    2637
    2638
    2641
    2639
    2642
    2636
    2639
    2639
    2638
    2640
    2642
    2640
    2637
    2638
    2643
    2210
    2646
    2641
    2638
    2639
    2640
    2639
    2638
    2636
    2636
    2639
    2639
    2637
    2639
    2637
    2637
    2635
    2639
    2635
    2641
    2636
    2639
    2637
    2641
    2635
    2638
    2639
    2639
    2635
    2639
    2638
    2639
    2635
    2642
    2636
    2640
    2637
    2643
    2641
    2638
    2638
    2643
    2636
    2643
    2637
    2637
    2638
    2643
    2634
    2643
    2638
    2638
    2637
    2641
    2636
    2641
    2640
    2642
    2636
    2641
    2637
    2639
    2639
    2640
    2640
    2639
    2638
    2640
    2640
    2642
    2641
    2638
    2640
    2640
    2636
    2640
    2640
    2643
    2635
    2641
    2637
    2637
    2639
    2643
    2639
    2639
    2641
    2640
    2636
    2641
    2638
    2643
    2636
    2642
    2641
    2638
    2639
    2641
    2641
    2637
    2637
    2640
    2640
    2638
    2638
    2642
    2641
    2639
    2637
    2643
    2637
    2639
    2638
    2639
    2637
    2643
    2635
    2642
    2639
    2643
    2639
    2638
    2639
    2639
    2640
    2639
    2637
    2641
    2640
    2642
    2641
    2638
    2637
    2643
    2636
    2642
    2640
    2643
    2640
    2643
    2640
    2638
    2639
    2640
    2639
    2639
    2639
    2640
    2635
    2640
    2638
    2638
    2640
    2642
    2637
    2643
    2637
    2639
    2639
    2642
    2640
    2639
    2638
    2641
    2640
    2642
    2636
    2644
    2639
    2638
    2638
    2640
    2640
    2643
    2635
    2643
    2640
    2638
    2639
    2643
    2639
    2643
    2637
    2639
    2640
    2643
    2640
    2639
    2640
    2640
    2636
    2640
    2637
    2638
    2638
    2640
    2635
    2643
    2640
    2643
    2640
    2642
    2641
    2638
    2636
    2643
    2641
    2642
    2635
    2640
    2637
    2639
    2637
    2639
    2636
    2641
    2636
    2638
    2638
    2639
    2640
    2643
    2635
    2644
    2637
    2642
    2641
    2638
    2639
    2643
    2636
    2640
    2638
    2643
    2634
    2208
    2644
    2639
    2640
    2638
    2639
    2636
    2638
    2638
    2645
    2636
    2637
    2636
    2637
    2638
    2643
    2639
    2635
    2638
    2638
    2638
    2645
    2639
    2636
    2636
    2639
    2636
    2644
    2637
    2638
    2635
    2637
    2639
    2634
    2640
    2640
    2636
    2640
    2638
    2636
    2643
    2640
    2636
    2640
    2641
    2634
    2640
    2640
    2641
    2635
    2641
    2640
    2642
    2635
    2639
    2643
    2636
    2637
    2639
    2635
    2641
    2637
    2637
    2640
    2640
    2634
    2642
    2640
    2641
    2634
    2642
    2640
    2637
    2640
    2641
    2635
    2639
    2638
    2637
    2640
    2641
    2639
    2642
    2640
    2641
    2636
    2641
    2642
    2642
    2634
    2639
    2640
    2636
    2636
    2639
    2636
    2639
    2638
    2636
    2639
    2640
    2635
    2639
    2643
    2637
    2636
    2640
    2642
    2637
    2638
    2639
    2635
    2640
    2640
    2641
    2636
    2638
    2636
    2637
    2639
    2639
    2640
    2636
    2639
    2642
    2635
    2638
    2641
    2636
    2637
    2642
    2642
    2637
    2637
    2641
    2640
    2637
    2636
    2639
    2636
    2641
    2636
    2638
    2643
    2641
    2640
    2641
    2636
    2640
    2639
    2636
    2641
    2641
    2639
    2642
    2636
    2641
    2638
    2636
    2640
    2639
    2635
    2638
    2640
    2637
    2636
    2640
    2636
    2638
    2638
    2641
    2642
    2642
    2639
    2636
    2643
    2638
    2639
    2638
    2640
    2638
    2635
    2638
    2642
    2638
    2637
    2638
    2641
    2641
    2634
    2639
    2642
    2636
    2637
    2641
    2635
    2640
    2640
    2641
    2637
    2641
    2636
    2638
    2640
    2638
    2635
    2639
    2642
    2637
    2637
    2641
    2643
    2636
    2637
    2639
    2635
    2639
    2638
    2641
    2636
    2643
    2640
    2637
    2641
    2638
    2634
    2639
    2642
    2641
    2636
    2208
    2647
    2641
    2636
    2638
    2642
    2639
    2640
    2638
    2637
    2639
    2637
    2635
    2643
    2639
    2638
    2636
    2640
    2636
    2638
    2636
    2639
    2636
    2637
    2638
    2644
    2639
    2638
    2637
    2646
    2635
    2635
    2636
    2635
    2637
    2640
    2636
    2639
    2639
    2640
    2641
    2636
    2642
    2636
    2637
    2640
    2638
    2635
    2639
    2642
    2642
    2634
    2639
    2643
    2637
    2637
    2640
    2635
    2638
    2640
    2637
    2640
    2642
    2635
    2638
    2641
    2641
    2635
    2640
    2639
    2636
    2638
    2638
    2642
    2638
    2637
    2641
    2636
    2641
    2637
    2642
    2635
    2638
    2640
    2641
    2635
    2640
    2636
    2636
    2640
    2641
    2639
    2638
    2642
    2639
    2635
    2638
    2642
    2637
    2637
    2641
    2636
    2639
    2637
    2642
    2642
    2636
    2637
    2638
    2642
    2637
    2637
    2641
    2641
    2641
    2635
    2642
    2639
    2641
    2640
    2637
    2639
    2639
    2635
    2639
    2641
    2636
    2636
    2639
    2635
    2639
    2640
    2637
    2642
    2639
    2639
    2641
    2636
    2639
    2637
    2642
    2635
    2639
    2638
    2641
    2635
    2642
    2639
    2637
    2640
    2638
    2635
    2642
    2639
    2636
    2637
    2641
    2635
    2642
    2639
    2636
    2640
    2641
    2634
    2639
    2637
    2641
    2635
    2638
    2639
    2642
    2634
    2638
    2641
    2636
    2639
    2639
    2642
    2641
    2636
    2639
    2640
    2638
    2640
    2641
    2637
    2642
    2637
    2637
    2639
    2639
    2640
    2637
    2640
    2639
    2639
    2641
    2636
    2639
    2636
    2642
    2636
    2639
    2637
    2637
    2643
    2640
    2634
    2641
    2641
    2636
    2639
    2640
    2636
    2641
    2637
    2637
    2638
    2639
    2639
    2636
    2641
    2639
    2635
    2640
    2640
    2641
    2636
    2641
    2642
    2642
    2218
    2646
    2641
    2637
    2639
    2639
    2639
    2642
    2639
    2638
    2640
    2640
    2635
    2641
    2636
    2637
    2638
    2638
    2638
    2638
    2638
    2636
    2639
    2638
    2634
    2641
    2636
    2639
    2635
    2638
    2638
    2641
    2638
    2635
    2637
    2640
    2635
    2639
    2637
    2637
    2638
    2640
    2640
    2639
    2640
    2640
    2641
    2638
    2637
    2641
    2637
    2639
    2640
    2639
    2640
    2641
    2636
    2640
    2639
    2643
    2635
    2641
    2638
    2638
    2638
    2643
    2635
    2641
    2640
    2638
    2638
    2643
    2639
    2638
    2639
    2640
    2640
    2638
    2638
    2643
    2635
    2640
    2639
    2643
    2635
    2640
    2639
    2637
    2637
    2640
    2636
    2643
    2638
    2637
    2639
    2643
    2634
    2641
    2638
    2642
    2640
    2643
    2640
    2643
    2640
    2642
    2641
    2643
    2639
    2638
    2639
    2639
    2640
    2643
    2635
    2643
    2640
    2642
    2641
    2638
    2636
    2641
    2641
    2638
    2637
    2643
    2635
    2640
    2638
    2637
    2638
    2643
    2639
    2638
    2638
    2642
    2636
    2640
    2638
    2643
    2635
    2640
    2639
    2639
    2636
    2643
    2636
    2639
    2637
    2643
    2640
    2638
    2638
    2642
    2636
    2640
    2636
    2638
    2638
    2639
    2640
    2643
    2635
    2640
    2638
    2642
    2637
    2641
    2637
    2643
    2641
    2637
    2637
    2641
    2635
    2641
    2640
    2637
    2638
    2643
    2639
    2638
    2641
    2642
    2636
    2641
    2640
    2639
    2638
    2642
    2637
    2643
    2637
    2643
    2641
    2637
    2640
    2640
    2640
    2643
    2636
    2642
    2641
    2639
    2639
    2640
    2637
    2639
    2640
    2639
    2637
    2643
    2636
    2639
    2639
    2643
    2639
    2643
    2637
    2640
    2640
    2643
    2636
    2643
    2637
    2638
    2639
    2640
    2639
    2638
    2641
    2640
    2636
    
  10. nomakewan commented on Jul 29, 2026

    @nomakewan
    ContributorAuthor

    Here are the numbers for 3.10.3 using the same code:

    156
    121
    109
    108
    108
    109
    108
    109
    108
    109
    109
    108
    108
    109
    108
    108
    108
    108
    108
    108
    109
    109
    108
    108
    112
    108
    108
    109
    109
    108
    109
    108
    108
    108
    109
    108
    108
    109
    108
    108
    108
    108
    109
    108
    108
    108
    108
    108
    108
    108
    112
    113
    109
    108
    108
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    112
    109
    109
    109
    108
    109
    108
    108
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    108
    108
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    108
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    108
    109
    109
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    108
    108
    108
    109
    109
    109
    109
    109
    112
    109
    109
    108
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    112
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    108
    109
    109
    108
    109
    108
    108
    109
    109
    108
    109
    108
    112
    108
    109
    108
    108
    108
    108
    108
    108
    108
    109
    108
    108
    108
    108
    108
    108
    108
    108
    108
    108
    108
    109
    108
    108
    109
    111
    108
    108
    109
    109
    108
    109
    108
    109
    109
    109
    109
    109
    109
    109
    108
    108
    109
    108
    108
    108
    108
    108
    109
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    108
    109
    108
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    112
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    108
    109
    109
    109
    108
    109
    112
    109
    108
    109
    109
    109
    109
    109
    109
    108
    108
    109
    109
    108
    109
    109
    108
    108
    108
    109
    109
    109
    109
    109
    109
    109
    112
    109
    109
    109
    109
    108
    109
    109
    109
    108
    109
    108
    109
    109
    109
    109
    108
    109
    109
    108
    109
    109
    109
    109
    108
    109
    112
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    108
    108
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    112
    109
    109
    109
    108
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    112
    108
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    108
    109
    108
    109
    109
    109
    109
    108
    108
    109
    109
    112
    108
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    108
    108
    109
    109
    108
    109
    109
    108
    109
    109
    109
    109
    111
    108
    109
    109
    109
    109
    108
    108
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    112
    109
    109
    109
    109
    108
    109
    109
    108
    109
    108
    108
    109
    108
    108
    108
    108
    108
    108
    108
    109
    109
    109
    109
    109
    108
    111
    108
    109
    108
    108
    108
    109
    108
    108
    109
    108
    108
    108
    109
    108
    109
    109
    108
    109
    108
    108
    108
    109
    108
    108
    108
    111
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    112
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    112
    108
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    108
    108
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    112
    109
    109
    108
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    112
    109
    109
    108
    108
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    108
    109
    108
    109
    109
    109
    109
    109
    108
    109
    112
    109
    109
    108
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    108
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    112
    109
    109
    108
    109
    109
    109
    109
    108
    108
    109
    108
    109
    109
    109
    109
    109
    108
    109
    108
    108
    109
    109
    109
    108
    109
    112
    109
    108
    109
    109
    108
    109
    109
    109
    109
    108
    109
    109
    108
    109
    109
    109
    109
    108
    108
    109
    109
    109
    109
    109
    109
    112
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    108
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    112
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    109
    108
    109
    
  11. zackees commented on Jul 29, 2026

    @zackees
    Member

    Investigation state — everything needed to finish this

    Consolidating here so this issue is the single home. The meta-tracker #3765 has been closed; its retraction, narrowing and tooling work is done and summarized below.

    Where the cause is NOT

    tests/profile/fastled_show.cpp (added in #3767) measures the CPU-side per-frame path at ~2 us/frame for a 35-LED strip. The drift reported here is 268-1012 us per fade step — three orders of magnitude larger. So the regression is not in EngineEvents, the three-pass controller walk in CFastLED::show(), dithering/scaling, or the #2428 channels dispatch.

    It is also not the refresh-rate throttle. That path provably never executes in this sketch: show() is gated at 5 ms, and minus the ~1050 us WS2812 transmit the throttle sees ~3950 us against a 1250 us threshold, so its loop exits on the first check. (That was my earlier wrong root cause; #3764 is a real improvement to that path but does not affect this report.)

    Where the cause most likely IS

    Everything the host benchmark structurally cannot see, because the host driver is a stub:

    • per-frame encoding / bit transposition
    • DMA setup and completion
    • driver wait loops (e.g. waitForCondition tiers, Channel::showPixels's waitForReady)

    The per-step excess (268-1012 us) is the same order as one 35-LED WS2812 frame (~1050 us of wire time), which is consistent with show() having become synchronous or gained a per-frame wait where 3.10.3 returned earlier.

    How to settle it — one command per version

    ci/autoresearch/test_timing_drift.py (added in #3768) drives the timingDriftTest RPC that AutoResearchTimingDrift.h has exposed all along. Note it had no host driver until now, which is why this reproducer had never actually been run.

    uv run python -m ci.autoresearch.test_timing_drift --port <PORT> --iterations 20

    It reports two things:

    1. per-sequence wall time against the 2495 ms theoretical (index 0 skipped — partial by construction)
    2. show() duration — min/mean/max. The firmware already returned show_min_us/show_max_us/show_total_us; nothing consumed them.

    Decision rule: the WS2812 wire floor for 35 LEDs is 35 x 24 x 1.25 us = 1050 us. If show()'s mean on master sits well above that floor and 3.10.3's does not, the excess is the regression, and the delta localizes it. Then bisect between the two tags on that number.

    Remaining work

    • Run the above on master and on 3.10.3, same board, same geometry.
    • Bisect once a delta is confirmed.

    Blocker on my side

    I could not run it. The ESP32-C6 on this bench dropped off the USB bus during deploy attempts (VID_303A no longer enumerates) and needs a physical replug. The fbuild failures that preceded it — orphaned mkdir install-locks with no staleness timeout, a stale daemon endpoint surviving daemon stop, and an esptool provisioning failure that killed the daemon — are filed as FastLED/fbuild#1213.

    @nomakewan — if you would rather not wait on our bench, the two checks in my previous comment are the fastest path: timing show() directly on both versions, and confirming setMaxRefreshRate(0) changes nothing on master.

  12. added 3 commits that reference this issue on Jul 29, 2026
    abc1b78
    8835336
    7ffacc2
  13. zackees commented on Jul 29, 2026

    @zackees
    Member

    Reproduced on hardware, and the cost is localized

    Board back on the bench. Ran ci/autoresearch/test_timing_drift.py (#3768) against current master on an ESP32-C6 @ 160 MHz.

    The bug reproduces

    35-LED geometry, 7 usable sequences:

    theoretical      : 2495 ms
    min / mean / max : 2564 / 2595.3 / 2646 ms
    drift vs theory  : +69 .. +151 ms
    

    Your report was +68 .. +257 ms. Same signature, so whatever this is, it is not specific to your board or sketch.

    show() has a fixed ~1.84 ms per-frame cost

    This is the finding. Varying strip length while measuring show() min:

    LEDs wire time show() min excess over wire
    35 1050 us 2036 us +986 us
    70 2100 us 2202 us +102 us
    200 6000 us 2902 us -3098 us

    Least-squares fit: 1843 us fixed + 5.29 us/LED.

    Two things fall out of that:

    1. Transmission is asynchronous. At 200 LEDs, show() returns in 2902 us while the wire needs 6000 us. Doubling 35 -> 70 LEDs added only 166 us to show() while wire time grew 1050 us. So show() is not waiting for the wire.

    2. There is a ~1.84 ms fixed cost per frame, independent of strip length. The 5.29 us/LED slope is plausible CPU encoding work. The intercept is not explained by anything in the frame's actual work.

    For your 35-LED ring that fixed cost is 986 us of excess per show() — which lands squarely inside the 268-1012 us per fade step implied by your drift. That is the mechanism: ~1 ms of fixed overhead per show(), compounded over 254 steps.

    At 200 LEDs the mean (4516 us) runs well above the min because wire time now exceeds the 5 ms call interval and show() starts blocking on the previous frame. That is expected saturation, not the defect.

    What I have NOT established

    The identity of the 1.84 ms. It is suspiciously close to two FreeRTOS ticks at CONFIG_FREERTOS_HZ=1000, which would point at a tick-quantized wait in the driver path — the same class of mechanism I wrongly attributed to the refresh throttle earlier, but in waitForCondition/driver teardown rather than CFastLED::show()'s throttle.

    Important caveat before anyone acts on that hunch: the AutoResearch firmware links WiFi/BLE, so NetworkDetector::isAnyNetworkActive() may be returning true here, which routes waitForCondition's tier-3 yield to the deep vTaskDelay(1) path. Your sketch has no WiFi. If the fixed cost is WiFi-gated, my number would overstate your case — except that your measured drift matches it closely, which argues it is not gated. I am flagging this rather than asserting either way; it needs one more experiment to separate.

    Next

    Bisect the fixed cost between 3.10.3 and master using the same harness, and instrument inside show() to attribute the 1.84 ms. Tracking on #2994.

    Raw data and the harness are reproducible with:

    uv run python -m ci.autoresearch.test_timing_drift --port <PORT> --iterations 8
    uv run python -m ci.autoresearch.test_timing_drift --port <PORT> --iterations 4 --num-leds 70
  14. zackees commented on Aug 20, 2026

    @zackees
    Member

    Burn-down disposition: DEFER ? not resolved, and not a bounded source-only fix.

    Evidence reviewed:

    Restart criteria:

    1. Attach an ESP32-S3 with a 35-pixel WS2812/NeoPixel output and make its port healthy/selectable.
    2. Run the landed ci.autoresearch.test_timing_drift driver on FastLED 3.10.3 and current master with identical ESP-IDF/Arduino core, CPU frequency, wiring, and pixel count; preserve iter_ms plus show_min_us/show_max_us/show_total_us.
    3. Confirm a statistically repeatable show() duration delta, then bisect between 3.10.3 and the first bad commit.
    4. Fix the localized encoder/DMA/wait-path cause, add a regression threshold backed by the board measurement, and rerun with/without active networking to preserve ESP32-S3: I2S driver starves websockets, SPI output incorrect, Audio driver FPS impact #2254 fairness.

    #3764 is therefore not a resolving PR for this issue; the issue remains open and deferred until the required S3 evidence is available.

  15. added
    deferredTriage pass could not resolve this; skipped by ci/oldest_issue.py
    on Aug 20, 2026
  16. nomakewan commented on Oct 15, 2026

    @nomakewan
    ContributorAuthor

    Since there have been several official releases since this was last touched, I wanted to check in to see if some random commit somewhere had corrected this issue. It has not.

    I will start by applauding you for managing to get the compile sizes back down under control. Here are 3.10.7 versus 3.10.3 on the ESP32-S3, compiled with PlatformIO and the latest Pioarduino:

    3.10.3 (latest Pioarduino)
    RAM:   [=         ]   7.2% (used 23504 bytes from 327680 bytes)
    Flash: [          ]   3.9% (used 310596 bytes from 7995392 bytes)
    
    3.10.7 (Commit d4a2d3a, latest Pioarduino)
    RAM:   [=         ]   7.4% (used 24164 bytes from 327680 bytes)
    Flash: [          ]   4.3% (used 346303 bytes from 7995392 bytes)
    

    Below is the microsecond capture for the "corrected" version of 3.10.7 which actually compiles on ESP32-S3, commit d4a2d3a. Note also that this build is updated to the latest stable version of Pioarduino, https://github.com/pioarduino/platform-espressif32/releases/download/55.03.312-1/platform-espressif32.zip.

    ESP-ROM:esp32s3-20210327
    <LF>Build:Mar 27 2021
    <LF>rst:0x15 (USB_UART_CHIP_RESET),boot:0x9 (SPI_FAST_FLASH_BOOT)
    <LF>Saved PC:0x40378dca
    <LF>SPIWP:0xee
    <LF>mode:DIO, clock div:1
    <LF>load:0x3fce2820,len:0x10cc
    <LF>load:0x403c8700,len:0xc2c
    <LF>load:0x403cb700,len:0x30b0
    <LF>entry 0x403c88b8
    
    E (28) cache: esp_cache_msync(113): invalid addr or null pointer
    
    2360
    2374
    2356
    2355
    2357
    2357
    2357
    2355
    2359
    2357
    2357
    2356
    2357
    2356
    2357
    2355
    2356
    2356
    2356
    2356
    2357
    2356
    2357
    2356
    2356
    2355
    2356
    2357
    2356
    2358
    2359
    2356
    2357
    2355
    2357
    2356
    2355
    2359
    2357
    2353
    2357
    2355
    2355
    2358
    2356
    2353
    2357
    2357
    2359
    2355
    2357
    2355
    2356
    2357
    2358
    2357
    2355
    2358
    2356
    2357
    2355
    2356
    2356
    2355
    2356
    2355
    2355
    2355
    2355
    2356
    2355
    2355
    2359
    2355
    2355
    2355
    2360
    2354
    2358
    2357
    2355
    2359
    2356
    2357
    2356
    2355
    2356
    2357
    2359
    2357
    2356
    2357
    2359
    2359
    2355
    2357
    2356
    2355
    2357
    2356
    2354
    2357
    2356
    2359
    2356
    2359
    2356
    2355
    2357
    2357
    2356
    2358
    2357
    2357
    2357
    2357
    2356
    2358
    2355
    2355
    2359
    2357
    2356
    2358
    2356
    2355
    2358
    2356
    2359
    2354
    2359
    2357
    2355
    2357
    2355
    2354
    2360
    2357
    2359
    2356
    2355
    2356
    2356
    2357
    2357
    2356
    2355
    2355
    2355
    2357
    2357
    2356
    2355
    2355
    2356
    2357
    2358
    2360
    2355
    2358
    2359
    2359
    2357
    2357
    2357
    2356
    2359
    2355
    2356
    2357
    2357
    2357
    2359
    2359
    2359
    2354
    2357
    2357
    2355
    2358
    2357
    2354
    2357
    2357
    2353
    2358
    2357
    2359
    2360
    2354
    2357
    2357
    2354
    2357
    2355
    2353
    2358
    2357
    2355
    2358
    2358
    2354
    2360
    2357
    2355
    2357
    2355
    2353
    2358
    2359
    2354
    2358
    2355
    2355
    2359
    2357
    2355
    2357
    2357
    2354
    2360
    2356
    2354
    2357
    2358
    2353
    

    As you can see, while the timing has improved by about 300 microseconds from commit 1e70d60, it remains roughly 21x slower than 3.10.3 which seems to explain the timing drift observed in FastLED since that release.

  17. nomakewan commented on Nov 3, 2026

    @nomakewan
    ContributorAuthor

    Since I see there have been two more official releases since my last post, I shall update this once again to show that 3.10.9 does not fix this issue either. It is still 21x slower than 3.10.3.

    2329
    2328
    2328
    2328
    2326
    2328
    2326
    2328
    2329
    2329
    2326
    2328
    2326
    2329
    2329
    2328
    2326
    2328
    2330
    2326
    2326
    2326
    2330
    2328
    2326
    2329
    2328
    2329
    2329
    2328
    2326
    2328
    2324
    2326
    2328
    2328
    2328
    2329
    2329
    2328
    2329
    2324
    2328
    2328
    2328
    2328
    2324
    2328
    2328
    2324
    2328
    2328
    2328
    2324
    2328
    2328
    2329
    2329
    2328
    2329
    2324
    2326
    2329
    2329
    2329
    2324
    2328
    2326
    2329
    2329
    2324
    2326
    2329
    2324
    2329
    2328
    2324
    2330
    2329
    2326
    2328
    2329
    2324
    2328
    2328
    2324
    2329
    2328
    2329
    2328
    2330
    2324
    2329
    2328
    2328
    2328
    2328
    2326
    2329
    2326
    2329
    2328
    2329
    2324
    2329
    2326
    2329
    2329
    2329
    2329
    2328
    2329
    2329
    2329
    2329
    2329
    2329
    2329
    2326
    2326
    2328
    2326
    2328
    2329
    2328
    2324
    2328
    2329
    2328
    2328
    2324
    2326
    2326
    2328
    2324
    2328
    2329
    2324
    2328
    2329
    2329
    2328
    2329
    2326
    2328
    2326
    2329
    2329
    2329
    2329
    2324
    2330
    2329
    2328
    2328
    2328
    2329
    2329
    2329
    2329
    2329
    2329
    2324
    2328
    2328
    2324
    2329
    2329
    2329
    2324
    2328
    2329
    2326
    2330
    2328
    2328
    2326
    2330
    2329
    2328
    2328
    2324
    2329
    2328
    2329
    2329
    2326
    2326
    2328
    2324
    2328
    2329
    2326
    2326
    2329
    2328
    2328
    2328
    2324
    2328
    2328
    2326
    2328
    2328
    2329
    2325
    2328
    2326
    2329
    2329
    2324
    2328
    2328
    2329
    2329
    2328
    2329
    2328
    2329
    2326
    2329
    2328
    2329
    2326
    2328
    2329
    2326
    2330
    2328
    2329
    2329
    2328
    2329
    2329
    2328
    2326
    2328
    2329
    2328
    2324
    2328
    2329
    2328
    2324
    2076
    2328
    2326
    2329
    2328
    2328
    2328
    2330
    2326
    2329
    2328
    2329
    2326
    2326
    2326
    2328
    2328
    2328
    2326
    2329
    2328
    2328
    2328
    2327
    2326
    2328
    2326
    2326
    2326
    2329
    2329
    2328
    2326
    2328
    2329
    2325
    2327
    2329
    2330
    2324
    2331
    2328
    2325
    2328
    2329
    2324
    2327
    2329
    2329
    2328
    2330
    2328
    2325
    2328
    2330
    2324
    2327
    2328
    2329
    2326
    2331
    2328
    2327
    2328
    2330
    2328
    2329
    2330
    2329
    2330
    2330
    2326
    2327
    2329
    2325
    2328
    2327
    2326
    2327
    2328
    2329
    2324
    2327
    2328
    2329
    2328
    2325
    2328
    2330
    2329
    2325
    2329
    2329
    2324
    2329
    2328
    2325
    2328
    2330
    2326
    2331
    2326
    2325
    2328
    2329
    2329
    2329
    2326
    2331
    2328
    2325
    2328
    2329
    2329
    2325
    2328
    2330
    2328
    2330
    2326
    2327
    2326
    2325
    2328
    2329
    2329
    2325
    2328
    2330
    2329
    2325
    2328
    2329
    2328
    2325
    2327
    2330
    2329
    2329
    2329
    2329
    2324
    2327
    2329
    2325
    2328
    2330
    2328
    2329
    2326
    2327
    2328
    2330
    2328
    2325
    2328
    2330
    2324
    2327
    2327
    2330
    2329
    2325
    2328
    2327
    2328
    2325
    2327
    2327
    2328
    2325
    2328
    2327
    2329
    2329
    2326
    2327
    2329
    2325
    2328
    2327
    2326
    2327
    2326
    2325
    2328
    2327
    2326
    2331
    2328
    2330
    2326
    2331
    2328
    2325
    2327
    2330
    2329
    2325
    2328
    2327
    2328
    2325
    2328
    2329
    2328
    2330
    2328
    2330
    2324
    2327
    2328
    2325
    2328
    2330
    2326
    2327
    2326
    2330
    2324
    2327
    2329
    2330
    2330
    2325
    2326
    2327
    2328
    2330
    2324
    2330
    2329
    2330
    2324
    2329
    2328
    2325
    2076
    2329
    2328
    2328
    2328
    2329
    2328
    2328
    2329
    2329
    2327
    2328
    2328
    2329
    2326
    2328
    2328
    2328
    2329
    2328
    2328
    2329
    2326
    2329
    2328
    2328
    2326
    2326
    2330
    2329
    2328
    2329
    2330
    2328
    2330
    2329
    2328
    2328
    2329
    2328
    2326
    2328
    2328
    2328
    2329
    2329
    2326
    2328
    2328
    2329
    2329
    2329
    2324
    2329
    2327
    2329
    2326
    2328
    2326
    2329
    2329
    2329
    2326
    2328
    2326
    2328
    2324
    2328
    2329
    2328
    2329
    2329
    2328
    2329
    2324
    2328
    2328
    2328
    2326
    2329
    2329
    2328
    2328
    2329
    2324
    2328
    2328
    2328
    2324
    2328
    2328
    2328
    2329
    2328
    2326
    2328
    2328
    2328
    2330
    2328
    2328
    2329
    2329
    2328
    2329
    2329
    2329
    2326
    2329
    2328
    2328
    2328
    2324
    2328
    2329
    2329
    2326
    2329
    2329
    2329
    2329
    2328
    2330
    2328
    2328
    2328
    2326
    2328
    2328
    2329
    2326
    2328
    2328
    2328
    2329
    2329
    2328
    2328
    2324
    2326
    2326
    2329
    2324
    2328
    2328
    2329
    2328
    2326
    2328
    2328
    2328
    2329
    2324
    2328
    2328
    2328
    2326
    2328
    2326
    2329
    2326
    2328
    2328
    2329
    2328
    2328
    2324
    2328
    2329
    2324
    2328
    2326
    2326
    2328
    2328
    2326
    2329
    2329
    2329
    2326
    2327
    2328
    2326
    2328
    2328
    2329
    2329
    2326
    2329
    2328
    2326
    2328
    2328
    2329
    2328
    2329
    2328
    2328
    2328
    2329
    2329
    2328
    2328
    2324
    2330
    2328
    2326
    2328
    2328
    2329
    2326
    2328
    2328
    2329
    2328
    2328
    2326
    2328
    2329
    2329
    2329
    2326
    2330
    2328
    2329
    2329
    2329
    2326
    2326
    2328
    2328
    2326
    2329
    2328
    2326
    2328
    2328
    2326
    2328
    2328
    2329
    2329
    2326
    2328
    2326
    2329
    2328
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    deferredTriage pass could not resolve this; skipped by ci/oldest_issue.pystatus: triageNeeds initial assessmenttype: bugSomething is broken

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions