Skip to content

pthread_create doesn't use in a thread from the thread pool (PTHREAD_POOL_SIZE=20)  #16763

Closed
Closed
pthread_create doesn't use in a thread from the thread pool (PTHREAD_POOL_SIZE=20) #16763

Description

@aharondavid
aharondavid
opened on Apr 20, 2022

Step to reproduce:

  1. link with -s USE_PTHREADS -s PTHREAD_POOL_SIZE=20 -s ALLOW_MEMORY_GROWTH=1
  2. run the following code:
void* __stdcall ThreadFunc(LPVOID param)
{
   return 0;
}
void main()
{
        for (int h = 0; h < 25; h++)
	{
               //sleep for give to the created thread a chance to end. 
		usleep(1000000);
              ////////////////

	       pthread_t thread;
	       pthread_create(&thread, NULL, ThreadFunc, 0)		
	}
       // expected thread is 20, but what I am see is 25 (in the Chrome DevTool )
}

Version of emscripten/emsdk:
latest

Activity

  1. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    If you want that workers in the pool the be reused you need to join (or detech) the threads. As it stands the threads you are creating here will stick around forever.

  2. aharondavid commented on Apr 20, 2022

    @aharondavid
    aharondavid
    on Apr 20, 2022
    Author

    Hi,
    Thanks for your commnet.

    Sorry, of course, I do pthread_detach after the create, here are a snippet to my code:

    pthread_create((pthread_t *)(&(th->PThreadID)),
                            (const pthread_attr_t *)(&(th->PThreadAttributes)),
                            (void *(*)(void *))(sl_beginthread_func),
                            (void*)th);

    pthread_detach(th->PThreadID);

    but I still see 25 worker threads in the DevTool Chrome debugger instead of 20

  3. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    How about if you do pthread_join()?

  4. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    I believe the issue is related to the fact that recycling detached threads requires the main thread to be running its event loop, and you are keep the main thread busy and never returning the event loop. The cleanupThread cannot be received without this returning the to even loop.

    I believe you can work around this in one of several ways:

    1. Use pthread_join to explicitly join and recycle the threads.
    2. Build with PROXY_TO_PTHREAD
    3. Use emscripten_set_main_loop to ensure the event loop is running.

  5. aharondavid commented on Apr 20, 2022

    @aharondavid
    aharondavid
    on Apr 20, 2022
    Author
    1. for using pthread_join I need to change my Core that are serv a huge multiplatform code.
    2. cannot be done because another issue: Cannot Rendering and calling to Fetch API Sync in the main thread #16760
    3. unfortunately, i didn't find a good example of using in emscripten_set_main_loop

  6. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    There is documentation in https://emscripten.org/docs/api_reference/emscripten.h.html#c.emscripten_set_main_loop_arg, and plenty of example if you look in the tests/ directory.

  7. aharondavid commented on Apr 20, 2022

    @aharondavid
    aharondavid
    on Apr 20, 2022
    Author

    Using emscripten_set_main_loop doesn't help .

    void* __stdcall ThreadFunc(LPVOID param)
{
   return 0;
}
void main()
{
        emscripten_set_main_loop(myRenderFunc,0,0);

        for (int h = 0; h < 25; h++)
	{
               //sleep for give to the created thread a chance to end. 
		usleep(1000000);
              ////////////////

	       pthread_t thread;
	       pthread_create(&thread, NULL, ThreadFunc, 0)
	       pthread_detach(thread);	
	}
       // expected thread is 20, but what I am see is 25 (in the Chrome DevTool )
}

  8. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    You would need to create your threads from myRenderFunc and return to the main event loop each time you want to recycle a worker.

    Basically you can't recycle workers if you keep the main thread busy in the way that your current code is doing.

  9. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    @kripken @tlively do you think its worth adding a mechanism for recycling/reusing of detached workers without involving the event loop? I guess it would mean communicating the exit of detached threads via shared memory instead of or in addition to the cleanupThread postMessage?

  10. tlively commented on Apr 20, 2022

    @tlively
    tlively
    on Apr 20, 2022
    Member

    Hmm yeah, my guess is that there is a bunch of code out there that spawns a lot of threads without ever returning to the event loop. Reducing the number of workers in the system would be a large runtime memory win. I guess we can get a sense for the potential impact by asking folks to tell us if they get warnings from PTHREAD_POOL_SIZE_STRICT.

  11. aharondavid commented on Apr 20, 2022

    @aharondavid
    aharondavid
    on Apr 20, 2022
    Author

    So, I have a big problem here, .. because I heavily use in pthread_create outside the myRenderFunc.
    is there any workaround? can I initiate calling to cleanupThread from myRenderFunc ?

  12. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    I don't think we have any other workaround for you at this point. The 3 options I gave already are the only ones I can think of.

    Out of interest, don't you need to be returning to event loop (i.e. returning from main) in order to not block the main thread of the browser? i.e. don't you already need to restructure you application such that it periodically returns the event loop, for example so that the web UI updates?

  13. aharondavid commented on Apr 20, 2022

    @aharondavid
    aharondavid
    on Apr 20, 2022
    Author

    Sorry for bothering you , It is just my first experience with emscripten 😊

    Yes, I do need to return the event loop.
    So, if understand correctly, if I press On_Press_UI_Buton several times, I will never exceed 25 threads.
    is it true ?

    void* __stdcall ThreadFunc(LPVOID param)
{
   return 0;
}

void On_Press_UI_Buton()
{
       for (int h = 0; h < 10; h++)
	{
               //sleep for give to the created thread a chance to end. 
		usleep(1000000);
              ////////////////

	       pthread_t thread;
	       pthread_create(&thread, NULL, ThreadFunc, 0)
	       pthread_detach(thread);	
	}
        
       //In ending of this func,  do I return to the Event Loop ?
}

void main()
{
        emscripten_set_main_loop(myRenderFunc,0,0);

       //In ending of this func,  do I return to the Event Loop ? if so, why the cleanupThread  doesn't called ?
}

  14. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    Yes, every time you return from On_Press_UI_Buton any dead threads will be recycled. Until you return the threads cannot be recycled.

  15. sbc100 commented on Apr 20, 2022

    @sbc100
    sbc100
    on Apr 20, 2022
    Collaborator

    I would expect the total thread count not to exceed 10

  16. aharondavid commented on Apr 24, 2022

    @aharondavid
    aharondavid
    on Apr 24, 2022
    Author

    Ok,
    Thanks a lot

  17. aharondavid
    closed this as completedon May 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions