Can anything please be done about the extremely short deadlines? They are causing all tasks to run in panic mode, which messes with multiple GPU machines, causing all but one to sit idle.
____________
Reno, NV
Team: SETI.USA
/ 
Reply
Quote
rebirtherVolunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 2 Jan 13
Posts: 7661
Credit: 44,264,218
RAC: 6
Can anything please be done about the extremely short deadlines? They are causing all tasks to run in panic mode, which messes with multiple GPU machines, causing all but one to sit idle.
Do you have set some cores free for the GPU?
/ Reply
Quote
Yes, I have the app_config.xml set to reserve a full thread per GPU. he problem is not that the GPUs do not get enough CPU cycles. The problem is with BOINC scheduling. When CPU tasks go into panic mode, they take over all available threads, and only one GPU tasks will run.
____________
Reno, NV
Team: SETI.USA
/ Reply
Quote
rebirtherVolunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 2 Jan 13
Posts: 7661
Credit: 44,264,218
RAC: 6
Yes, I have the app_config.xml set to reserve a full thread per GPU. he problem is not that the GPUs do not get enough CPU cycles. The problem is with BOINC scheduling. When CPU tasks go into panic mode, they take over all available threads, and only one GPU tasks will run.
You should set your CPU cores in BM to 90%. So you have always 1 core free for GPU.
/ Reply
Quote
That doesn't solve the problem.
But I did find a work-around using the app_config.xml:
<app_config>
<project_max_concurrent>N</project_max_concurrent>
</app_config>
It constrains the number of CPU threads available to the project. Setting this to be number of threads minus the number of threads needed for the GPUs.
____________
Reno, NV
Team: SETI.USA
/ Reply
Quote
rebirtherVolunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 2 Jan 13
Posts: 7661
Credit: 44,264,218
RAC: 6
That doesn't solve the problem.
But I did find a work-around using the app_config.xml:
N
It constrains the number of CPU threads available to the project. Setting this to be number of threads minus the number of threads needed for the GPUs.
ok, but its not working with older clients.
/ Reply
Quote
Agreed. Longer deadlines would solve the problem for everyone. Panic mode causes the BOINC client to do abnormal things.
____________
Reno, NV
Team: SETI.USA
/ Reply
Quote
rebirtherVolunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 2 Jan 13
Posts: 7661
Credit: 44,264,218
RAC: 6
Agreed. Longer deadlines would solve the problem for everyone. Panic mode causes the BOINC client to do abnormal things.
Only the small bases have a 1 day runtime because if I make it longer and a mix of long and short ones will have a longer waiting time and run out of work.
/ Reply
Quote
Something that would help - but I'm not sure if it can be done so easily :
Delay the resend of the long ones after a timeout by 2 days without changing the deadline itself.
This would give the slower hosts a better chance to return them before the server side scheduler sends them out a second time, even if they are somewhat behind the deadline.
According to the server status page, they currently have a worst runtime of 52+ hours, hard to do within the deadline if they are not on top of the stack on client side.
p.s.: if it is hard to do by application ... this behaviour wouldn't hurt for any kind of results, I even think it should be default for BOINC servers. The client still would do its best to stay within the deadline but the server would be more tolerant. I made that proposal a few years ago but the Berkeley guys didn't pick up the idea :-/
/ Reply
Quote
rebirtherVolunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 2 Jan 13
Posts: 7661
Credit: 44,264,218
RAC: 6
Something that would help - but I'm not sure if it can be done so easily :
Delay the resend of the long ones after a timeout by 2 days without changing the deadline itself.
This would give the slower hosts a better chance to return them before the server side scheduler sends them out a second time, even if they are somewhat behind the deadline.
According to the server status page, they currently have a worst runtime of 52+ hours, hard to do within the deadline if they are not on top of the stack on client side.
p.s.: if it is hard to do by application ... this behaviour wouldn't hurt for any kind of results, I even think it should be default for BOINC servers. The client still would do its best to stay within the deadline but the server would be more tolerant. I made that proposal a few years ago but the Berkeley guys didn't pick up the idea :-/
I have reduced the deadline from 6 to 4 days for the long runners. As long as the first WU was sending back and the second was not running by another host the server is sending an abort signal.
/ Reply
Quote