was v31 compiled with explicit support for CC_6.0 in the NVCCFLAGS section of your makefile or build script?


the argument should be something similar to this:

--generate-code arch=compute_60,code=sm_60

this was what I was suggesting when I mentioned recompiling it to add 6.0 support.

you need to add support for every cc level explicitly.

the whole section would be something like this:

--generate-code arch=compute_50,code=sm_50 --generate-code arch=compute_52,code=sm_52 --generate-code arch=compute_60,code=sm_60 --generate-code arch=compute_62,code=sm_62 --generate-code arch=compute_70,code=sm_70 --generate-code arch=compute_75,code=sm_75 --generate-code arch=compute_72,code=sm_72 --generate-code arch=compute_80,code=sm_80 --generate-code arch=compute_86,code=sm_86 --generate-code arch=compute_89,code=sm_89

plus whatever other flags you have there

v32 recompiled with the fix, I hope...

I wanted to try v32, but even after resetting the project in the client, the server only sent me v28 cuda100 tasks. Oddly enough, the cuda100 tasks used to work on the P100 but now they don't. So I put this as an app_config.xml:

<app_config> <app_version> <app_name>TF</app_name> <plan_class>cuda120</plan_class> <min_gpu_ram_mb>384</min_gpu_ram_mb> <gpu_ram_used_mb>384</gpu_ram_used_mb> <gpu_peak_flops_scale>0.22</gpu_peak_flops_scale> <cpu_frac>0.01</cpu_frac> </app_version> </app_config>

Now I can run the new v32 and it works fine, but the app_config has to be used at least on this particular computer. I have not tried the rest yet.

EDIT: Actually, I spoke too soon. This app_config does NOT prevent the download of v28 cuda100 tasks. So I'm still not able to run all my GPUs.

your cuda100 app P100s are now failing with this:

./mfaktc.exe: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory

maybe you removed the cuda toolkit?

all cuda120 tasks error out on my V100s. in the same way that crash's P100s were erroring.

did you omit support for CC 7.0 (titan V and V100 at this CC level)? as i pointed out before you need to explicitly add support for every CC level individually.

all cuda120 tasks error out on my V100s. in the same way that crash's P100s were erroring.

did you omit support for CC 7.0 (titan V and V100 at this CC level)? as i pointed out before you need to explicitly add support for every CC level individually.

We can try to use cuda100 WUs with an app_config. Thats the oldest app which we are running.

<app_config> <app_version> <app_name>TF</app_name> <plan_class>cuda100</plan_class> <min_gpu_ram_mb>384</min_gpu_ram_mb> <gpu_ram_used_mb>384</gpu_ram_used_mb> <gpu_peak_flops_scale>0.22</gpu_peak_flops_scale> <cpu_frac>0.01</cpu_frac> </app_version> </app_config>

tried it. didnt work. this app config does not work. it will not limit the project to only getting the cuda100 app.

same results on a Titan V (same cc-7.0), instant error.

I'm guessing the problem is how the app was compiled without cc 7.0 support.

all cuda120 tasks error out on my V100s. in the same way that crash's P100s were erroring.

did you omit support for CC 7.0 (titan V and V100 at this CC level)? as i pointed out before you need to explicitly add support for every CC level individually.

We can try to use cuda100 WUs with an app_config. Thats the oldest app which we are running.

<app_config> <app_version> <app_name>TF</app_name> <plan_class>cuda100</plan_class> <min_gpu_ram_mb>384</min_gpu_ram_mb> <gpu_ram_used_mb>384</gpu_ram_used_mb> <gpu_peak_flops_scale>0.22</gpu_peak_flops_scale> <cpu_frac>0.01</cpu_frac> </app_version> </app_config>

tried it. didnt work. this app config does not work. it will not limit the project to only getting the cuda100 app.

same results on a Titan V (same cc-7.0), instant error.

I'm guessing the problem is how the app was compiled without cc 7.0 support.

We can only try this

<app_config> <app_version> <app_name>TF</app_name> <plan_class>cuda111</plan_class> <min_gpu_ram_mb>384</min_gpu_ram_mb> <gpu_ram_used_mb>384</gpu_ram_used_mb> <gpu_peak_flops_scale>0.22</gpu_peak_flops_scale> <cpu_frac>0.01</cpu_frac> </app_version> </app_config>

Or the app was not compiled with cc7, P100 was working, right?

this method of using an app_config to try to force what app gets sent, do not work. i've never had this work reliably and it didnt work for crashtech when you suggested it last time. an app config is designed to enact some configuration on apps that are already downloaded. it does not impose any influence on what the project server actually sends me.

i think your app was not compiled with CC 7.0 support. and if you didn't do it on the latest app, I would guess you didnt do it on the older apps either.

P100 works as far as I know. but that's because you fixed it by recompiling the app to include the CC 6.0 support.

when compiling cuda apps you cannot just put a minimum value. you need to put ALL values that you want to support. to support everything from CC5.0+ you need to explicitly state every version (5.0, 5.2, 6.0, 6.1, 7.0, 7.5, 8.6, 8.9)

5.0 - GM100 Maxwell
5.2 - GM200 Maxwell
6.0 - GP100 Pascal
6.1 - GP102+ Pascal
7.0 - GV100 Volta (Titan V and V100)
7.5 - TU102+ Turing
8.0 - GA100 Ampere
8.6 - GA102+ Ampere
8.9 - AD102+ Ada Lovelace
9.0 - GH100 Hopper

and you're going to have to recompile again in the future to add 10.x something if you want to support the upcoming Blackwell GPUs.