Project server code update

I'm keeping an eye on my CPU runs as the gaps slowly fill in, but not generating any new data. Haven't spotted any sign of a drift.

But I'm coming up to a run of resends on BRP4G, so new trend data will fill in quickly there.

With the CPU case my feeling is that if there was any drift, it would be over a much longer period. The actual throughputs are closer to estimated peaks. Since the estimated peak throughputs are incorrectly low figures though, the actual throughputs make for a higher ratio, levelling the overall scaling downward. Once the latest/slowest comers roll in (highest pfc's... highest APR/peak_flops), if not already, then those should indeed be relatively stable (even if scaled to the wrong value)

In the meantime, try these two.

WU 620035
WU 606522

Same wingmate, validated within 80 minutes of each other. 450 (20%) difference in credit.

What's an Oland, anyway? Ah - it's slowed down. Maybe running 2-up now?

Here's some graphs from my validated tasks and to keep the post shorter I'll post links to the pictures.

BRP4X64 - A bit unstable, varies around 50 credits/task ±5 credits and with a few outliers.

BRP4G - Clear upward trend with no sign of coming back down. A few of the high outliers are late validations.

S6CasA (CPU only) - Not as unstable as BRPx64 but bigger difference between high and low, 270 - 330 credits/task. Low number of completed tasks so far.

BRP5 iGPU - The start of an upward trend with only 10 valid tasks so far.

BRP5 Nvidia - Big difference in high and low (20,59 - 8091,06) credits/task. Few completed tasks but might also be starting an upward trend.

And as before here's a link to the Excel document with all the data and graphs.

OK, here's another graph for the mix.

Just my own host, linear scale. The green markers are when the actual validation/credit award was made (or, to be pedantic, when the validating task was reported). That should be a more honest trendline - if you can see one.

For clarification - if I report first, you'll see first a red mark (to the left) for my reporting time - that's what we've been using up until now. But it'll be followed by a green mark, some time later, when the wingmate trails in.

If I report second, validation takes place near-enough instantly, and the green mark will superimpose over the red mark - you'll only see green.

Probably time for a stats show, then.

[pre] Jason Holmis Claggy Zombie ZombieM Jacob RH
Host: 11363 2267 9008 6490 6109 10320 5367
GTX 780 GTX 660 GT 650M TITAN 680MX FX3800M GTX 670

Credit for BRP4G, GPU
Maximum 1791.03 1982.00 10952.0 2794.94 11847.5 10952.0 4137.85
Minimum 115.82 88.84 153.90 91.50 94.88 508.73 1355.49
Average 1003.97 895.37 4105.99 1105.00 2009.54 2541.91 2040.89
Median 1168.97 981.15 3140.36 1144.04 1664.94 1570.34 1832.12
Std Dev 576.67 570.61 3015.77 424.29 1790.58 2873.96 495.61

nSamples 47 65 52 401 196 21 118

Runtime (seconds)
Maximum 5027.36 5088.99 11295.0 5383.71 23977.4 12149.5 4169.93
Minimum 3259.10 3294.83 8122.09 1908.26 1512.16 11515.5 4061.45
Average 3674.58 4558.79 8906.78 4191.69 4207.76 11652.6 4119.66
Median 3581.10 4818.30 8837.27 4224.43 4175.31 11617.4 4123.75
Std Dev 346.83 486.74 554.28 550.95 1947.62 131.33 20.85

Turnround (days)
Maximum 6.09 3.91 2.75 3.44 2.94 7.49 0.87
Minimum 0.15 0.07 0.14 0.05 1.52 0.18 0.15
Average 1.66 1.83 0.84 2.09 2.33 2.14 0.61
Median 1.46 1.85 0.78 2.00 2.45 1.02 0.65
Std Dev 1.64 1.01 0.63 0.61 0.32 2.27 0.15[/pre]

seconds elapsed per credit, so smaller is better paying.

That one's less for technical value, more for the fractal art competition anyway ;)

Oh you are plotting against n not against N .....

sorry my maths genes need a serious liedown now.

I think it would be difficult to define an on-ramp in this case. Zombie, in particular, has been crunching for ages - I'm not sure how come we started the whole new convergence with the server code upgrade (I'd have expected all the runtime averages to have been in play for a long time, just disguised by the project's fixed credit policy). As you'll have been seeing, there hasn't been much tweaking in this area of code since it was first deployed four years ago.

What I would like to say to David is that, with fixed-size workunits, and a runtime standard deviation of 20 seconds in 4,000, I expect credit with an SD of 5 in 1000.

Hah! good idea, and there's the rub. Without damping those averages, won't happen.

I felt the need for number reset to observe what any new application would do when installed, point being that number of platforms coming on line is accelerating.

Your own (SD ~20 seconds -> variance 400 seconds -> 10%), then the same with your identical wingman. When averaging Murphy says he'll be 10% high and you'll be 10% low, so now it's 20%. Then both your host scales factor in some percentage you use your hosts for anything, and background tasks. That's a natural world input unfiltered/unconditioned (which is a no-no).

Notices from ordinary cruncher.

Some observations I haven't seen be mentioned here (maybe my bad observations or trivial for you - experts):
- Run time of Intel_GPU apps depends on type of CPU apps crunched by CPU, especially AVX/FMA3 application has strong effect (PG LLR/AVX&AVX2&FMA3, Asteroids AVX on Haswell, Beal and MindModeling SSE2 too),
- CPU time of CPU apps depends on type of Intel_GPU app concurrently running, e.g. Collatz mini has nearly no effect, on the other side Einstein BRP and Seti AP apps can double the CPU time,
- Run time of CUDA apps depends on type of Intel_GPU app concurrently running (not sure if CUDAOpenCL and ATIOpenCL too) - GPU load of CUDA is the same,
- Run time of some types GPU apps can be strongly shortened by manipulation with CPU process priority (if priority of BRP process set to Realtime, Run time is half-length on Intel_GPU).

A bit OT, but......it's my point of view:
What can I see for the time being:
- a hard work and analysis ,
- David's RNG seems to be a fixed credit compared to the granted credit here for GPU apps.
No offense, but are you sure by chance to catch up that chaotic system, as a Boinc space is? It is really a great deal.
I see as a simplest way the fixed credit scheme for tasks of application, which length vary -+ 15 % on "standard" machine. Your work is hard and great, but what is the goal? If "fair" credit scheme for tasks of application using different app_ver, platform and plan_class (SSEx, AVXx, FMA3, GPU Intel, CUDA, OpenCL) with big vary of length....what is the fair credit? For the same reference WU the same credit independently on crunching machine (close to a fixed credit scheme), or credit depended on a "benchmark" (i.e. vary credit for the same reference WU), which is nonsense in the world of AVX/FMA3 and GPU hosts? What I see is effort to reach the benchmark asymptote.
In despite of my point of view my machines stay here and helps to find the way.

It is very interesting for me to look over your work and analysis as my job are chaotic systems, too. (to be clear partially predictable by Poisson/Binomic distribution, i.e. "without memory")

Thanks jason_gee for explanation.
I agree with you, a fixed credit scheme (and a "measured work done scheme" - rosetta, AQUA, Seti MB years ago) means a lot of complications for developers, on the other side is popular independently on target RAC.

Notes (theory):
Maybe D part of regulation (feedback) is too strong or R part too weak because I see the same waves as on a oscilloscope measuring of not well designed regulation with tendencies to vibration (for some frequencies) missing the asymptote. In that case I part has to chance to win. Good test for stability is repeated Dirac impulse, theory e.g. here.

I know that CreditNew works very well for some projects (WEP M+2, Seti AP ...), but these has units with close run times. On the other side on projects having very different times it seems to be RNG (LHC vary 1:3 for my i5-4570S).
Run time/ CPU time/ Credit
20,679.88/ 20,046.41/ 291.51 WU
346.51/ 336.59/ 1.84 WU

Notes (how to bestir/cheat the CreditNew):
a) in stable state (CPU time ~ Run time) start high CPU load by third application (e.g. GPU crunching, Autocad 3D rendering...), credit per CPU time rises
b) to make a notepad overclocking = false benchmark. It works for cca 10 tasks, then credit normalises. If tasks are long enough, cheated credit is noticeable. After 10 tasks it is necessary to crunch other project and to return back after time for 100 tasks, in that case the system "forgets".

Cherry-picking: where do you see source causation for it, on the project's side or on the cruncher's side?

Current system penalises optimisation: it's what I really hate on CreditNew (and benchmark CreditOld too). I lived in "socialist" country for 34 years and be sure we don't have the same stomachs. If I work hard and lot (optimised AVX/FMA3 apps) I eat more and must visit fitness, chiropractor etc.. (pay more for electricity and CPU+MOBO and PSU and cooling system) and I anticipate more salary (more credit). That is what David Lenin Anderson misunderstood.

Another follow up although it's been extensively discussed already.

11 BRP5 Intel GPU tasks has now been validated and the APR has been calculated to 10.78 GFlops running one tasks at a time. The initial estimate was that the iGPU was 53,9 times faster than actual. The peak value reported by Boinc in the startup messages is 13,6 times faster and if that had been used then the tasks would have finished without me having to increase the rsc_fpops_bound value to avoid Boinc aborting the tasks with "maximum time limit exceeded".

Yep there are fixed bounds in play, and the 'safeties' are being tripped as new apps come online, actually introducing more problems ( like a car airbag that goes off at the slightest bump, and then has a spike mounted on the steering wheel behind it).

Part of the cause of that is gross scaling error for the onramp period. Further looks & fix decisions at where bounds should really be set are needed after gross scaling errors in CPU and GPU are improved. It appears that they may be too tight even with good initial scaling, because of diversity in applications and how people use their machines ... Basically the system seems to assume dedicated crunching to some degree, which is quite a false assumption. Then you have the multiple tasks per GPU situation, which is not factored in anywhere.

The reason some of these weaknesses are known is in part that I experienced none of the client side failsafes, because I use a modified client where I widened them. I could see the scales hard-lock to max limits and still give estimates too short. This means the initial server side scale being applied was way out of whack in comparison to the estimate of GPU speed.

It turns out one assumption, that GPUs are 10% of their peakflops, is commented in the server code as being 'generous'. In fact even for the common case of 1 GPU task per GPU this is not generous, but results in estimates that are too short, and when combined with the initial app scalings result in estimates divided by ~300-1000x... (common scaling case 0.05x0.05=0.0025 -> 1/400) before considering the multiple task per GPU case. Double application of scales that overlap there is also a problem easily remedied.

SO that's how both the initial scalings, stability, and possibly overtight failsafe margins interact, and addressign the first two issues should give a clear indication of where the third should be adjusted into safe limits.

I've been away over the weekend, and I won't have time to retrieve the data for the 'validation time' graph until much later this evening.

But here's the logarithmic scale graph, updated and extended. I think we can see that the 'upper mode' has a distinct, converging, trendline too - but there's still 300-400 credits difference between the upper and lower modes.

The median for the whole population for host 5367 (nSamples = 250) is 1880.63 (mean 2030.77): if you'd like to give me your formula for "second half of the month" - by reporting or validation date? - I can pull that out, but it doesn't look as if it would be as high as 2340.

Here's the current linear graph by validation time.

Updating both graphs, to show a new effect.

This morning, I was asked to change the running configuration on my host 5367, for an unrelated reason. As a result, the maximum runtime for these tasks went up from 4137.85 seconds to 4591.35 - nearly 11%.

The first task back after that - before APR had a chance to respond, obviously - is the high outlier at 2474.34

I think that's further evidence of the kind of instability we need to cure.

Seti Beta deployed the Blunkit based Optimised AP v7 yesterday, there the estimates are the other way round,
my Ubuntu 12.04 C2D T8100 took ~12 hours on it's first Wu, shame the estimates start at ~228 hours.

Claggy

AP v6 should only mix with AP v6, and AP v7 should only mix with AP v7.

Even better, the SSE2 app is Optimised, the SSE app in non-Optimised, the difference in runtimes is going to be huge, I see carnage ahead.

Claggy

A lot of my Gamma-ray pulsar search #3 v1.11 results are coming out as inconclusive, in each case they are matched with an intel GPU, and in each case that intel GPU is running OpenCL 1.1 drivers, shouldn't that app be restricted to Intel GPUs with OpenCL 1.2 drivers?

Validation inconclusive Gamma-ray pulsar search #3 tasks for computer 8143

Claggy

Once you're happy with that, there are other ways to simulate 'perfectly normal running conditions' that may induce similar divergent behaviour (or worse). One would be to downclock the GPU while Boinc's running (simulating a lower power state, driver timeout/failsafe, deliberate underclock, extended use of the GPU without suspending Boinc etc ...) I think a key takeaway is that the mechanism isn't really adaptive to reasonably normal variable running conditions.

Yes, existing CreditNew (no mods yet) with new app+host in all its glory. One of the big parts we're studying, because of its importance to keeping new users and applications or devices coming on-line.

That's the onramp period as the system tries to establish how fast you're crunching. It doesn't do it very well, but at least you're getting high credit and giving me some, Thanks! :P

You will be crunching faster than me because I'm doing lots of stuff with my machine lately, and haven't tweaked anything.... also I only have an old Core2Duo driving it.

Two more for your viewing pleasure.

I've started to take out the older hosts, which are returning very few tasks these days, but they served their purpose. Red is now Juan's 10351 (the one he linked two tasks from) - classic view for a new host.

And this is mine, still showing scatter from the new configuration. We'll have to wait a few days before Juan will fit on the same scale (although he validated a couple of my oldies overnight - thank you). I'll keep the configuration stable until sunday night/monday morning, but I'll have to flip back then - I have some held tasks with deadlines.

Yes, I'm planning to refresh the graph with new hosts, and they might be suitable.

What is most helpful is finding hosts with a nice, steady, continuous flow of data, and as little variation as possible in the running conditions (so that any noise in the credit granted can be attributed to external causes).

The sheer number of tasks pushed through isn't particularly important, but the consistency is. It didn't help that Zombie took the two hosts I'd picked off to another project (he's still running other hosts - they crop up in my wingmate lists from time to time), and Mikey leaving the project because it isn't exporting public stats would rule him out.

It's quite time-consuming to switch things over, so bear with me - for the time being at least, old results aren't being deleted here, so there's no rush.

RH - Please let me know if it would be more helpful to simply switch my 7950 from BRP5 to BRP4 or to "remove project" / "add project" (presumably that would create a new host and therefore start credit calcs fresh). Also, is it easier for you if I only run 1 WU at a time?