Let me work backwards on this:

We send and receive data to/from the researchers in batches. While a batch is in progress but does not yet have all of its workunits completed, we have to store it on our system and the researchers don't get the benefit of the data in them until the whole batch is completed.

The Mapping Cancer Markers batches have 10,000 workunits per batch. Within in each batch, there are what we call "hard luck workunits". These are workunits where the first couple of assignments go to computers that don't disappear (the owner goes on vacation, they reformat their computer, they uninstall, etc). Therefore it takes 7 days before the next ones are sent out. If left to its own devices, then these next two might go to someone with a large cache and they happen to accidentally unplug their machine and the job ends in an error. Then the next one goes to someone else who never returns it. And so and so on. When we first started using the BOINC software there were workunits that took longer than 90 days to complete due to "bad luck".

As a result, the first copies that are sent out and any additional copies sent out in the first three days are assigned to any computer that asks for them. However, after that time they are only sent to computers that have proven themselves to quickly return jobs correctly. They are also assigned a shorter deadline (30% of the original deadline). This allows most batches to finish in about 160% of the original deadline (1 full deadline and then two shorter deadlines).

However, there are only certain number of these "reliable" hosts. If there are more jobs requiring reliable hosts than there are reliable hosts asking for jobs, then job distribution gets 'clogged' and we are not able to efficiently distribute the work. As a result, we want to minimize the amount of these reliable jobs.

Recently, people have been spinning up a number of EC2 Spot Instances to contribute to World Community Grid. These are virtual machines where you can say how much you are willing to spend and if there is sufficient excess capacity on Amazon's cloud and if no-one else is willing to pay more than you, then your machines will be started and will start running what ever workload you provide. The catch to this is that if excess capacity on EC2 shrinks or if someone offers to spend more money than you do, then your instances are stopped and removed.

What this means for us is that 32 core virtual machines were starting up, requesting 5*32 jobs from us, and then being killed off and the results are never returned. All 160+ of those jobs then become "no result" jobs 7 days later and generate 160 jobs that need reliable hosts. Enough of these were being done that it was exceeding the grid's ability to assign all of those 'reliable' jobs to reliable hosts.

So in order to reduce the number of reliable jobs being produced, we are now limiting new devices to only 1*(number of cores on device) jobs for the first 3 hours of its existence. This will reduce the number of jobs assigned and quickly abandoned, but also allow devices that continue to participate be able to get plenty of work soon when they start to finish their first jobs. Note that if a computer completes and reports a job as complete during this three hour window, they will still be able to get a new job to work on - they just won't be able to build up a buffer until the three hour window expires.

I want to note that there are also a lot of these ec2 spot instances that are doing some significant work for World Community Grid so we do not want to hinder their use and involvement. We are just taking steps to ensure that this works for everyone.

From the time a workunit is created, until 72 hours later, all jobs created for the workunit will not require a reliable device to process them. The timing is related to when the workunit was created not when the first job was assigned nor the deadline of any of the jobs. At some point we would like to change this, but the workunit create time was the only data that was already present in the transitioner that could be useful for this. We load in work between 18-36 hours before it starts being distributed.

At what point should I abort running jobs that show they are past deadline? The system is Aborting tasks that have not started and are past the deadline.

Kevin,

re http://www.worldcommunitygrid.org/forums/wcg/...ead,36060_offset,0#443811 and the first 3 hour capping, theoretically a machine that has idle cores starts asking for work, and reports the results with that request. What happens if these first arrivals complete work within 3 hours. E.g. my quad and octo do FAHV in under 1.5 hours. Many an MCM finishes in under that time and presently, CEP2 does so too. Still have to wait for the 3 hours to pass [and the incrementing deferral to add]?

That's the way I had read Kevin's post (and I found it fine) but your interpretation is possibly correct too.
Needs some clarification from Kevin...