How is the Progress Bar Calculated for MTurk Toolkit Studies?

The progress bar represents the number of workers who have “completed” your study. In this context, completed refers to workers who have submitted the HIT on MTurk and been approved by you.

The progression of the progress bar depends on whether your study is set to manually or automatically approve workers. Generally speaking, the progress bar advances based on approved assignments submitted to MTurk.

Progress When Using Automatic Approval

When you use automatic approval, the progress bar reflects the number of workers who have submitted your study minus any workers you have rejected. The system adds to the progress bar each time a worker submits your HIT and the correct approval code because the system assumes the worker will be approved unless you intervene and reject their work. Anyone who submits the incorrect approval code will enter the "Manage Pending" section.

 

autoprogress

Progress When Using Manual Approval

If your study is set to manual approval, the progress bar reflects the number of workers who have submitted your HIT and who you have manually approved. When your study is set to manually approve workers, the progress bar does not add workers until you approve their work. Instead, workers are added to the “Manage Pending” section. Workers in the manage pending state are awaiting approval. Once you begin approving workers, they will be added to the progress bar.

 

manprogress  

Technical Details

The progress bar on the study Dashboard reflects the number of workers who have submitted and been approved for your HIT on Mechanical Turk. This number does not necessarily mean that the workers completed or even started your study. For example, it is possible to accept a HIT on MTurk and submit an incorrect approval code without ever having clicked on the link to enter your survey. Although workers rarely if ever do this, the separation of HITs on MTurk from the survey platform you use to collect data means there are times when the number of workers you see on TurkPrime may not match the number of responses in your data file. 

 

Finally, rejected workers are not added to the progress bar, but do count toward the completion of your study. This means if you run a study for 100 workers and reject 2 of them, your progress bar will show 98 out of 100 at the end of the study. You are not charged for workers you reject. To replace rejected workers, you can copy your HIT and launch a new study.