# Uses and Challenges of Mechanical Turk
<hr>
.author[
**Matthew Crump**,

_Brooklyn College of the City University of New York_

<hr>

**Todd Gureckis** ([@todd_gureckis](http://twitter.com/todd_gureckis)),

_New York University_  
]

---

# Agenda

1. **Overview, What is Mechanical Turk?** What is important to know about running psychology experiments online?
2. **Basics of using Amazon Mechanical Turk**
3. Building dynamic web experiments from scratch.
4. Introduction and overview of **psiTurk**
5. We will get a quick tour of **psiTurk**'s experiment exchange.
6. Using **psiTurk** to put our experiment on AMT.
7. General Q & A with any remaining time.

---

# If you need help

1. All tutorial materials are available online. [APSTurk.github.io](http://APSTurk.github.io)
2. Interrupt and ask questions
3. Find a code-buddy

---

# Online data collection?

---

<img src="images/Computer_Workstation_Variables.jpg" width="300">
]

1. Office-style interaction
1. General control over lighting, temperature, viewing angle, distance
1. A few people at a time (1-5/hour), each on individual workstations
1. Data saved to local disc or to a local file server
]

---

.left-column[
.center[
<img src="images/womenonlaptop.jpg" width="200"><img src="images/TabletUser.jpg" width="200">
<img src="images/smartphone.jpg" width="200"><img src="images/OutsideUser.jpg" width="200">
]
]

1. Variable lighting, conditions, computer system, etc...
1. International subject pool
1. Many people at the same time (10-100/hour), each talking to a centralized server
1. Data saved in database to allow concurrent reading/writing

<br><br>
**This may not be all bad actually depending on your research project!!**
]

---

---

- Developed by [Amazon.com](http://amazon.com)

- Originally for in-house use to detect duplicate product postings on Amazon's site .red[*]

<br><br>
.center[<img src="images/AMTLogo.png" width="300">
]

.footnote[.red[*] Nice summary in the [New York Times](http://www.nytimes.com/2007/03/25/business/yourmoney/25Stream.html)]

---

- **HIT** = Human Intelligence Task (a unit of work, e.g. a trial or an entire sequence of trials in an experiment)

- .orange[**Requester**] = an entity (e.g., researcher) who posts HITs

- .blue[**Worker**] = a person who performs the task

---

- Difficult for computer or machine learning systems
    - Provide three key words to describe an image
    - Does this photograph contain a car?
    - What event does this twitter search refer to?
    
- Traditional work
    - Write a positive review for this product online 
    - Translate this text from English to Spanish

- .green[Help with science!]
    - **Participate in my human cognition/perception/learning experiment!**

---

- 46.80% US, 34% India, 19.20% Other

- United States demographic
    - 55-65% female
    - Most make  <$60k/year
    - Median age of 30
    - Hold bachelor's and are young
    - Distribution mostly similar to US internet pop.
    
- See Ipeirotis, et al. (2010) or Mason and Siddharth (2011).

---

Tamir (2011)

---

# Stephanie Costello
<hr>
"Costello lives in a trailer at the edge of a desert town in the Southwest. She is 50 and has an associate’s degree in nursing, but she has been unable to find suitable work as a nurse. In 2007, Costello was working at a boring office job and, in slow periods, earning extra money by doing online surveys on Mechanical Turk. When she lost her job at the start of the 2008 recession, she took to Turking full time, often more. What started as a source of extra cash suddenly turned into her main source of income. According to the 2010 study, Costello’s situation may be representative of approximately one in eight Turkers in America, or one in five worldwide." (from [Marvit, 2014](http://www.thenation.com/article/178241/how-crowdworkers-became-ghosts-digital-machine))

---
class: middle

# Why are people doing it?

- One word alluded to in previous passage: <b>money</b>
- Not just money but access to jobs that are open, flexible, challenging, changing, committement-free
- No travel costs, workplace politics
- For fun?  Enjoyment of mental activities?  To fill up an otherwise boring day/evening?
- For science: people generally may view participating in science studies rewarding
- "Money, and the best way to earn it, underpins much of the discussion about AMT" (on Turker Nation forum)

---
class: middle

# Worth letting these point sink in...
<hr>

- Despite being a simple mouse click away, crowds are composed of **people**
- Online labor markets naturally are appealing to people without other income means
- In many ways this is **good**
- At same time risk of depersonalization due to anonymity.  These are real people and deserve respect and
thanks for participating in research

---

- Median wage is $1.38/hour

- Short tasks (~5 mins) award around 10 cents

- Requester can reject work and revoke payment

- And can also **award bonuses**

- Amazon takes 10% of payments

- Amazon tries to stay out of disputes

---

- Unfortunately, getting a requester account from outside the US can be a bit tricky

- Requires US billing address  and US debit or credit card

- If you can't get a US credit card, setting up the account with a US American collaborator is an option

- Or consider international/European alternatives to AMT, such as [clickworker](http://www.clickworker.com), [crowdflower](http://crowdflower.com). The latter actually uses AMT workers, but is accessible worldwide

- (the psiTurk toolbox could in principle be adapted to work with other platforms)

---

- **Convenient**  And the jspsych+psiTurk toolbox will help you even more
- **Fast**  Collect a lot of data quickly, also good for piloting
- **Affordable** Workers will sign up for $2.00 for 15-25 minute session
- **Anonymous** Subject never meets experimenter
- **Replicable** Very easy to share your experiment code

---

- **Selection Bias:** No control over who takes the experiment
    - 53% are self-identified liberals, 25% conservatives, 73% voted for Obama, see details [here](http://themonkeycage.org/2012/12/19/how-representative-are-amazon-mechanical-turk-workers)
    - Makes AMT sample probably less useful for research on ideology, or political attitudes (see [this](http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-who-problems-with-mechanical-turk-stud.html) blogpost for a discussion)

- On the other hand, 20 year old university students are a very special sample too!

- **Population size:** 
    - Amazon claims 0.5 million registered users
    - Effective population size of about 8000-10000 users

---
.task[Part 1: What is AMT?]
# Why not?

- **Contamination of subject base** 
    - Recent [study](http://www.jessechandler.com/uploads/2/8/0/5/2805897/13_chandler_mueller__paolacci.pdf) has addressed this question
    - Can be a problem for *very* widely used paradigms 
      - 56% had participated in Prisoner's Dilemma, 52% in Ultimatum game, 30% Trolley problem

- Cross talk (e.g. in AMT forums) seems to be rare (workers share information about payment, duration, etc.)

- For specific studies repeated participation can be prevented by recording worker's IDs
 - .blue[PsiTurk toolbox helps with this!]

---
.task[Part 1: What is AMT?]
# Why not?
- **Non-lab setting **
    - [This](http://www.jessechandler.com/uploads/2/8/0/5/2805897/13_chandler_mueller__paolacci.pdf) study showed
      - 27% were not alone while working on the HIT
      - 18% were watching tv
      - 14% were listening to music

- Recording how often participants change windows/take breaks might help clean data
  - .blue[PsiTurk toolbox helps with this!]

- Still, the lack of experimental control over a worker's environment might make some studies unsuitable for AMT

---

# How does the data compare to that collected in the lab?

---

- Gosling, S.D., Vazire, S., Srivastava, S., & John, O.P. (2004). [Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires](http://ww.w.simine.com/docs/Gosling_et_al_AP_2004.pdf). _American Psychologist_, 59, 2, 93-104.

- Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). [Running experiments on Amazon Mechanical Turk](http://repub.eur.nl/res/pub/31983/jdm10630a[1].pdf). _Judgment and Decision Making_, 5, 411-419.

- Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). [Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?](http://pps.sagepub.com/content/6/1/3.full). _Perspectives on Psychological Science_, 6(1), 3-5.

- Germine, L., Nakayama, K., Duchaine, B.C., Chabris, C.F., Chatterjee, G. & Wilmer, J.B. (2012).  [Is the Web as good as the lab?  Comparable performance from Web and lab in cognitive/perceptual experiments](http://www.springerlink.com/content/f0244t772070138w/)  _Psychonomic Bulletin & Review_,  19.5.

- Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). [Using Mechanical Turk to Study Clinical Populations](http://s3.amazonaws.com/academia.edu.documents/30554524/Clinical_Psychological_Science-2013-Shapiro-2167702612469015.pdf?AWSAccessKeyId=AKIAIR6FSIMDFXPEERSA&Expires=1374090987&Signature=%2B4nErhKWOQhoWYY9gpgV0EbvVa0%3D&response-content-disposition=inline). _Clinical Psychological Science_, 1(2), 213-220.

]

---

- Some members of our lab recently contributed to this study that tested some classic psychology experiments on AMT.red[*]

- Crump, M. J., McDonnell, J. V., & Gureckis, T. M. (2013). [Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.](http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0057410) PloS one, 8(3).

- Focus was on reaction time findings that require sustained attention from subjects and precise recording of responses

- And learning experiments involving more cognitive effort

.footnote[.red[*] For a summary of the paper see this [blog post](http://gureckislab.org/blog/?p=1297)]

---

Stroop

- People respond faster to congruent (e.g. .blue[blue]) than to incongruent (.green[blue]) items

Task Switching

- Faster RT if task stays the same (e.g. "Odd or Even?") than when tasks alternate ("Odd or Even?" / "Small or Large?")

Flanker

- When identifying a target (e.g. "h"), congruent flankers ("hhhhh") lead to faster RTs than incongruent ("ffhhff") ones

]

.center[.right-column[
<img src="images/StroopResult.png" width="160">
<img src="images/TaskSwitchingResult.png" width="160">
<img src="images/FlankerResult.png" width="160">
]]

---

Simon

- Targets that are spatially compatible with response key are identified faster

Visual Cuing

- Stimulus detection is faster for cued  versus uncued targets if cue is presented after short interval (<=300ms) and slower after longer interval (>=400ms)

Attentional Blink

- During rapid serial visual presentation visual target detection is impaired if target is displayed 100-500ms after another one

]

.center[.right-column[
<img src="images/SimonResult.png" width="160">
<img src="images/ContCueing.png" width="160">
<img src="images/AttentionalBlink.png" width="160">
]]

---

Masked priming

- Responding to arrow probes that are either compatibly (prime: >>, probe: >>) or incompatibly primed (prime: <<, probe: >>) at durations of 16, 32, 48, 64, 80, and 96 ms

-  Previous finding was that compatibility effects are negative for very short prime durations and positive for longer ones

- Only partially replicated: no negative compatibility effect

- Lower bound on display duration in browser! (but see, Barnhoorn et al., 2014)

---

- Classic Shepard, Hovland, & Jenkins (1961) paper

- Six different ways of classifying the same 8 items (geometric figures with 3 binary dimensions) into deterministic categories

- Generally Type I is learned faster than Type II which is faster that Types III-V, with Type VI the hardest
.center[
<img src="images/abstractstructureSHJ.png" width="150"><img src="images/dimstructureSHJ.png" width="170">
<img src="images/SHJDifficulty.png" width="550">
]

---

Although type I was learned easily, performance on all other types was significantly worse than in the original study
<br><br>

.center[
<img src="images/exp8-nosofsky.png" height="290"><img src="images/exp8-amt.png" height="290">
]

---

.task[Part 1: What is AMT?]
# Incentives too low?
There was little difference between high ($2 to $4.50 with bonus), medium ($1.50), and low-payment ($.75) groups
<br><br>

---

.task[Part 1: What is AMT?]
# Bad instructions?
Including manipulation checks that asked non-trivial questions about the task improved performance on types II to VI
<br><br>

---

1. Clinical (Shapiro, et al, 2013)

- Scores on a range of psychometric tests had high internal consistency and test-retest reliability  (r =.87)

2. Personality (Buhrmeser, et al, 2011)

- No difference in consistency on personality questionnaires at different payment levels, and absolute levels of consistency and test-retest reliability were high (r = .88)

3. JDM (Paolacci, et al, 2010)
  
  -  Compared classic JDM studies (Asian Disease, Linda, Physicians Problem) on AMT and university subject pool
  - No difference in failure rate of catch trials and pattern of behavior in studies but AMT subjects were more risk averse

---

- How much should you pay?  
    - As much as the market supports or should it match what you pay in the lab (roughly)?
    - AMT has been criticized for incentivizing companies to outsource labor without adequate compensation (for discussions see for example [here](http://priceonomics.com/who-makes-below-minimum-wage-in-the-mechanical/?utm_source=buffer&utm_campaign=Buffer&utm_content=buffer2d486&utm_medium=twitter), [here](http://economix.blogs.nytimes.com/2013/03/18/the-unregulated-work-of-mechanical-turk/), and [here](http://www.huffingtonpost.com/julian-dobson/mechanical-turk-amazons-underclass_b_2687431.html))

- Can you get IRB approval for your experiment?
    - Arguably, workers have more freedom of choice than lab participants (can stop experiment any time)
    - Wasn't very difficult to add to an existing IRB at NYU
    - Common at many, many universities now.

---

- They also share information about good and bad requesters:
<br><br>
.center[
<img src="images/Turkopticon.png" height="350">
]
 
---

1. Before running, .blue[test your experiment] on different browsers and the sandbox (AMT's testing environment)

1. Give explicit instructions and .blue[show warnings] about closing the page, going back in the browsers, etc.

1. Start collecting data in .blue[small batches]
 
1. If something goes wrong, provide an .blue[alternative payment method] for workers who are unable to finish or submit HIT
    - One way of doing this is to provide a HIT with no content specifically aimed at workers that had problems and pay them using a "bonus"
 
---

1. AMT  allows you to collect data quickly and conveniently

1. Web experiments can prevent experimenter effects and increase anonymity

1. Many classic psychology findings replicate with high fidelity
    - But  it's important to check for browser limits regarding presentation time, etc.

- It's also helpful to check participants' understanding of the task

1. Changing payment does not have a large effect performance but does affect signup and dropout rates

1. Make sure to build up a good reputation with AMT workers to make your life easier

---

# End of .blue[Part 1]
Questions?

---

# .blue[Part 2] 
# Using AMT as a Worker and Requester

---
name: part2
class: center, middle
 
.task[Part 2: Using AMT]
 
# Workflows (and tools to get started)

---

---
.center[
<img src="images/APSslides.002.jpg" width="750">
]

---
.center[
<img src="images/APSslides.003.jpg" width="750">
]

---
.center[
<img src="images/APSslides.004.jpg" width="750">
]

---

# Two perspectives:
 
1. What is it like to use AMT as a [**Worker**](#terminology)?
 
2. What is it like to use AMT as a [**Requester**](#terminology)?

# Two platforms:

1. The sandbox

- Worker sandbox ([workersandbox.mturk.com](http://workersandbox.mturk.com))

- Requester sandbox ([requestersandbox.mturk.com](http://requestersandbox.mturk.com))

2. The real deal ([www.mturk.com](http://www.mturk.com))

---

- A unit of work, or a "Human Intelligence Task"

- In our experiments, a HIT is typically an entire experiment (as might be done in the lab)
 
- However, it is up to the **Requester** to determine what constitutes a HIT
 
- Could be one trial in an experiment, and you just offer the .blue[**Worker**] as many HITs as they care to do

---

.task[Part 2: Using AMT]
# Using AMT as a .blue[**Worker**]
 
1. Sign up on Amazon's site
 
2. Search for tasks (i.e., HITs) to do online through the AMT interface.
 
3. Worker accepts a HIT.  The task appears in a frame within the AMT website.

3. Accepting a HIT means it is removed from the .orange[**Requester**] account. If the Worker decides not to complete the HIT, they have to _return_ it to the .orange[**Requester**] so someone else can do it

4. After completing the HIT, the worker signals to Amazon that they have completed the task
 
5. Payment will be credited to a worker's account some time after completion

---

.task[Part 2: Using AMT]
# Using AMT as a .orange[**Requester**]
 
1. Open a Amazon Web Services (AWS) account: [Subscribe here](http://aws.amazon.com)
 
1. Get a Mechanical Turk Requester Account: [Subscribe here](http://requester.mturk.com)
   
1. Create a "dummy" email account which you can use to deal with inevitable issues that come up with Workers
    - Workers ***will*** contact you when they have issues
    - Need a way to deal with this efficiently and quickly (your reputation depends on it!)
     
1. Use a built-in Amazon template or "External Question" approach to design your experiment
 
1. Test your code on the "sandbox"
 
1. Go live! 
 
 
---

---

###Why use them?
- Hosted directly on Amazon's servers
- Good for simple surveys (multiple choice, text fields, radio buttons)
- Easy to use (don't need your own server)
- Tools for helping you "build" the question
- Basic HTML can be included (to format texts, or load data, images, etc. from a separate datafile, for example)
- HITs can be individualized with different content for each

---

###Demo using the sandbox
- Create survey HIT as requester
- Complete HIT as worker
- Download results

---

###Demo using the sandbox
- Create a survey using [Google Forms](http://www.google.com/forms/)
- Create HIT using survey link template
- Worker sent to google forms link
- Data saved in [Google Drive](http://www.google.com/drive/) spreadsheet

---

###Advantages
- Simple tools for building a survey
- Easy to get started
- Data automatically saved to a spreadsheet
- Possible to randomize question order (with google forms)

---

###But...
- Dynamic elements cannot be included!
- Counterbalancing is difficult
- Limited use beyond surveys
- For complete control you can .blue[build your experiment as a website]
- then, link your website to use AMTs **external question** type

---
class: center, middle
.task[Part 2: Using AMT]
# Up Next

##1. Building a web-based experiment from scratch
##2. Using .blue[PsiTurk] to serve your experiment to AMT

---

### Behavioral Experiments:
- Present subjects with a task
- Control the order and timing of stimuli (images, words, sounds, video, etc.)
- Measure responses
- Store the data

### Good news! Web-pages:
- can present many kinds of stimuli
- record responses (keypresses, mouse-clicks, near millisecond precision)
- store custom-formatted data
- .blue[need to learn some programming]

---

### Challenging, but .blue[FANTASTIC skill to learn]

.center[
<img src="images/C1_Frustrated.png" height="200"><img src="images/C1_Happy.jpg" height="200">
]

---
.task[Part 2: Web-based Experiments]
# Why learn Computer programming?

### Creating Data (running experiments)
- You control the details of your experiment
- More flexibility 
- More creativity

### Interpreting Data (statistical analysis)
- Batch-analysis (saves time)
- more options for analysis
- less human error (easier to find and eliminate bugs)

### Marketable skill

---
.task[Part 2: Web-based Experiments]

---
.task[Part 2: Web-based Experiments]
# Building a web-based experiment from scratch

Web development has a lot of moving parts:

- Code a .blue[web page]
- .blue[Custom web server] to serve the experiment and interact with Amazon.
- .blue[Database] for the server to store data.

---
.task[Part 2: Web-based Experiments]
# Code a .blue[web page]:
- Structure coded in <span class="blue">HTML</span>
- Style defined in <span class="blue">CSS</span>
- Programs written in <span class="blue">Javascript</span>

# Not enough time to teach programming...
- but enough time to show how the parts work together
- see helpful learning resources on workshop website [http://APSTurk.github.io](http://APSTurk.github.io)

---
.task[Part 2: Web-based Experiments]
# Tools you'll need (they're all free!)
- Text editor: [Textwrangler for Mac](http://www.barebones.com/products/textwrangler/)
- Web-browser with Javascript console: [Google Chrome](http://www.google.com/chrome)

---
.task[Part 2: Web-based Experiments]
# HTML
- text commands to format a webpage

.left-column[
```html
<html>
	<head>
		
	</head>
		
	<body>
		<p> Hello world </p>
		<button> Click me </button>
	</body>
</html>
```
]
<iframe srcdoc="
		<body>
			<p> Hello world </p>
			<button> Click me </button>
		</body>" </iframe>

---

<body>
<p id="p1"> Hello world </p>
<button id="b1"> Click me </button>
</body>
</html>
```

```css
#p1 {
 	font-size: 24pt;
 	color: blue;
 	text-align:center;
}

#b1 {
 	font-size: 20pt;
 	color: black;
}
```
]
<iframe src="sampleHTML/sample1.html"</iframe>

---
.task[Part 2: Web-based Experiments]
# Javascript & JQUERY (js library)
- dynamic control of web-page elements

.left-column[
```html
<html>
<head>
<link href="sample1.css" rel="stylesheet">
<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script> 
<script src="basic.js"></script> 
</head>
<body>
<p id="p1"> Hello world </p>
<button id="b1"> show </button>
<button id="b2"> hide </button>
</body>
</html>
```

```css
#p1 {
 	font-size: 24pt;
 	color: blue;
 	text-align:center;
}
#b1 {
 	font-size: 20pt;
 	color: black;
}
```
]

.right-column[
```javascript
$(document).ready(function(){
	$("#b1").click(function(){
		$("#p1").show();
	});
	$("#b2").click(function(){
		$("#p1").hide();
	});
});
```
<iframe src="sampleHTML/sample2.html"</iframe>
]
---
.task[Part 2: Web-based Experiments]
# Presenting a series of Stroop items
- dynamic control of web-page elements

.left-column[
```html
<html>
<head>
<link href="sample1.css" rel="stylesheet">
<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script> 
<script src="basic2.js"></script> 
</head>
<body>
<p id="p1"> Stroop? </p>
<button id="b1"> Next </button>
</body>
</html>
```

```css
#p1 {
 	font-size: 24pt;
 	color: blue;
 	text-align:center;
}
#b1 {
 	font-size: 20pt;
 	color: black;
}
```
]

words=["red","green","blue"];
colors=["red","blue","green"];
counter=-1;

$("#b1").click(function(){
counter++
$("#p1").html(words[counter]);
$("#p1").css("font-color",colors[counter]);
});

});
```
<iframe src="sampleHTML/sample3.html"</iframe>
]
---

.task[Part 2: Web-based Experiments]
# Collecting responses and times
- dynamic control of web-page elements

.left-column[
```html
<html>
<head>
<link href="sample1.css" rel="stylesheet">
<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script> 
<script src="basic3.js"></script> 
</head>
<body>
<p id="p1"> keys and times? </p>
</body>
</html>
```

```css
#p1 {
 	font-size: 24pt;
 	color: blue;
 	text-align:center;
}
#b1 {
 	font-size: 20pt;
 	color: black;
}
```
]

$("body").keypress(function(event){
 d = new Date();
 $("#p1").html(d.getTime()); 
 $("#p1").append(" " + String.fromCharCode(event.which));
 });

});
```
<iframe src="sampleHTML/sample4.html"</iframe>
]
---
.task[Part 2: Web-based Experiments]
# Basic Stroop demo
.left-column[
```javascript
$(document).ready(function(){
words=["red","green","blue"];
colors=["red","blue","green"];
counter=-1;
data=[[]];

//PRESENT STIMULUS
$("#b1").click(function(){
$("#b1").hide();
counter++
$("#p1").html(words[counter]);
$("#p1").css("color",colors[counter]);
onsetTime=new Date();
trial=[words[counter],colors[counter]];
});

//COLLECT RESPONSE
$("body").keypress(function(event){
 responseTime = new Date();
 RT=responseTime-onsetTime;
 key=String.fromCharCode(event.which);
 data.push(([trial,RT,key]).join());
 $("#b1").click();
 });
 
});
```
]

---

.task[Part 2: Web-based Experiments]
# Stroop Demo with Bells and Whistles
- [Load demo](https://dl.dropboxusercontent.com/u/14462007/JavaScript%20Tutorial/StroopDemo8.html)

---

Lots of resources for learning: E.g., [codecademy.com](http://codecademy.com):

---

# If you need help

1. Tutorial materials are available online. [APSTurk.github.io](http://APSTurk.github.io)

---

# So you've made your experiment, How do you run it on AMT?
## Web-Based Experiment -> Amazon Turk

---

.task[Part 2: The mechanics]
# External questions 
 
- You are responsible for:
    - Hosting the experiment
       - **At minimum** you need a computer with a web-accessible HTTP server running (a static IP address may simplify things)
       - You probably want to avoid trying to host it over WiFi

- Posting your link to Amazon
     
    - Communicating back to Amazon when the HIT has been completed
     
    - Approving completed HITs for payment in a timely manner
     
    - Dealing quickly with worker issues
     
    - Saving your data in a secure place, etc..

---

- Interactions with AMT that psiTurk handles:

- Submit a request for HITs

- Provide a link to an "ad" that describes the HIT for Workers (this appears within the AMT site when they click 'View a HIT in this group')

- After HIT is accepted, provide a link to the experiment

- After HIT is complete, provide a button Worker will click to inform Amazon that they have finished. This submits a form with their identifying information to AMT.

---

---

---

---

---

---

---

---

---

- **Command line tool** (this is the "player")
  - Allows you run experiments from a computer of your choosing
  - Allows you to interact with AMT to pay people, assign bonuses

- **psiturk.org Cloud-based services** 
  - provides secure hosting (Ad Server)
  - experiment exchange (system for sharing experiments between researchers)

---

---

---

---

---

---

---

---

**System requirements **

- psiTurk works only on OS X and Linux. Windows is not supported (cloud-based option coming soon!).

<br>
**Software dependencies**

- **Python**

- If you've never used Python before, we recommend you use the [Enthought Python distribution](http://www.enthought.com/repo/.epd_academic_installers), 
  because it comes with pip, a tool for easily installing Python packages.

---

**Software dependencies**

- **pip** (open source app store for Python)

- If you installed Python using Enthought you don't need to install pip separately.
    If you want to use another version of Python, you may have to install pip, for example like this:

```bash
cd /tmp
curl -O 'https://raw.github.com/pypa/pip/\
master/contrib/get-pip.py'
python get-pip.py 
# If you get a permissions error, try
# sudo python get-pip.py
```

---

**3. Installing psiTurk**

- To install psiTurk simply run:

```bash
pip install psiturk
# If you get a permissions error try
# sudo pip install psiturk
```

---

# Command line features

**1. Experiment server**

- Hosts lab and web experiments
  - Provides database and websserver functionality
  - Blocks repeat participants

**2. Interactive shell **

- Interface for Amazon
  - post ads, pay workers, change settings, etc.
  - Launch and debug code

**3. Javascript library**

- Provides common experiment functionality
  - save data to server, load content, log window changes

---

class: center, middle
# .blue[Part 3]
### .gray[Checkout psiturk.org and documentation]

---

# Workflow

**1. Local testing & debugging**

- Test experiment locally
  - Javascript (expt logic) ←→  expt server

**2. Remote testing (AMT sandbox)**

- Test that your code works on Amazon's servers
  - Javascript ←→  expt server ←→ AMT

**3. Live (AMT)**

- Collect data!
  - Javascript ←→  expt server  ←→ AMT

---

# Draw together example

***Goals***

- Test experiment locally
- Collect & export data
- Post a HIT in AMT sandbox (requires AMT account).
- Maybe collect live data

**1. Load demo experiment code**

```bash
psiturk-setup-example
cd psiturk-example
```

**2. Start command line**

```bash
psiturk
```

**3. Command line overview** (demo)

---

class: center, middle
# .blue[Part 3]
### .gray[Begin semi hands-on psiturk demo]

---

```bash
psiturk-setup-example
```

???

---

???

---

???

---

???

---

???

---

# exp.html

```html
<html>
  <head>
    <title>Experiment</title>
    <script src="static/lib/raphael-min.js" type="text/javascript"> </script>
    <script src="static/lib/jquery-min.js" type="text/javascript"> </script>
    <script src="static/lib/underscore-min.js" type="text/javascript"> </script>
    <script src="static/lib/backbone-min.js" type="text/javascript"> </script>
    
    <script type="text/javascript">
    // Subject info, excluding condition and counterbalance codes.
    var uniqueId = "{{ uniqueId }}";
    var adServerLoc = "{{ adServerLoc }}"; // the location of your ad (so you can send user back at end of experiment)
    var mode = "{{ mode }}"; // is this running live, sandbox, or in debug mode?
    </script>
    <script src="static/js/utils.js" type="text/javascript"> </script>
    <script src="static/js/psiturk.js" type="text/javascript"> </script>
    <script src="static/js/task.js" type="text/javascript"> </script>
    <link rel=stylesheet href="static/css/task.css" type="text/css" media="screen">
  </head>
  ...
</html>

```

???

---

???

---

???

---

???

---

# Stylesheets

```css
body {
  background: black;
  color: white;
  text-align: center;
  font-family: Verdana, Helvetica, sans-serif;
  margin: 0 auto;
        margin-top: 100px;
  width: 800px;
}

@import url(http://fonts.googleapis.com/css?family=Crimson+Text:400,600italic);
h1 {
        font-family: "Crimson Text";
        font-size: 42pt;
        font-style: italic;
}
```

???

---

???

---

???

---

A small library that will help you:

- Handle page transitions
- Load images
- Record and save data

psiturk.js will .blue[automatically]:

- Register if user resizes their Window
- Register if user switches to another tab/window

---

.task[Part 3: Building dynamic web experiments]
# psiturk.js: <br> .blue[Condition and Counterbalance]

```javascript
// Put this at the top of your file:
exp = new Psiturk(uniqueId, adServerLoc, mode)

// returns subect's server-assigned condition:
exp.taskdata.get("condition")
// returns subect's server-assigned condition:
exp.taskdata.get("counterbalance") 
```

---

```javascript
exp = new Psiturk(uniqueId, adServerLoc, mode)

// Records data for the current trial
// A list of things which will be put in columns in the spreadsheet
exp.recordTrialData(["anything", "you", "decide", 1, "L", "CORRECT"]) 
// Unstructured data, e.g. the answer to a question in a form
exp.recordUnstructuredData("Age", 28)

// Save data to server
exp.saveData()
```

---

- `psiturk.taskdata` — Data about task/subject
    - `.get("condition")` — Get server-assigned condition number
    - `.get("counterbalance")` — Get server-assigned counterbalance number
    - `.get("useragent")` — Get browser's user agent string.
    - *Includes event listeners to record resize/focus change.*
- `psiturk.recordTrialData` — Accepts list of items, which will be a row in `trialdata.csv` along with the subject's unique identifier
- `psiturk.recordUnstructuredData` — Accepts `key` followed by `value`, encoded along with unique identifier in `questiondata.csv`
- `psiturk.saveData` — Send all data in `taskdata` to the server.
- `psiturk.finishInstructions` — Warn participants not to close window.
- `psiturk.teardownTask` — Allow window to close.

---

???

---

These are currently dependencies of `psiturk.js`, we also use them in all our experiments.

- [Jquery](http://jquery.org) – Manipulating the website
- [BackboneJS](http://backbonejs.org) – Organizing your code
- [UnderscoreJS](http://underscorejs.org) – Helpful functions, e.g. `_.shuffle()` to shuffle stimuli

---

# End of .blue[Part 3]
Questions?