Handbook for CREED experiments
Tips, hints and internal rules
Joep Sonnemans

[ Overview] [ Design] [ Computer programs] [ Financials] [ Internal CREED seminar] [ Reservation lab] [ Instructions] [ Questionnaires] [ Scenario] [ Pilots] [ Recruiting] [ Receipts] [ Logbook] [ Problems] [ Afterwards]

New! Sander Onderstal's "An experiment in 40 steps". It was written for a specific experiment but may be useful to others.

Overview

Design of experiment

Designing an experiment starts with formulating the (main) research question. While working on your design, always ask yourself: "Is this design really the best way to answer this question?".

There are some important boundary conditions for your design. Deception of participants is forbidden in the CREED lab because of the negative external effects on other experiments. In fact, for all other elements of your design that may have negative externalities, contact CREED seniors.

Most common problems:

• A common beginner's error is a too complicated design, very often as an attempt to stay close to an existing economic model. If your question focuses on a specific part of the model, you may better design an experiment without the (for your research question) irrelevant parts of the model.
• Too many conditions. While discussing a design you will come up with other questions of the kind "it would be interesting to know whether it matters if you change this or that". It is tempting to have many conditions because of these secondary research questions. There are three disadvantages of having many conditions. First, because of financial or time restrictions having many conditions will probably mean less observations per condition and therefore less statistical power. Second, interpreting results is much harder when you have many conditions. Third, it is easier to write (and publish) a paper when you have one or maybe two specific research questions and only few conditions. Too many conditions can make the paper hard to follow for the reader.
• Incentive structure. The behavior of the participants should have a substantial influence on the earnings (or at least the participants should have the impression). You will have more noise in your data if the participants think that their decisions have only a little influence on their earnings.

Computer programs

You can make your own program or ask our programmer (Jos). If you plan to do a lot of experiments in the future, programming yourself can be a very good investment.

If you program yourself, thare are many possibilities. Most used are:

• Ztree.
  Advantages:
  • very easy to program
  • Many examples around
  Disadvantages:
  • If you want something that Ztree cannot do, you are stuck (you cannot adapt Ztree itself)
  • Limited possiblities to adapt the look-and-feel and design of the screens.
• php/mysql.
  Advantages:
  • Myabe a little bit less easy to learn (but Joep is willing to give introductinairy courses)
  • Extremely flexible
  • Millions of examples around and many helpful internet forums
  • Also possible to run on other computers without installing software. (computers only need a browser and internet access).
  Disadvantages:
  • If you need real time feedback without delay, programming becomes moer difficult.

Asking Jos Theelen to make the program
If you ask the programmer you have to fit into his planning. The programmer can only start if you have very clear ideas about your design. You have to explain the order of events in the experiment, you have to draw pictures of the computer screens (what information is seen by the participant) and you have to specify the output. Changing the design after the programming has started is a waste of time and effort and the programmer (rightly) will hate you for that.

The most critical part of software development is testing. The experimenter is responsible for testing the software. If there is a software failure during the experiment, the experimenter is to blame for not testing well enough. Testing should be done systematically: all conditions (parameters), appropriate number of participants logged in, appropriate number of periods. Test also whether the output file is correct! It is advisable to have a restart-option in order to relogin after a hard ware failure (or a participant who turns of the computer...).

If you need a (small) change in the program because referees asked for additional treatments and you would like to do this on short term, don't try to sneak to the front of the waiting list, but ask the programmer who is standing in the waiting list and try to convince these colleagues that it is important that you will be helped first.

Most common problems:

• Underestimation of the time you need to program and (especially) test the program. If the program has to be changed the night before the experiment you cannot test it well enough (sometimes the solution of one software problem creates another problem only observed after rigorous testing).
• Testing with n computers and running the experiment with m>n participants and then finding out that the group compositions or communication is going wrong.
• Testing not all conditions
• Testing only a for a few periods to find out at the experiment that the program crashes at round 12 or so, because of a data buffer overrun, too small arrays, or whatever.
• Testing the program, eliminating all errors, and at the last moment decide to make a small unimportant change (like the appearance of the computer screen). If you change the program, you should test again!

Financials

Payment of the participants consist of a show up fee and the money they can earn during the experiment. In most cases the show up fee is 7 euros. However, if they can earn a large amount during the experiment, you may consider a smaller show up fee or no show up fee (but the participants that are sent away will receive 7 euros).

If more participants show up than you need, you have to send away participants with 7 euros each (first ask for volunteers. Determine random who has to leave if there are not sufficient volunteers.)

Because the behavior of the participants is not known before the experiment (of course, that is why you want to run the experiment) you cannot exactly determine the average earnings. Based upon theoretical predictions and experience you can make an estimate of the average earnings. Average earnings should be approximately 10 euros per hour (including show up fee). Short experiments pay a little better (to compensate for the effort to come to the lab). Experiments where you expect large differences between earnings should also pay a relatively high average (to make the lowly paid participants not too unhappy to participate again in CREED experiments).

We have to work for every euro we spend on participants. If you find out you need a lot of money, you may consider using less participants by cutting conditions (has also other advantages) or for example using a smaller group size.

At least three weeks before you need the money, you have to tell the CFO (CREED Financial Officer, Karin) because it takes some time to get the cash from the bank. You also need your OEN (Official Experiment Number). The day of the first session (or the day before that) you will have to sign for the cash you get. The form you sign is an excel-file, you will get an electronic copy on which (after the experiment) you fill in the cash you return, and the amounts on the receipts you hand in.

Internal CREED seminar

In the internal CREED seminar you explain the background of your experiment, you present the research question(s) and you explain your design. It may be helpful if you also specify any problematic decisions you had to make designing your experiment. You can also show the program (if available) and use this as a (limited) pilot. If the experiment is complicated, it can be helpful to show the experiment first and explain the background, design etc afterwards.

Some CREED staff members make their best remarks while sitting behind a pint of beer. Going to Krater afterwards and talk a little more about the experiment can be a good investment.

Most common problems:

• Attendance. The underlying principal of the internal CREED seminar is reciprocity. While you profit from the intelligent remarks of your colleagues, they expect you to be there when they need your remarks about their projects. Remember that negative reciprocity is also possible: if you find the experiments of your colleagues not interesting enough to attend the seminar, they are likely to return the compliment.
• Not so common, but unforgivable: not bringing cake to the internal CREED seminar
• Taking too much time (especially discussing the background), exhausting the audience and receiving less feedback than possible. The seminar should be internal, not eternal.

Reservation lab and Official Experiment Number

After you have presented an internal CREED seminar and you are confident you can run the experiment within a few weeks, you make a reservation for the lab with the Experiment Coordinator (Joep). Karin can provide you with an OEN (Official Experiment Number), to be used on recruitment forms and receipts.

Scenario

Running an experiment is not difficult, but there are a lot of things you have to think of, and you want every session to be an exact replication of the other sessions. Therefore you need a scenario. A scenario is a complete description of all procedure. Typically it starts with all the things you have to do before the start of a session (including software), what to do if too many or too few participants show up, what to say in the reception room, what to do and say during the experiment, what to answer to specific questions, etc. Download an example of a scenario (word file in Dutch).

Instructions

Most computerized experiments have instructions on screen. One of the nice things of instructions on screen is that you can interactively check for understanding. Multiple choice questions focus the participants on specific (important) parts of the instructions, but are not that good for checking understanding (everybody can answer a MC question after some tries). Therefore also use open form questions, for example where you ask the participant to look up things in a table.

Even if you have instructions on screen, you will also need a summary on paper. Try to fit this on one page (with additional tables if necessarily).

It is hard for the designer of an experiment to judge the clarity of instructions. Ask colleagues or naive friends to read and comment.

Questionnaires

In some experiments questionnaires are essential (for example if the research question is about experienced emotions) and in some it is only an easy way to get extra information. Making a good questionnaire can cost a lot of time and effort. In psychology it takes months or years to produce and validate a questionnaire and you are overly optimistic if you think you can do it from scratch in an afternoon. Look around, ask, borrow or steal if you are not prepared to spend a lot of time.

In my opinion the use of a questionnaire at the end of the experiment is always a good idea. It cannot influence the decisions because it is after the experiment and it costs no or only a little time (after the last decisions of the experiment you have to produce the receipts anyhow, you can do that while the participants fill in the questionnaires). The benefits can be enormous, especially if the behavior in the experiment is against your intuition. In that situation, the questionnaire can give ideas about what was really going on.

It is a good idea to use both open and closed questions. Some researchers dislike open questions, because answers can be hard to interpret and coding can cost a lot of time. However, there is a conventional wisdom that the inclusion of open questions improves the reliability of the closed questions and can be worthwhile even if you decide not to code them. The reason is that a respondent who feels he doesn't get the chance to give his opinion can be irritated and badly motivated to fill in the questionnaire. They can always loose their egg (?) in open questions like "Can you tell me about your reasoning during the decision making?". Furthermore, although the answers to the open questions are not easily reported in the paper, it can help you to understand any strange behavior or misunderstanding.

Download an example of a questionnaire (word file in Dutch):

Pilots

Pilot with colleagues
If you want to know whether the instructions, screens etc are easy to understand, you can ask colleagues in a pilot.

• This is only worthwile when you still have time to make changes to your experiment (so don't do this a few days before the experiment)
• A pilot with colleagues is NOT for testing software: you can do that yourself.
• Don't forget to bring the cake (or drinks afterwards).

Pilot with student subjects
Sometimes a pilot experiment with paid participants is needed if you are unsure about specific parameters. For example, if you have no idea about teh number of periods you can play within 2 hours, or you need a baseline condition without floor or ceiling effects (so the behavior in the experimental conditions can be more extreme in both directions), a pilot is a good idea. If you are only a little unsure, you can plan your first session on Friday. If things go as you expected, you continue with the other sessions as planned, if not, you change your design over the weekend and consider the Friday session as a pilot.

Recruiting

• New experiment After a new experiment is defined the "add sessions" page (see below) is entered
• Open existing experiment
  • Modify experiment
  • Send email announcements (one per second; takes a while but makes it more unlikely that the email is labeled as junkmail)
  • Add sessions (Standard or more complicated conditions for participations can be set). Read the instructions on the site
  • Delete session
  • Show sessions
    • Modify session
    • Send email reminders
    • Make lab-list for this session
    • Add spontaneous showups this session (experimenter needs name and Students ID)
    • Finish session (very important, needed for the requirements of other experiments)
• Search (this is a simple search-tool. Subjects can be selected. If a more complicated search of the database is needed, contact Joep)

Safety
+Experimenters can only login from the uva-network
+The experimenters login is from a subdirectory (makes it a little harder for hackers to find the login page)
+Experimenter password is encrypted (if someone hacks the database the password is still secret)
-All experimenters can change experiments and sessions. It is possible to adapt the sytem that only the person who defined an experiment can do this, but this is impractical because most experiments are done by more than one experimenter.
+Subjects need password
-Subject password is not encrypted (otherwise the emailsystem of forgotten passwords couldn't be used)

Standard receipts

Logbook

After the experiment

• Clean up the lab
• Remove any links on the desktop of client-computers
• Return the remaining cash and the receipts to the CFO ASAP!
• Make a back up of all your data, and also of all experimental material (like programs, instructions, questionnaires, etc)
• The time between doing an experiment and getting published can be longer than you would like. Therefore it is a good idea to have some kind of diary during your statistical analyzing, to remember what you did and (especially) why.