Usability testing

This is a meta issue we should thanks @gibiansky for.

It's composed from a few extract of private mail exchange, and is made to publicize the "Data"
(if we can call this data).

I'll edit later for better readability and/or move each issue in a separate thread.

---
- One user could not figure out what the dark circle in the top right was (the kernel busy / available indicator). She spent the entire half an hour session wondering aloud what it was but never figured it out. So perhaps the "kernel busy" indicator (right now a circle that turns light or dark) could also be replaced with a spinner of some sort.
- One user was very confused at all the cell types besides Markdown and Code, and why they were necessary. But if I recall correctly these are going away in 3.0 anyways.
- One user was a little frustrated by the last of find and replace, but it seems that this issue has also been noticed in the issue tracker.
- One user, after trying my app, wanted to see how it looked in a normal web browser. She was frustrated at one point because she was using Safari, and had a very small screen (effectively), and had a very difficult time reading the keyboard shortcuts, because the keyboard shortcut dialog renders very badly (does not scroll, too, making it impossible to see most of the shortcuts if you have a small screen).


I think I recall seeing that Safari was on the list of supported browsers, so this may be worth filing an issue for (I had to work around this in my app with a lot of extra CSS for the keyboard dialog, since WebViews use the same renderer as Safari, and because for my users it is actually a very big issue and occurs on a lot of people's computers).
- One user spent a long time at the beginning trying to figure out how to make a new cell. She clicked around everywhere in the notebook and in the end found out that if you click on the white area at the end of the notebook, a new cell appears. However this took her a while, so perhaps this could be made more discoverable, I'm not sure though.
- One user tried to export some HTML and had some issues with lists being formatted incorrectly. The Markdown cell renders correctly if you have some text, a single newline, then a list, but if you export it to HTML it requires there to be an extra newline between the text and the list. I suspect these are issues with the pandoc / etc integration, but I looked around the issue tracker and could not find an issue that exactly matched this. Do you know if this has been observed and reported and if there is a fix planned? (I know that you guys are planning on shipping some different python-native conversion utilities, or something?)

I very much agree with what Brian said – many of these issues were due to not understanding the IPython notebook abstractions. However, one thing I was surprised by was that the command mode did not seem to cause people any problems. I don't think any of them realized there was an "edit" and "command" mode, but they didnt ever get confused why something worked in one mode but not the other – they very quickly learned that they could click outside a cell to get into "command" mode and then do what they wanted.

Also, I only really tested with three or four users, two of whom had used mathematica somewhat extensively, so these results are in no way scientific. If you continue with your own user testing, I found it to be very helpful to give people a list of tasks and then just watch them do it, offering no guidance whatsoever. I also asked them to talk out loud to explain what they were thinking. Here is the list of tasks I used, retailored towards ones you might use:
1. Go to www.ipython.org/notebook.html
2. Explain: What is IPython notebook and what does it do
3. Download and install it
4. Make a new document and give it a name.
5. Make a markdown cell
- Put a heading in it
- Put a list in it
- Put some math in it
- Evaluate the cell to render it
6. Edit the first markdown cell in your new document.
7. Make a code cell with some code
- Write a for loop with a print
- Evaluate the cell to run the code
8. Make a code cell that makes a plot of f(x) = sin(x^2)
9. Turn off spell check for markdown cells
10. Explain: What is a kernel?
11. Restart the kernel
12. Make an interactive plot that plots f(x) = sin((x + y)^2) where a slider varies y
13. Find the shortcuts for making new cells and moving between cells
14. Save your notebook as a webpage
15. Install a new package (try "sqlalchemy" or "configobj")

This usually took users between 25-45 minutes to complete, and they got a bit frustrated at times, so there were times where after they could not figure something out for 10 minutes I would give them an important hint.

Hope all of this is useful. As before feel free to share it in any way you see fit.