One of the most popular conversations to have on iPhone development mailing lists is, “How long has your app been waiting in review?” This almost always lead to a comment by somebody about how the review process is nonsensical and hopelessly broken, etc. It shouldn’t bother me, but it does, because that attitude is disrespectful and ignorant of the scale of the problem Apple is tackling. Everybody thinks they could do a better job of everything when they are sitting in an armchair.
Anyway, I wound up writing a long email about my theory on how the App Store review process probably works, attempting to use actual data. I’m reproducing it here with some light editing.
First, here are the facts:
- The App Store opened July 10, 2008 (about 8 months ago).
- According to Apple’s iPhone OS 3.0 presentation:
- There are over 25,000 apps on the App Store.
- In February, 96% of submitted apps were approved.
- In February, 98% of approvals were in 7 days or less.
- Podcaster was in review for about one month before rejection for “duplicating iTunes functionality.”
- Several web browsing apps were in review for a month or more, before being approved more or less at the same time. The same thing happened to fart sound apps.
- Tweetie was submitted around March 2 and rejected on March 10, because an F-bomb appeared on the Trends screen.
- App Store growth (by number of apps) has been accelerating.
On average, Apple has been adding apps to the store at a rate of 3,125 per month. The pace is actually accelerating (it took three months to reach 3,400) so the rate today is probably closer to 4,000 per month. That works out to 1,000 per week, or 25 apps per hour, assuming a 40 hour work week.
If each application review takes about an hour, you would need a staff of 25-30 testers to maintain this rate. They probably install the app, tap around, look for problems, etc., then make an approve/reject decision.
In an ideal world, the entire staff would be highly intelligent and could be trusted to make carefully reasoned decisions in every scenario. In California, where Apple is located, those people are doing more interesting jobs. The vast majority of apps are going to be fairly clear cases, so it would make more sense to hire cheaper and have the testers work from a checklist. The checklist might look like:
- App does not require additional payment to unlock functionality.
- App uses standard icons as prescribed by the interface guidelines.
- App does not use any of the seven words you can’t say on television.
- App does not show boobies.
If this were true, it makes sense that Tweetie was rejected rather quickly: the Tester, who probably doesn’t use or care about Twitter, was working down his list, saw the curse word (which is an instant reject), hit the reject button, and started testing the next app in his queue. After all, he’s got to meet his quota if he wants to keep his job.
On the other hand, Podcaster was in review for about a month before it was ultimately rejected. What’s the difference? Podcaster was rejected for duplicating the functionality of the iTunes app in a then-unreleased version of the iPhone OS. (Unfair, but nobody said life was fair.) This is a much less clear-cut situation. In addition to approve and reject, the Tester probably also has the option to pass the app along to a committee. Again, you probably don’t want to bother the committee too often, but sometimes you’re not sure what to do, or you don’t have the authority to release information about unpublished Apple software.
The committee probably has to do things like check with the Legal Department and iPhone OS Engineers and Designers. They debate what should pass and try to revise the checklist to handle similar cases more clearly in the future.
The committee probably also looks for similar cases to deal with them as a group. That would explain why the web browser apps were released from review limbo at roughly the same time: they probably clarified internally what counted as duplicating Safari, then re-reviewed them. Same goes for the fart machine apps, as Apple clarified their definition of obscene.
Based on my professional experience, that is how I would organize things. (I might have each app checked by two testers, budget-permitting.) The view from the outside seems to confirm the basic idea that they are gradually improving the process as they approve more apps and update their checklist.
Given the volume of apps they are processing, it’s not surprising that there are going to be some mistakes like what happened to Tweetie: but that was hardly a fiasco, it was eventually approved.
If you’re going to say that Apple should have had a flawless process in place on Day One… well, I’m sure the view is lovely from up there on your high horse. Before the App Store began accepting submissions it would have been impossible to write one set of rules that would clearly apply to every application. So they did the best they could and have revised it over time, with experience.
Now, if you want to object to the rules (that Apple must approve the apps in the first place, that you can’t do porn, etc.) that is certainly your right. And if you want to complain that Apple is too secretive, yeah, but it’s worked so well for them that I don’t think they’re going to give it up.
But given that, the process hardly seems broken.