waffle

(As reported to Apple under Radar ID 5793259. Written to heavily stress how this hurts Apple, in order to induce that special kick-them-in-the-wallet brand of sympathy.)

Update, three days later: “This is a follow up to Bug ID# 5793259. After further investigation it has been determined that this is a known issue, which is currently being investigated by engineering. This issue has been filed in our bug database under the original Bug ID# 5788773.”

Title: App Store-only distribution makes beta testing impractical
Product: iPhone SDK
Version/Build Number: N/A
Classification: Enhancement
Reproducible?: Always

Summary:
The approach taken by Apple to limiting distribution of applications to the App Store will raise the probability of a low quality experience of iPhone applications due to difficulties in coordinating beta testing.

Steps to Reproduce:
1. Develop an iPhone application of a non-trivial size.
2. Reach the point where it is necessary to assemble a team of beta testers that can’t be assumed to all have access to the source code and/or the signing certificate.

Expected Results:
Apple provides a way for any iPhone developer to redistribute iPhone applications outside of the official App Store track, and a way for iPhone user to install these applications.

Actual Results:
Apple does not provide such ways.

Regression:
N/A.

Notes:
As any application grows in capability and features, it will become increasingly harder for anyone, including the developer, to test everything about the application. As is established, most developers will enlist the help of users to help beta test the applications. If these users are not themselves developers with valid certificates and are not shipped the source code of the applications, they may not reasonably be expected to be able to beta test the applications.

There are three alternatives a developer can take to this, and all three are bad for Apple and the developer:

1. The developer tries to do as much beta testing on her own as is possible. It is statistically likely that she will not catch all bugs that her beta testers would do, had she been able to distribute the application to them. This is bad for Apple since the final product will be of worse quality, since there will have been less effort spent on beta testing and quality assurance.

2. The developer uses the App Store as a channel to deliver the beta. While the intended beta testers will be able to download and install the beta, so will anyone else, which means that people who have not been properly advised regarding the quality of the application will be exposed to an application of significantly worse quality than the final version of the application will deliver. This, like alternative 1, will unleash a product on the App Store that is not the quality of the final product, and this is bad for Apple; however, it is also bad for the developer who won’t be able to, should she choose, conduct a closed beta program. I think Apple, of all companies, would see the value in being able to do so, even if you don’t have plenty of spare iPhones or QA resources.

3. The developer simply gives up on the application since she won’t be able to deliver a product whose quality is up to her standards. This is bad for the developer, who won’t be able to deliver a product; it is bad for Apple, who won’t be able to provide as wide of an application line-up and it is certainly bad for the user, who may be unable to find a suitable application where there otherwise would have been one available.

I can suggest a few workarounds, but the one that I think will gain most traction for Apple if it were to adopt one is to allow the use of alternate data sources for the App Store, where beta testers can just enter another repository and download their application the same way they usually do, without the developer needing to expose the lower-quality, unfinished application to the rest of the world.