There’s something rotten in our industry, and I am sick and tired of it. It’s this fundament: easy things are easy until they become complicated, then they are really complicated.
It’s hard to illustrate, but it’s like a red thread through everything. Working with parallel execution is one thing, where threading complexity is reasonable until you actually try to use all those CPUs. Working with databases, especially when it has to scale, is another thing.
Peter van Dijck has a list of presentations on website scaling. The recurring theme is that everything goes well until you demand performance, when you need to start doing crazy shit. It’s not surprising that there are a bunch of tactics (like sharding) developed to handle this. It is surprising that you have to give up on significant functionality to do so. It is surprising that you have to architect this yourself when it happens.
I wish we could all just stop the world to resolve this.
“Okay, stop. Guys working on larger capacity hard drives, flash drives, cheaper memory, better power supplies, fundamentally different CPUs and cures for cancer, you can keep going. The rest of you, spend five years fixing the fundamental issues. Make something new to replace threads, or at least implement something like NSOperation as low-level as you can get away with. Solve database access performance problems for me – if dynamically creating queries is suicide because you have to calculate new execution plans several hundred times per seconds, then think up something else, because we can’t design stored procedures for everything and retrofit every advance we make in programming. Implement UTF-8 and UTF-16 everywhere, and drop the old stuff and stuff that’s not government mandated in the next major version.”
I wish for someone in every field to step forward, assess the situation, come to terms with that it’s madness, talk to everyone at every level and together design good solutions that spans the range, stops the pie-throwing between layers (”you’re so slow!”, “no, you’re inflexible!”) and pushes everything forward. Something like ZFS and Wide Finder.
That is real innovation. That is leadership. That is capacity. So you’ll have to excuse me if I’m not interested when “Web 2.0″ now means that a fifth of the participating sites are mildly useful, unlike the last time.
[...] – 2.4Device driver updates causing Vista to deactivate Jesper takes another cracking shot at why stuff keeps staying broke, despite constant development marching forward: “Okay, stop. Guys working on larger capacity [...]
By waffle on fundamentally wrong – smack|foo · 2007.11.05 00:34
NSOperation isn’t quite enough. It sounds good, but its a bit too primitive, especially for a closed-source framework.
OpenMP (which is NOT OpenMPI) looks interesting, because the compiler does the work for you. It’s part of the next gcc release for leopard that integrates with llvm, which is also interesting.
But what the world really needs is “staged event driven architecture” that are integrated into the OS.
Here’s a link:
http://www.eecs.harvard.edu/~mdw/proj/seda/
By http://openid.aol.com/opnnbstrd · 2007.11.06 00:06
NSOperationis primitive, but it’s an improvement over the thread pool offered in, for example, .NET in that the number of threads currently used can be automatically tuned by the system (by default) and in that you can specify dependencies and just keep loading operations onto the queue.I am familiar with the gist of OpenMP – pragmas to specify that, for example, this
forloop is ripe for auto-parallelization. Another is .NET’s upcoming Parallel FX where you can use aParallel.Formethod, provide a numeric range and a closure and get a similar parallelizedforloop but with more sophisticated stuff like work-stealing, where all threads keep their queues filled at all times by dequeuing items from the other queues to make the job more efficient.Solutions on the individual levels of OpenMP and NSOperation should co-exist. Some solve the small problems, some solve the bigger aggregate problems. The best thing would be to have stuff like SEDA too, of course. The more abstraction, the better.
By Jesper · 2007.11.06 00:58