Tuesday, September 29, 2009

The Uncanny Valley of computational research

(Update, Oct 17: Greatly expanded with the aid of Chandru, the most inexhaustible source of all fundaes on postgrad in IITM. He also insists that this post gives a biased view that only modeling sucks, and so an edition detailing the heinous atrocities and frauds of experimental research is forthcoming.)

I've been very irritated by almost all of computational modeling research I've been reading up on. I specifically mean research in modeling and simulating systems that we (humans) have not built from scratch. For example, trying to model blood flow in an artery, or the mechanical behavior of tissue or modeling the weather. It struck me that this situation is similar to an idea in human-robot interactions, called the Uncanny Valley.

(Image, as mostly everything you'll find on this blog, from Wikipedia)

This is a graph of a feeling of 'familiarity' or 'liking' vs how human a robot looks. Near the left, an object, say a robot, looks nothing like a human (for example an industrial press) and there's no feeling we attach to it. As it starts to look more and more humanlike (like for example the Asimo), people start finding it 'cute' and likable. But then, as it as it gets to the point of being almost human, uncannily human, there suddenly is a repulsion for it, a kind of disgust like what we feel for a prosthetic or a made-up corpse. Further on however, we're barely able to make out the difference and the familiarity rises up rapidly.

I think a very similar effect occurs in modeling. In the beginning, models are simplistic, produce no consistent predictions and no one cares about them ("The earth is flat"). Then we have models that are simple, and yet have very useful predictive qualities. We can measure the Earth if only we assume a few numbers are 'much smaller' than others (Eratosthenes)! An Engineer's First Sin - just assume a certain linear dependence and you can measure something as elusive as time itself (sin(x) ~ x, simple pendulum)! Say a particle can only have two states and can interact with only its nearest neighbors, and poof! You can explain an event as amazing as phase-change (the Ising model)! Assume everything that can be computed can be written down as instructions on a piece of paper, and voila! You come up with fascinating limits on what a computer cannot do (a Turing machine, and the Halting problem)! (And the subsequent hilarity when the Halting problem is outsourced)

But then, the party stops. Here's a bunch of negative reactions about modeling research:

1. It's pointless: the model is extremely complex and the improvement in results are just not worth the effort. The current models are 'close enough' to make any extension seem useless. There's no intellectual pleasure.

2. Talk about a double whammy: modeling seems to have overtaken reality. Many models, especially in bio+engineering areas, are no longer a critical step in deciding anything and their predictive power is never brought into question in any non-trivial problem. Even a model that is used in a critical circumstance is eaten whole by a very roomy engineering safety factor. This is brushed off by saying "Our models will take 20 years to find their use! Look at potential instead of bare reality!"

3. Modeling is also beginning to be used very deviously to demonstrate that a lot of work was done, and therefore the work is worth publishing. Reminds me of a joke that says CFD stands for Colorful Fluid Dynamics. Another related abuse is to put in a section on modeling in an otherwise purely experimental paper. The modeling is supposed to prove that 'what we think is happening is what is actually happening', but in most cases it serves to fill a boilerplate template mandated by some journal.

4. Pauli is once said to have lamented about a paper he was asked to read - 'This is not right. It is not even wrong'. I'm shocked at the number of modeling papers out there which don't even have a hypothesis, or any hint of the model's predictive abilities, usefulness or even a whiff of a reason as to why some poor sod of a grad student wrote 50,000 lines of code. The import of the entire work seems to be "This work was done".

5. A model that has 56 parameters can match any dataset. It's just pushing the problem of 'understanding what is happening' to the level of finding the right parameters. And that level happens to filled with grunt work and boredom.

6. A misconception that better tools lead to better research is rife. Nowhere is this more true than modeling. A supercomputer cluster is trivial to buy - just a few thousand dollars and you have very decent 'computing resources'. I find it especially grating when universities announce with great pomp the opening of a 'High Performance Computing Center'. This is the textbook definition of cargo-cult.

7. Among a host of adapted dick size metrics, a common one is 'full 3D simulation'. A vast majority of engineering modeling consists either of 1D or 2D models, because till the last decade it was impossible to even dream of full 3D simulations. It is debatable where exactly full 3D simulations are necessary - but such trifles don't stop people from merrily performing 3D simulations in the aforementioned High Performance clusters. This unfortunate tendency has resulted in pitiable cases of 5th year PhD students being told that their work will be 'sufficient' only if they re-test their hypothesis (if they were lucky enough to have such intellectual luxuries) with a fancy solver on a cluster, and finding that everything is wrong, and lest they take solace in knowing at least that for a fact, inconsistent.

8. Quite the same complaint applies to 'non-linear', 'parallelizable' and 'scalable'. I think it's like inventing a fine temperature controller for the swimming pool on top of the Titanic. Note well however, that this is a more refined and arguably less criminal form of intellectual fraud. The more dangerous epithets of ill-repute are 'nano-' (in a former birth, 'micro-'), 'bio-', 'ab initio', 'biomimetic' and 'multi-scale'.

9. Continuing on the 3D bashing, even presenting results is extremely hard to standardize. Contours aren't cool. Slices take too much space and are too hard to interpret. But who cares about the results and understanding them? Animations, preferably with lighting and a suavely-accented voiceover, are in. A sectioned 3D artist's impression adds more to the mojo than a graph with sensible axes. These days, in addition to the panegyrics and pedigree and GRE and grades, a graphics design background would be a wise thing for modeling labs to insist on if they are to keep the reams running (or what has become equivalent, surviving).

10. At a deeper level, most models' greatest strength is the audience's ignorance, and this is best seen in interdisciplinary work. If you don't know a lot about something, you cannot honestly be critical about it. Even if you are, you do not have the moral right or standing to express it. In fact, the fundamental paradox of knowledge is that the more you know, the more you realize how little you know, and so your self-measured moral right to critique something diminishes as you learn! Russel once said "The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt". This is a most happy outcome for the fraudulent researcher. In a work on 'Distributed finite element modeling of cancer growth', the solid mechanician can't rightfully criticize the parallel computing algorithm or the biology; the algorithms expert/programmer can't criticize the mechanics or the biology; and the oncologist can't criticize the mechanics or the algorithm.

11. Extending on the previous point, collaboration seems to be similar to Simpson's Paradox in statistics, where each part has one trend (here, honesty) and the whole the opposite trend. This is what makes it possible for a US president to be elected to power even though the total number of votes he has is lesser than that of his rival.

12. (Thanks Serdar) Some researchers take the idea of changing topics to remain 'fresh and motivated' a tad too far. Put their finger in every pie, collect all the low-hanging fruit and jump on to a new topic. This is particularly easy in modeling, where neither resources or, as we have come to see, knowledge is necessary to be productive. It is no longer cool to work on a topic for long enough to truly advance frontiers of knowledge. Stay till the Papers/Effort ratio peaks, and move on. These dilettantes are pejoratively termed 'butterflies'.

13. In the end, simulations are easy on the pocket. Take 6 computers and 6 desperate grad students, mix well, add a pinch of conference publication hopes, funding threats and sublte emotional blackmail and voila! You have a paper factory all oiled up. Continue for a few years and you'll be light-years ahead of anyone trying to actually do science. If experiments are your thing, there's a slightly modified but equally despicable protocol.

14. Another aspect Petre rightly pointed out: Fudge Factors. We've all heard of Skinner's constant (also called Flannegan's finagling factor), the number which, when multiplied by, divided into, added to, or subtracted from the answer you got, gives you the answer you should have gotten. While it goes about giving people a smile, it's evil cousin lurks deep in simulation codes magically making them coincide with experiments/predicted results. Nearly any work that simulates a non-trivial non-toy phenomenon will have a couple of constants that are either not known, or are heavily disputed. In fact, in some places, the reason simulations are done is because those constants can't be known, and so simulations can give a kind of sample space. But that is quickly lost when the goal is to produce 'good' results. In an example I know well of, that elusive variable was the conduction speed of a cardiac cell. The speed varies with time, depends on location, and changes drastically when a cell is taken outside the heart or if a probe is put in! So right now there's no way to measure it, and so the constant was 'set to appropriately typical values' to ensure the results were 'correct'.

15. This is perhaps the most finicky and least acceptable of objections, but still: when the famous Four-Color problem was finally solved using a computer in 1976, there was a murmur of dissent. Yes, we finally know for sure that only four colors are required to color any map, but no living human knows why it is so. You need a few hundred pages of analysis to reduce the problem to a couple thousand non-trivial base-cases, and then a computer to verify that indeed all these base-cases can be colored with 4 colors. So one more cold fact is known about the world, and there's nothing more to it.  It wouldn't have been very different if the skies opened up and a thunderous unquestionable voice declared "4 is enough!". Similarly, there are some works which simulate flow of some idealized fluid in some idealized turbulent regime, and proudly claim that they have done it using 200,000 CPU hours. This kind of work is neither here nor there - it is too simplistic to be actually experimentally verified, and it is too complicated for anybody to truly understand.

    Every one of these woes carries echoes of being almost there, but not quite. Taking up the task of removing the 'not quite' can be a frightfully unrewarding and disappointing experience.

    To complete the isomorphism, we should look at very complex models that make very useful/impressive predictions. Flight simulators and today's video games come to mind - the complexity of the Physics or Graphics engines used is mind-boggling, and yet, they are so convincingly real that after a few trials, you think the Gravity Gun in Half Life 2 is just another weapon (as opposed to OMFG AWESOME!!!1!!). Astronaut training programs have a necessity to be painfully accurate, and do quite a good job. As we go along, our very definition - that the 'high-end' of modeling is when you can't distinguish between the model and 'reality' - starts to get to us. Is language a model for expressing thought, or is it something on its own? Is your perception of the World, which determines your reaction to everything that happens to you, a model of the world, or is it You?

    --
    Update: I found two fantastic articles by Bertrand Russel, which convincingly strike at the very roots. Pure joy!

    1. 'On Youthful Cynicism', the 'Truth' and 'Progress' sections in particular.
    2. 'Icarus, or the Future of Science'.

    Update (May 23,2010): A very nice wiki article: Map-Territory relation
    --

    18 comments:

    Mahesh Mahadevan said...

    The uncanny valley of computational research - clearly a brilliant spot of _that_ uncanny resemblance.
    I'd appreciate it if you could link to the PDF version of Raghuvamsha canto#1 on Sanskrit Documents.
    Among introductions self-effacing,
    would you warn us for ourselves we're bracing,
    With some kALidAsa,
    and your parihAsa,
    We might need some de-interlacing.

    From recently watching this man, and reading this article of sorts, I guess, maybe shameless presentation presentation of biased opinions is a bit underrated.

    tejesh kinariwala said...

    amazing insight!. The best part was the two questions at the end.

    Karthik said...

    You don't even need 56 parameters! Here's a model for every dynamical system ever.

    Fantastic post. I nodded my head off reading it.

    Mohan K.V said...

    @Nai: I'm itching to start a nerdcore rap thread, but I'll pass for now :) Strong one! And the vid and article were great!

    @Tejesh: Thanks, I'm glad you liked it1

    @Karthik: Holy @#$(@*!$*%&(!! That equation is bloody AWESOME! I can confidently say that this is one of those wildly incredible comments that totally change your entire view of a topic. I'll head right in to see how exactly it manages work, but where did you come across this? Apart from Mathworld, I don't recognize any of the other sites that pop up when I search for UDE on Google. Plus, MW says the first one was discovered as late as 1981! Freaking awesome!

    Karthik said...

    'tis a long-ish story, but here goes:

    A professor of mine was explaining how you could construct a cellular automaton (CA) to generate a specified time-series, and mentioned that with a complex enough rule, (by increasing, say, the number of neighbouring cells influencing a given cell) you could generate any arbitrary output for an arbitrary time. He mentioned that such non-elementary CAs usually aren't good models for physical systems, just like really high order ODEs (>4) aren't good models. He casually added that Wolfram and he used to set graduate students on UDEs (and universal-ish CAs) all the time for fun.

    I googled.

    I don't understand at all how they work; when you do, could you do a breakdown for the rest of us? (Like you did with the Quine a long time ago. That was amazing.)

    Mohan K.V said...

    Wow, that's amazing!! I'm really beside myself with amazement because the space to which that MathWorld article belongs is completely orthogonal to the space of all articles on the internet that I can possibly access by myself: It's unlikely to come up on geek soc-net sites like reddit because it's _too_ involved; I couldn't have come across it on Wikipedia because there's no page on it yet (I created a stub just today); Nor is there any page that talks _about_ a universal differential equation, except for an orphan page on Keith Briggs; No blog or non-journal media ever talks about it; The highest number of citations for any paper on UDEs is 39, and that's for Rubel's original paper in 1981, and all of them are either by people working on UDEs or some sufficiently-deep aspect of abstract math; also, the 1981 original paper has a very contorted, hard form anyway. The more easier one, by Briggs in 2002, has only 3 citations. If it wasn't for you, there's no way in hell I'd have known about this!

    Besides being beside myself, I'm also all puffed up and on cloud g64 [*] and generally overly unhealthily intoxicatedly flattered that you still remember the Quine post! I really am! It will take me many weeks to get back to my original undeformed configuration :D

    I'll surely write more extensively on this as I figure it out, but here's an sneak peek meant exclusively for the patrons of this comment space :-
    The thing I'm most curious in this about is information compression. In the simplest UDE (Briggs 2002), there's only one constant, n, and some 4 BCs (which can be trivial). All the info in any interval is compressed into that one parameter n. This is almost exactly the like in the math quine, where all the info about the quine was stored (very cleverly) in the ordinate value of the plotted range (the constant 'a' in my post). I have a half-finished blog on Chatin's brilliant brilliant idea that compression means comprehension (Must read!). Let's see how that works out. Also, Rubel's original paper says it is 'related' to the idea of a universal Turing machine. That seems very interesting, too.

    But my dear, dear chap, all this understanding-blunderstanding business aside, we should be talking about po$$ibl€ u$€$ here! For precisely the reasons I've mentioned above, we can make a killing publishing a million papers in a million journals modeling a million physical phenomena! Imagine if we published a paper on modeling the stock market with this! We could rake in billions, start a hedge fund of our own and win a Nobel prize or two, lose a couple billion dollars of investor money, and if we're lucky, wreck the world economy! We're gonna be rich I tell you! Rich rich rich! All we need to do is to ..take care of the people who know about this already. I hear the Tunisian mafia is particular good with academician ...assignments. :P

    [*] - A very, very, very brilliant article on Big Numbers by Scott Aaronson. Actually, this (along with Chaitin's article) serves as a very nice basic guide to Turing machines and computation, too.

    (I know something's ..different when my comments are longer than the post, contain footnotes referred to by geekification of common idioms, are obnoxiously littered with links, pitch billion dollar proposals and need a terminal introspective parenthetical remark)

    Mohan K.V said...

    Please ignore the bit about information compression in 'n' - that was a wrong guess. I figured out how it works, and will write a post asap.

    Dileep said...

    Wow! That short last comment destroyed me. So,.... ahem.... the Tunisian mafia was it??.....

    Karthik said...

    OK, I just talked to the professor again, prodding him for details. I got a few facts out of him. He talks on a level of abstraction that's very hard for me to hang on to for more than a few minutes, so it's all over the place. Here's what he had to say:

    1. Rubel was a friend of his. Apparently, much of the work Rubel did on UDEs, he never wrote down or published. He found a couple of them (UDEs), then discovered a general rule for constructing and/or modifying any differential equation into a UDE. It begins with writing the equation in the form xdot = {Fourier series in x} with randomly chosen coefficients. (Which is clearly a UDE.) Then it gets fuzzier. With appropriate choice of coefficients, he claimed to be able to convert every system of ODEs of sufficiently high order into a UDE.

    2. He tied the idea of the UDE to analog computers, analogous to the way Turing machines are related to digital computers. He gave (over dinner, apparently) an explanation of the limitations of analog computers using his analogy.

    3. Rubel spent the last few years of his life disillusioned with his findings, and gave up on Physics entirely shortly before he died an eccentric preaching the futility of math modeling. We can only guess at what he must have discovered.

    I've read Aaronson's essay; I think he linked to it at some point on Shtetl-Optimized. Thanks for the link to Chaitin's essay. (It's in my read-queue; the first two paragraphs provide an inkling of things to come.)

    Which brings us to wealth creation by academic obfuscation. If we pick our journals carefully, we might actually pull it off. Any attempt at modeling the stock market, though, is bound to be attacked on grounds of post-hoc fallacy. If we have fake identities ready, we could still scam 'em once before disappearing. Ahh, Villa in Rome.

    You've figured it out?!
    Wow. That took a while to sink in. I gave up halfway through Rubel's paper. I can't wait to read your blog post.

    The compression bit threw me off- I thought n was just a parameter for a family of UDEs, and by definition a free variable. ('a' from quines was a large _constant_, no?)

    (A comment that's longer than the post, contains footnotes referred to by geekification of common idioms, is obnoxiously littered with links, that pitches billion dollar proposals and requires an introspective terminal parenthetical remark is clearly the epitome of comments, an ideal every comment must aspire to.)

    Karthik said...

    (I did get the idea that he was approximating an arbitrary function piece-wise by a family of ("nearly elliptic") functions, each member of which satisfied the UDE. And so the only real trick was to realize that once you can eliminate the parameters of this family by successive differentiation, you get an ODE with only constants. But that's not a real understanding, is it? It just feels... hacked together, as proofs by construction are often wont to do. It's a lucky catch of sorts. Man, I'd love to learn the general theory behind this method!)

    Mohan K.V said...

    @Dileep and Karthik: Wait, I'm sorry, that last short comment came out far stronger, smug-ger and more narcissistic than I intended. The true sentiment is better approximated by "I think I may have a plausible idea of how Rubel came up with the UDE, in a way similar to my Quine post".

    I'm writing up the post now, and it is more-or-less what Karthik wrote in the 12:26 comment but with a few added bells and whistles. But beware! As is my wont , I'm dramatizing it with such pretentious pomp and parade that an innocent reader skimming the post can't be faulted for thinking I solved the Riemann hypothesis in my sleep with just my left little toe.

    @Karthik: Wow! Just who is this prof?

    Re. #2, Proof by Personal Communication, hmm?

    Re. #3, OMG, I'm ready to accept Rubel as my Prophet this instant. Anyone who is disillusioned by math modeling is my hero.

    Re. $tuff: Pick our journals carefully? Boy, who filled your head with such dangerous notions? Numbers baby Numbers! Just the other day, we figured out that a certain prof I know had to have produced more than one paper a day to explain his publication count. Needless to say, he's one of the biggest swinging dicks in the area. Also of interest, an Ignobel laureate in Literature, 1992.

    Re. Finance: Have you read Taleb's Fooled By Randomness? If no, PLEASE stop what you're doing and get yourself a copy. The post-hoc fallacy is for the birds, we're Financial Engineers now! Villa in Rome, hm? I'd prefer a Chateu in South France, but I'm a reasonable man, I'll negotiate.

    Re. n, you're right, I jumped the gun.

    ('an ideal every comment must aspire to' - ROFLMAO :D :D :D)

    Kutraj said...

    Nice one!

    Nodded in agreement too :)

    One thing at the end. Game physics are nowhere near as complex as the phenomena they attempt to model.

    For one thing, some vague, shady Activision game (don't recollect the name) was advertising using CFD to simulate cloth movement! It turns out that all they were doing was solving a very simplified version of the Navier Stokes equation, and manipulating the solution (which reminds of the good old mech lab 'sessions' in insti :)

    Anyway, with games, it seems the solution only has to LOOK real, so that the gamer THINKS it is. Their predictive power is thus, in many ways, rather limited.

    One more thing I really really agree with is the 'publishing because you've DONE something' thing. Thats a little sad to be honest.

    Mohan K.V said...

    @Kutraj: Thanks! When the Revolution comes, the true worth of us modeling-critics will come forth with great glory. Until then, "hmm, I think a couple more graphs and we can cleanly get enough pity points to make the December conference" :) .

    Games - tread carefully young man, this is a minefield. First, clothes are no simple business! L.Mahadevan got the very prestigious MacArthur geniu$ grant this year for modeling wrinkling in clothes! (Google him; perhaps I should be honest and mention that he also got the Ignobel prize in Physics in 2007 for exactly the same work). Sheet wrinkling is an unbelievably complicated problem!!

    Second, while I agree that there are a lot of shortcuts and manipulations, there is still some mindblowing physics in today's games. Consider just the simple aspect of lighting. Today's simplest FPS essentially implements almost the entirety of the publication spectrum of 80's SIGGRAPH, and that's VERY hard math. If we start about collision detection, a lot of research-grade molecular dynamics software is developed alongside physics game engines!

    And come now, even if they cheat and manipulate, they at least have _some_ results to show for it. It's like downplaying Magellan's achievement of circumnavigating the globe because he used trade-winds instead of rowing all the way :)

    Kutraj said...

    Ah, true. Now I have to rant a little bit, sorry :D

    On a vaguely related note, do you still follow GPU stuff? If you do, you might want to check out nVidia PhysX - they now boast of GPU accelerated physics! (If that second name sounds familiar, remember Ageia? The thing that Hegde fellow started, with their physics co-processor stuff? Yeah, nVidia bought them out! :D)

    It seems most of the SIGGRAPH stuff is now being put onto GPUs and the n-core processors that Intel and Co. are dreaming up(n>8). But you're right, that was some stuff they had back then, hell it was a pain in the butt to even think of doing all of those things in realtime. Now they're doing n-body simulations on GPUs, and with nVidia's CUDA thing, even the famous trio of Tom, Dick and Harry can do it on their own GPUs!

    Stud analogy with the trade winds though! Lol. But you're still assuming he traveled around the globe, when he could've just MADE you believe he did ;) :D

    Karthik said...

    Well...

    Ba said...

    I am in partial agreement with you. While arts and science may not yield tangible results, they have been surviving till date purely because of the intellectual fun they can give.
    Engineering is just a packaging of physics and science saying that it will use those concepts and see what people need and bridge the two by creating technologies and products for sale. So at some level, science has been of help in engineering.
    (Now, I am giving my biased opinion) Engineering is almost done. Almost all human requirement has been met to form a stable situation and we need to relax in this state until we get bored and tired of it till a majority of the people get convinced that they want to become even more lazy and therefore need more engineering and thus more science.
    Since that has not happened yet, doing engineering and science now has little meaning. We are perfectly fine with the way things are new technologies hardly have any use.
    (I lost track of why I started writing this comment :| )

    Mohan K.V said...

    ...now.

    Satyaanveshi said...

    'Mithyaabhaas' would be a good translation of simulation, which is frequently used to express our perception of the world.

    The problem with the world, I think sir, is that they are trying to understand nature using science not spirituality, :).