Ayan Bhattacharya is Assistant Professor of Finance at The City University of New York, Baruch College. He has a PhD from Cornell University and his research focus is financial economics, especially financial market design and asset pricing.

Many analysts have marked the record sale of Flipkart’s stake to Walmart this month as a turning point in India’s startup ecosystem. The thinking behind the optimism is simple: as developed markets age and slow, big players can ignore India only at their own peril. After all, no serious venture would want to miss being part of the India story, especially as the Chinese miracle plateaus off! Less than three years back, however, it was a completely different tale. Many international investors were then busy marking down their India portfolios as a series of startups—most notably Housing.com—began imploding under the pressure of scaling up. What changed between then and now? One could come up with a variety of reasons, but underlying all of them would be a single idea: change in expectations.
Unlike any other field, finance, almost entirely, is fueled by expectations. Markets—whether formal ones like the financial exchange, or informal ones like the neighborhood kirana store—name a price almost always before the fact, in expectation. But what exactly is this expectation?

1. How Do We Expect?
For a field so dependent on the notion of expectation, you would presume finance to have a clearly articulated, experimentally verifiable definition of the notion. That, sadly, is not the case. In fact, the deeper you dig, the more slippery it becomes. Financial accountants will tell you confidently that expectation is all about analyst forecasts, but push them about where forward looking parameters in analyst models come from, and the consensus will disappear. Financial statisticians, on the other hand, will give you a fancy formula for expectation: just multiply the probability of a scenario by its outcome, across all scenarios. But push statisticians about where exactly these probabilities come from, and they will go silent. Expectation, at its core, seems to be closely linked to our human ability to learn, and the budding field of cognitive neuroscience is increasingly making clear to researchers the huge gaps in our current understanding of our own brain’s ability to learn and form expectations. Yet the business of expectations has always been at the heart of finance.
The basic “atoms” of finance are prices, and the price of any asset, whether physical or financial, is the value that a buyer hopes to derive from its possession in the future. This value is all about expectations, because the future is yet to unfold when the transaction is sealed. Different people may expect to derive different value from possession, or they may expect the future to unfold differently—thus they bargain and trade. So how does finance come up with explanations for prices despite the many gaps in our current understanding of expectation formation? Well, as we’ll see below, by a shrewd sleight of hand!

2. Expecting Without Expectations
Many early economists struggled with the notion of expectations. John Maynard Keynes, arguably the most influential of last century’s economists, spent many years thinking about the origin of probability and expectations before embarking on a full-time career in economics [1], and many of his influential macroeconomic theories demonstrate a deep appreciation of human expectations. Yet, he never put forward a rigorous formulation. It took many years and many false starts before the field hit upon two novel ways to handle expectations.
The first was the concept of rational expectations. In a pioneering paper in 1961, John Muth, then at Carnegie Mellon University, proposed the idea that rational economic agents’ prognosis about the future should be consistent with the economic models used to predict the future [2]. The underlying principle was one of consistency. Sitting today if an agent posited a model of the future that included the agent himself, yet did not behave according to his own model’s prediction when the future actually unfolded, he would be irrational! Such irrational agents would surely not be interesting economic agents, it was believed, since they would fall a prey to Darwinian survival. A similar idea animated Harsanyi’s extension of game theoretic equilibrium to incomplete games [3]. Thus economic agents’ expectations of the future was encapsulated in the models they built today, and at the same time, the models they built today had to be accurate descriptions of the future, since all agents were rational. In effect, economists had managed to replace the neurobiological mechanism of expectation formation with the logical apparatus of consistency! Many of the widely influential theories of finance that explain asset prices, starting with the Capital Asset Pricing Model, rely on this logical apparatus.
The second was the technique of no-arbitrage, or no-free lunch. No-arbitrage simply meant that there could be no free profit opportunities in the price system, because if there were, everyone would go after them, and they would evaporate instantaneously. No-arbitrage started with a bunch of given expectations (or prices) and was agnostic about where these baseline expectations came from. The power of this theory was in using the technique of no-arbitrage to derive other expectations in the economy once the baseline expectations were assumed as given. Once again, the underlying principle driving the technique was consistency. The baseline expectations could be arbitrary in principle, but all other expectations in the economy had to follow consistently from them. Once more, the neurobiological mechanism had been cleverly avoided using the logical apparatus. The Black-Scholes-Merton option pricing and many other theories of finance exploited this technique to great effect.
Dissatisfied with these techniques derived from logic, some finance researchers began dabbling in ideas from cognitive psychology in the hope of understanding human behavior better. This led to the birth of behavioral finance. While the new approach provided many new insights, it still depended on a notion of consistent expectations for aggregate predictions. The problem really was that researchers did not (and still do not) fully understand the internal algorithms of the brain. Cognitive psychology largely depended on outcomes of experiments to infer how people think. While this was an improvement for finance, the black box of actual expectation formation still remained out of bounds. At the same time, the apparatus already developed by the logic based techniques were mathematically rigorous, reasonably simple to use, and provided useful predictions. Over time, with minor tweaks, the behavioral methods were co-opted into the logical framework.

3. Can Logic Fail?
The big question, then, for researchers and practitioners is: when and how—if ever—does the consistency based logic underlying expectations fail? Can financial modelers know in advance, before events really move off the grid? In other words, sitting through the Housing.com fiasco in 2015, could one have rationally expected the bounce-back in India’s startup scene? The question is also important for regulators, since regulatory approval, too, is based on anticipation of future impact on the competitive landscape. So for those reading between the lines, most of the briefs to the US district judge deciding the $85 billion merger between AT&T and Time Warner in the last few months have been really about competing visions and expectations of the future [4].
Researchers realize that understanding the departure of expectations from predictions is an important question, and it is high up on their “to-comprehend” list. Yet the honest answer at the current moment is that we do not know how it happens. Two paths seem to be emerging in the literature, however. One, pioneered by the late Stephen Ross, is the Recovery theorem approach [5]. Briefly, the idea is to recover accurate expectations from empirically available market prices, rather than rigidly impose theoretical no-arbitrage conditions. The second, inspired by the artificial intelligence literature in computer science, is to approximate the process of expectation formation through variants of machine learning algorithms [6]. Both are still nascent approaches, and it is anybody’s guess as to which path will be successful. Real life expectations, after all, are way more complicated than Dickens’ fictional Great Expectations!

—

[1] Christian P. Robert. (2011). “Reading Keynes’ Treatise on Probability,” International Statistical Review / Revue Internationale de Statistique, Vol. 79, No. 1, pp. 1-15.
[2] John F. Muth. (1961). “Rational Expectations and the Theory of Price Movements,” Econometrica 29, pp. 315–335.
[3] John C. Harsanyi (1967). “Games with incomplete information played by “Bayesian” players, I-III. part I. The Basic Model”. Management Science, special issue: Theory Series. INFORMS. 14 (3), pp. 159–182.
[4] The Wall Street Journal, “AT&T-Time Warner Trial,” May 08, 2018. https://www.wsj.com/livecoverage/att-time-warner-antitrust-case
[5] Stephen Ross (2015). The Recovery Theorem. The Journal of Finance, 70(2), pp. 615–648.
[6] Sergiu Hart and Andreu Mas-Colell. “Simple Adaptive Strategies”. World Scientific Publishing, Singapore, 2013.