Weather Forecasters are Better Forecasters than Stock Market Experts

Weather forecasters are said to have markedly better batting average making predictions or are far more accurate prognosticators than most stock market experts. (Well this applies to foreign private weather forecasters and not the Philippine government counterpart.)

Justin Rohrlich at the Minyanville writes,

According to New York Times statistical wunderkind Nate Silver, the National Hurricane Center’s accuracy has improved 250% over the last 25 years.

More accurate weather predictions benefit the economy, boosting the efficiency of businesses like FedEx (FDX), which employs 15 in-house meteorologists, as well as companies working offshore, like BP (BP) and Transocean (RIG). Weather is such an important factor in financial markets that Goldman Sachs (GS) employs staff meteorologists, as do Citigroup (C) andJPMorgan Chase (JPM).

While meteorologists have improved, other analysts -- specifically financial ones – are still off the mark more often than not.

“In November 2007, economists in the Survey of Professional Forecasters -- examining some 45,000 economic-data series -- foresaw less than a 1-in-500 chance of an economic meltdown as severe as the one that would begin one month later,” Silver writes.

“Why are weather forecasters succeeding when other predictors fail? It’s because long ago they came to accept the imperfections in their knowledge. That helped them understand that even the most sophisticated computers, combing through seemingly limitless data, are painfully ill equipped to predict something as dynamic as weather all by themselves. So as fields like economics began relying more on Big Data, meteorologists recognized that data on its own isn’t enough.”

So the admission of the knowledge problem is one crucial factor contributing to the weather forecaster’s edge.

I’d add that the unwillingness to think outside the box has been another key variable to why stock market experts underperform.

More…

In a slightly larger than usual nutshell, the crux of the issue was this: Weather forecasters have an “awareness of uncertainty” about the natural world that “causes these experts to manifest a lower overconfidence effect than experts from the other domain.”…

Further, financial analysts were also found to be unwilling or unable to be self-critical after a failure -- something that Raymond Dacey, Professor of Finance and of Statistics and Adjunct Professor of Philosophy at the University of Idaho, suggested to the authors “could pertain to the clients.”

Simply put, the “Disposition Effect” has to do with risk attitude and what Shefrin and Statman colloquially term “get-evenitis” -- an aversion to loss realization.

Another major obstacle is the overconfidence bias

I’d add that egotism has always been a hurdle to self discipline.

Here is a relevant investment ‘war’ tip from Sun Tzu’s Art of War

If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle

Flooding from Heavy Monsoon Rains Exposes Central Planning Failure

From the Manila Bulletin,

Malabon City government officials are criticizing the Camanava Area Flood Control and Drainage System Improvement Project for “failing to meet the expectations of the residents.“

After the meeting with Engr. Carla Bartolo, head of the project, Acting Malabon City Mayor Antolin Oreta III said the Camanava flood control project, “did nothing as regards to the perennial flooding particularly in Malabon.“

The Php 5.2 billion project also covered nearby areas in Caloocan, Navotas and Valenzuela, but a big bulk of the amount was purportedly utilized for constructing pumping stations, navigation gates and polder dikes in Malabon, according to Malabon City Engineer Edgar Yanga.

Two pumping stations were supposed to serve Caloocan, Malabon and Navotas areas, Yanga said.

The city officials noted some “flaws“ in the construction. Yanga said, “Ang expected na gagawin ay hindi ginagawa, paunti-unti. No target completion.“

He added: “The project is very much delayed. Five years na delayed.“

The project was started in 2003 and was supposed to be fully operational by 2007.

In the meetings with city officials, including 21 barangay chairmen, Bartolo attributed the delay of the project to the presence of some informal settlers covered by the project and changes in the conditions of the locations.

Oreta said the city continues to experience floods, citing some barangays which were affected by the heavy rains spawned by typhoon “Gener“ and recent tropical storms.

First of all, the nature of politics has all been about the blame game, where political agents benefit from stepping on someone’s shoes. Critics make the cavalier presumptions that under their guidance such problems will unlikely emerge.

Second, censures become the mechanical reaction once an event has already taken place. The usual culprit has been the private sector, but in one of the unusual case above, one government agency excoriates another.

But since politics has mainly been about the fetish for short term problems and fixes, fleeting popular concerns leads to intuitive shifts in policy directions.

This known as the time inconsistency dilemma, as per Wikipedia.com,

situation where a decision-maker's preferences change over time in such a way that what is preferred at one point in time is inconsistent with what is preferred at another point in time

Yet since experts cannot predict on the precise dislocations from weather disturbances, social policies result from “whack the mole” dynamics or from “fighting the last war” or to shifting priorities. So there will never be an end to central planning failures on reactionary based populist social policies.

Third, finger pointing will always be about mismanagement, deficiency of funding and or the lack of regulatory oversight.

In reality, since government treats the symptoms than the problem, the outcome will always be a gamut of unintended consequences.

Paradoxically, failures and inefficiencies (and corruption) will be rewarded through demands for more taxpayer expenditures.

Yet the biggest fundamental flaw emanates from the public’s mysticism over the infallibility of the nanny state.

Ironically many, if not most, have been jaded to the reality of serial failures of central planning, as I previously wrote:

Two more important things to drive at:

The first is the KNOWLEDGE problem.

The fact is that while there are instruments to help predict the changes in the weather, that knowledge is limited. This means that policy responses will ALWAYS be insufficient, no matter what they do.

The second point is that these has been all about the HOT POTATO problem—everyone seems to toss the responsibility to another party.

Everyone has been HARDWIRED to EXPECT that the government must and shall deliver us from environmental disruptions and disasters.

Yet no matter the horrible track record, we maintain this illusion of infallibility.

People cannot seem to accept that government are composed by people, and like everyone else, has limitations in the possession of knowledge.

Most of the dogmatic belief on the ascendancy of the state emanates from economic ignorance and mass indoctrination.

As the great Professor Ludwig von Mises warned

What makes many people blind to the essential features of any socialist or totalitarian system is the illusion that this system will be operated precisely in the way that they themselves consider as desirable. In supporting socialism, they take it for granted that the "state" will always do what they themselves want it to do.

Don’t Expect the Government to Solve Social Problems Caused by Weather

Spot the mistakes of this article.

From Sunstar.com.ph (bold emphasis mine)

MALACANANG appealed for understanding from the public following another confusion in the announcement of class suspension in Metro Manila and nearby provinces that have experiencing heavy rains since Monday night.

Deputy presidential spokesperson Abigail Valte said since there is no storm signal, the responsibility of suspending classes is left to the discretion of the local chief executives.

“We will be asking for a little bit of your patience because it is a new devolved system and our local government units will need to also get used to that system,” she said.

To remind the local officials with their task, Malacañang again posted on the Official Gazette Executive Order 66, which authorizes municipal mayors, as respective chairpersons of respective Local Disaster Risk Reduction and Management Councils (LDRRMC), to cancel classes and work of government offices during inclement weather.

“We would like to take this opportunity to remind them that the responsibility to suspend classes in times of inclement weather when there is (sic) no storm signals belongs now to the local government officials,” Valte said.

She added it is stated in the EO that suspension of morning classes must be announced by 4:30 a.m. so as not to inconvenience students, school staff and parents.

Classes from pre-school to high school levels in Metro Manila were cancelled Tuesday morning due to continuous rains caused by a low pressure area last seen at 40 kilometers west of the capital.

Several parents, however, complained about the late announcement of class suspension as their children were forced to go to school early in the morning despite the flooding.

Local Government Secretary Robredo, in a radio interview, admitted the need to improve the local government units’ announcement of class suspension.

Robredo said mayors, village chiefs, and school principals should coordinate in making the necessary announcement regarding cancellation of classes during heavy rains since they are the ones who see the situation on the ground.

Ok this news is an ex-post observation of the recent flood. Hindsight is 20/20. It’s easy to finger point based on the past.

Yet the problem is tomorrow. The problem is accurately anticipating the weather and its potential impact in both quality and quantity dimensions to affected areas and the social policy response.

The next problem is that families expect schools to make the necessary announcements. Unfortunately schools have NO EXPERTISE in forecasting weather. So it will be judgment call for schools.

The same problem plagues the local government units, hardly any of them have the ADEQUATE knowledge of the impact of nature’s challenges. So it will be judgment call for LGUs.

This leads us back to the national government through the government weather bureau PAG-ASA. Again unfortunately, outside conventional storms, it seems that PAG-ASA hardly can forecast accurately the nitty gritty of the weather changes. So it will be judgment call for the national government through PAG-ASA.

Even in conventional storms, PAG-ASA has hardly been successful in accurately predicting the exactitudes of nature's disturbances.

All these reveal that the fundamental problem is the failure of centralization in weather forecasting. But media's and the political approach has been to finger point.

Two more important things to drive at:

The first is the KNOWLEDGE problem.

The fact is that while there are instruments to help predict the changes in the weather, that knowledge is limited. This means that policy responses will ALWAYS be insufficient, no matter what they do.

The second point is that these has been all about the HOT POTATO problem—everyone seems to toss the responsibility to another party.

Everyone has been HARDWIRED to EXPECT that the government must and shall deliver us from environmental disruptions and disasters.

Yet no matter the horrible track record, we maintain this illusion of infallibility.

People cannot seem to accept that government are composed by people, and like everyone else, has limitations in the possession of knowledge.

In reality, people should NOT depend on government for making these calls.

Everyone should take the responsibility to assess and act upon the tradeoff of allowing you and your family members to go to school or work during inclement weathers.

There are many risks attendant to bad weather e.g. leptospirosis and other diseases, potential accidents (falling trees, open manhole, electrocution among the many). This should be done individually and depending on the circumstances of the environment one operates on.

Government will never know the details of each and everyone of our lives.

Of course there are institutional solutions to these, such as not only the privatization of PAG-ASA but importantly to de-politicize them or subject them to market competition.

People or social institutions, such as schools, can reward or punish private institutions for providing accuracy in weather forecasting through the profit and loss system. Under the government monopoly, sustained mistakes should be expected.

Weather derivatives can also be used as an insurance against tail events or weather based calamities. Institutions can now make their calls based on appropriate assessment of cost and benefits.

There are many REAL things the markets can do that may alleviate the current predicament. But under the current popular mindset, such is a state of vacuum so the problem becomes reiterative or self-reinforcing

Thus the biggest among all the fundamental flaws is the public’s mysticism of government.

As Professor Don Boudreaux wrote,

Too many people, including otherwise very smart people, believe in secular magic. They believe that words written on paper by people, each of whom receive a majority of votes on certain days of the year of adult citizens living in certain geographic areas, and who utter ritualistic pronouncements under marble domes in buildings conventionally called “capitols,” are somehow endowed with greater understanding of society’s complexities and with superhuman capacities to care about the welfare of strangers. These priests preach devotion, dedication, and sacrifice to the One True State (your own government), even while each recognizes that legitimate disputes about the details of the dogma divide various cliques of the secular clergy. When they speak and act in their official roles, they expect – usually correctly – that the laity pay their words special heed as if these words have extraordinary power.

We will never improve on our approach to solve the weather predicament, until we come to realize of the existence of other alternatives…the markets.

Privatize Pag-Asa or Open Weather Forecasting To Competition

Weather forecasting personnel from the government institution are reportedly in mass exodus.

According to the Inquirer,

The Philippine Atmospheric, Geophysical and Astronomical Services Administration (PAGASA) has lost 24 key personnel, most of them experienced weather forecasters, in the past 10 years to lucrative offers from abroad, the Inquirer has learned.

Brain drain is not the only disturbance beclouding the state agency, on whose forecasts depend the lives of countless Filipinos. The problem of outdated equipment has battered it for years.

According to PAGASA personnel who talked on condition of anonymity, most of the weather forecasters have accepted offers from the state weather agency in Dubai, which is strengthening its forecasting system in its bid to attract investors and tourists.”

This is to be anticipated for the following reasons:

One, PAGASA is just one of the many tentacles of government agencies and thus becomes the object of concern only when political expediency calls for it.

Two, because PAGASA’s priority is based on political whims, thus, her financing is also subject to political priorities.

[I’d like to add that “brain drain” is a non-sequitur here, brain drain is the result of government or bureaucratic failure.]

Evidence from the same article,

After 1998, PAGASA decided to chuck the master plan.

But Nilo said the Arroyo administration was more supportive of PAGASA’s plan calling for much-needed equipment improvement.

In 2005, Nilo and PAGASA embarked on a new plan that included the upgrading of PAGASA’s existing Doppler radars.

Unfortunately, the Arroyo administration toward the end of its term slashed PAGASA’s budget for 2010.

The agency had submitted a P1.7-billion budget covering personnel and maintenance expenses and including capital outlay for the purchase of new equipment. But it was told by the Palace to stay within the ceiling of P614 million.

For 2009, PAGASA got a P757-million budget that included some amount for capital outlay.

Three, because government bureaus are likewise subject to public opinion, PAGASA serves as a favorite whipping boy or “passing the hot potato” (blame) for political leaders. In politics, which essentially is a zero sum game, someone has to take blame, hence if it is not greedy entrepreneurs it is the small fry (bureaucrats). Never will the blame fall on themselves or the bureaucracy or the legal system that supports it.

From the same article,

PAGASA has been under a microscope after failing to accurately track Typhoon “Basyang” (international codename: Conson) and its officials were publicly reprimanded by no less than President Benigno Aquino III.

The agency has upgraded the capability of two of its Doppler radars to improve storm tracking. Aside from that, the new radars can now provide information on wind speed, wind direction and rainfall amount.

The agency is set to upgrade five more radars in the coming months.

As shown above, government always are almost always reactive in approaching social problems, and that’s because the primary concern of politicos have been to generate favorable public opinion, since the essence of the preservation of their politically privileged status is in the substance of a popularity contest . Hence, since social issues are fungible or concerns which varies on a fleeting day to day affair, so goes with the political priorities.

Finally what people don’t see is that weather forecasting services could be better offered by the private sector.

In the US private companies are reportedly much better or more accurate in weather forecasting.

This from the Fox,

Private companies with a lot at stake would often rather pay for private forecasts than rely on the “free” forecasts from the government. Hugh Connett, the president of Bridgeline, a gas pipeline company in Louisiana, claims that the government’s hurricane forecasts are too imprecise. He says that private companies such as AccuWeather do it better, because they give more accurate predictions and provide hour-by-hour forecasts of a storm’s path.

His position is not ideological – Connett’s firm monitors the past accuracy of hurricane forecasters to make sure paying extra for the private service is worth it.

It is not just for hurricanes that private forecasting comes out on top. A new study by Forecast Watch, a company that keeps track of past forecasts, found that from Oct. 1, 2006, through June 30, 2007, the government’s National Weather Service did very poorly in predicting the probability of rain or snow. Comparing the National Weather Service to The Weather Channel, CustomWeather, and DTN Meteorlogix, Forecast Watch found that the government’s next-day forecast had a 21 percent greater error rate between predicted probability of precipitation and the rate that precipitation actually occurred.

In looking at predicting snow fall from December 2006 through February 2007, the National Weather Service’s average error was 24 percent greater.

“All private forecasting companies did much better than the National Weather Service,” the report concludes.

The government doesn’t do any better with forecasting temperature. For the largest 50 cities in the U.S. over the last year, ForecastAdvisor.com ranks the National Weather Service’s overall predictions for high and low temperatures as well as precipitation as dead last among the six weather forecasting services they examined.

It has only been in the last several years that comparisons between government and private weather companies have been possible, as the National Weather Service has made its data more readily available. But none of this should be very surprising. Incentives matter. If the private companies don’t do a good job, they go out of business. Government agencies never even shrink.

The key difference? Private sector is subject to profit or losses, thereby are incented to produce accurate or precise forecasting or risk losing capital, whereas the public sector’s performance goes only on the spotlight, when problems emerges.

Thus, from motivational issues, the lack of incentive to serve consumers, scant funding to shifting public priorities by political leaders, the mass personnel exodus from the government agency should be expected. The alternate solution isn't for government to spend more but to open weather forecasting to competing private enterprises.

Ludwig von Mises laid out the premise why governments are no better in providing "public services" needed by the people (bold emphasis mine).

In public administration there is no connection between revenue and expenditure. The public services are spending money only; the insignificant income derived from special sources (for example, the sale of printed matter by the Government Printing Office) is more or less accidental. The revenue derived from customs and taxes is not “produced” by the administrative apparatus. Its source is the law, not the activities of customs officers and tax collectors. It is not the merit of a collector of internal revenue that the residents of his district are richer and pay higher taxes than those of another district. The time and effort required for the administrative handling of an income tax return are not in proportion to the amount of the taxable income it concerns.

In public administration there is no market price for achievements. This makes it indispensable to operate public offices according to principles entirely different from those applied under the profit motive.