15 years ago when Microsoft, IBM, and the like ruled the earth, product management was mostly a “technical” function. Technical not in the sense of function (ie coding), but in the sense of its scope. Today, Product Managers have become the heart and soul of many technology companies . . . extending its tentacles through out the entire organization, owning P/L’s, running strategy session, and generally holding court for the rest of the company. Ask any person at a technology company, if you have an idea or concept, who do you grab first? Your favorite PM, ideally with an offer of coffee or lunch!

Things have changed rather quickly. And it all started with Hotmail and eventually culminated with social networks (friendster -> myspace -> facebook).

So what happened? Back in the day (the days of pascal, assembly language, and C w/o the “+” ) software delivery and marketing was distinct disciplines. You talked to the customer, you asked them what they want, you build what they want, and you throw it over the wall to the marketing and the sales guy. Brochures and presentation are produced (sometimes even a prototype), and off goes the sales person knocking down doors trying to make a sale. For the shrink wrap guys (MSFT), it was a little different but still the same old. The marketing guy figure out the print and TV ads, while a partner/sales guy try to get your box on the shelves of ComputerLand and bundled with the latest PS/2 (no, not playstation 2).

The Internet came along and changed EVERYTHING. Instantaneous distribution, usage, installation, and ACQUISITION. Word of Mouth changed to the Spread of the HyperLink. The wall between acquisition (which used to be a pure marketing/sales) function and product definition / management crashed down. . . But no one quite noticed (too busy searching for porn? ) until Hotmail came along and BUILT ACQUISITION DIRECTLY into the product. Hotmail didnt need a sales guy, it didnt need ComputerLand, it didnt anything but a group of developers and a very smart Product Manager (who understood marketing, sales, and acquisitions).

Friendster came along and added fuel to the fire. It added Vanity & Peer Pressure into the “product as an acquisition channel” equation. Potential Hotmail users simply noticed the link and signed up. Potential Friendster users were FORCED to signed up by their friends. Back in the days of Jobs & Woz, they wold call it the transition from simple word of mouth to evangelism. Your existing customers were REQUIRED to advocate your company/solution as part of getting the most out of their own investment in the product.

The geeks who knew how to productize the acquisition channel, skipped the rest of the company and reached directly into the minds of the customer . . . and became superstars for helping to build internet empires.

Today when I look at a new company or product idea, this is the first thing I looked for. Whether the company has productize (or the opportunity to productize) the customer acquisition channel. Whether the founders (or PM’s) understand the customer conversion funnel (from awareness to close), and built FUNCTIONALITIES to address each of the steps in order to drive potential customers towards usage, conversion, and retention. Its not always as simple as Hotmail or a social network; not all business models, fit so nicely into this framework. But all along the way, there are many many opportunities to use product features to shepherd potential customers towards an desired conversion event.

The shrink wrap machines are sitting pretty idle these days . . .

Graham R. Joyce over at Pragmatic Marketing (a product mangement consultancy) shot over to me a link to their latest e-book The Secrets of Market-Driven Leaders a few weeks back. Didn’t have a chance to look at it until this weekend (sorry Graham!). Take a quick scan, it has some funny anectdotes as well important lessons for product managers.

Here is the main nugget of the pdf (if you are too lazy to read).

I rarely embed pictures into my posts eventhough I should (more readers). . .but usually I’m too lazy and posting is an impulsive activity I do randomly throughout the day. But this time, I thought it was important enough to get into my flickr account and make sure this graph is in the post.

Now, ofcourse its important to be at the upper right hand side, but even more important is to realize that the green arrow ALSO represent a time axis for a startup. IE. . . startups are often first founded on (1) a technical idea or breakthrough. The MBA then comes on the team and helps put together a business plan by (2) analysis of competitors and how to position in the market in respond to those pressues.

Now here is the hard part . . . once a company is launched (already hard enough to get to the second step) , a sucessful company will move toward either of the 3 quandrants left. Most start ups, if they are lucky enough, become sales or customer driven. Being customer/sales driven (ie focusing on your CURRENT/EXISTING/ALMOST customers) is not a bad thing to do. I can only wish more companies are like that.

But, sometime in the near future for a startup to scale (or another word, to cross the chasm) it needs to focus on solving the need of its entire target market. The sales and customer driven quandrants offer Calypso-like bounties but entraps the startup in a state where it is narrowly focused on a niche and thus caught in a detour from its true mission.

Perhaps a little counter intuitive, the startup needs to be a little more selfish & strategic in its priorities. It needs to understand that Market Driven vs. Customer/Sales Driven is the difference between $50M and $500 in revenue. Of course, its hard to do, thats why most startups fail and even less become an industry changing company.

There is actually a really good article over at news.com over the usability testing travails of hotmail 2.0. . I can’t tell you how many times I’ve been in exactly the same situation . . . trying balance innovation, strategy, and immediate user feedback.

The program was too slow to load, too different and, well, just not like the old Hotmail it was intended to replace.

It was a painful realization for the more than 100 managers and developers on the project. In banking on a snazzy Web 2.0 application to try to catch up to rivals Yahoo and Google, Microsoft had dramatically overshot its audience.

Classic mode wasn’t the only bitter pill the development team had to swallow. Even in the full version, it turned out that many customers still wanted to select messages using check boxes rather than a mouse click or keyboard shortcut, much to the dismay of Microsoft’s programmers.

“They were digging in their heels,” Sim said.

In the end, users hates any types of change (good or bad) . . . they will eventually adopt changes that are clear improvements, but that will still take a while. . . there is no such thing as a magic switch in the first place.

People that believe being “customer focused” is the ONLY goal of any product manager doesnt really understand innovation and strategy (ie innovator’s dilemma). On the other hand, product managers that are married to pushing the envelope on innovation and un-reasonably attached to their frameworks (information architecture, product strategy etc) really belong on Sand Hill road instead. There is a middle ground but its a case by case basis . . . knowing when to turn back (like the hotmail team) is a good first step. For all of MSFT’s faults, I’m actually quite impressed by this story.

Another random note, M&A teams of major companies are usually completely clueless on stuff like this. They believe that 1+1 = 3 while in reality 1+1 = 1.5. For example, taking a popular rumor over the weekend, if Microsoft and Yahoo merged what would be the combined userbase of their instant messaging product in 12 month if they tried to merge the two clients into one product? It would for sure be less than the combined marketshare of the two clients currently. The disastrous Sprint + Nextel merger was another perfect case study . . .

Josh K. has an interesting post that contrasts the acquisition & retention model of social networks (and community sites) in general.

The winners have a “catch and keep” model where the site is “sticky” while the less succesful players have a “catch and release” model where repeat visits are low and value is garnered linearly.

Another way to put it . . . we should try to build a site with a set of functionalities which gives users incrementally more value on the second visit than the first visit . . . third visit more than the second vist . . . and so forth. In many ways, this is a practical intent of metcalfs law and perhaps much more actionable than simplying trying to build a large userbase and claiming to have reach critical mass and network effects. The goal for any website would be creating accretive value per incremental visit.

Another interesting interpretation of Josh’s post is that social networks that are augmenting/turbo charging offline & parallelsocial interactions seems to have higher value and relevancy to the user than ones which tries to be completely virtual. If a site has relevancy to the daily (and thus offline) lifes of its user (campus life for facebook users, professional & personal networking for meetup users etc) it will remain more sticky than a site which is simply trying to act as a bridge or replacement for physical interactions. (Yes, I believe the future lies in “singularity”)

Lastly, to the snarky title of this post. It wasnt long ago (12 month?) that the web is buzzing about “attention aggregators” which mostly became just “catch and release” websites. It is very possible to create a “edge” based site which has the “catch and keep” value proposition (google for one) . . . but in truth, the functionality and features required to provide accretive value per incremental visit can rarely be achieved by an aggregator unless aggregation can provide such a value (which only occurs very very rarely . .. such as in google’s case). In most cases, aggregation is not enough and product/feature based value can only be built and captured through a “walled garden” like website which nurtures and builds value within a tighly controlled environment. A potentially winning model (I have to think more about this) is a hybird site like digg which combines edge aggregation with wall-garden like product and features . . .

I was partaking in the “two screen” experience yesterday - surfing and watching TV at the same time (30% of young adults between 16-28 does this regularly according to a presentation I attended by AOL/TIME Warner VP of Market Research). On my TV was E! True Hollywood Story (”THS”) on the making of Mean Girls (you know, Lindsay Lohan & Rachel McAdams!). On my laptop I was reading Jeremy Liew’s Game Mechanics which led me to spend the rest of the show googling Amy Jo Kim. It was about 15 minutes into this experience that I realized that what Amy Jo Kim was talking about is not so different than re-creating high school (or the first year of MBA) online (ie Mean Girls). I had called it Vanity Marketing about a year ago. She merely called it “social design” or “game mechanics” for online communities.

Here is a good summary of what Amy teaches. The women has been at it since 1996! . . . a whole 10 years before all this became so hot . . . a pure genius.

Thinking back ten years ago. . . software product design was all about value proposition, ROI, defining pain points, identifying catalysts for change, finding stakeholders, and many many more boring adult stuff. . . all really coming back to money. As such when websites (esp web apps) were sprouting up every where in 1998, product managers/designers/entrepreneurs simply followed the paradigm borrowed from the software development world where interactions where focused between an application and its user (rather than user to user). Ease of use, “information architecture,” features, funtionality, workflow, integration . .. etc etc . . .

eBay changed all that. (I’m not kidding, Amy Jo Kim really believe eBay is one of the best designed communities out there . . and obviously the first to ever scale). Today, the emotional aspect of product strategy is not something that can be ignored any longer. Drawing a parallel, behavioral finance has fundamentally changed the way economists and wall-street view the markets for financial instruments . . so will this discipline.

Building products catering to the psychology of user is a completely orthogonal discipline than what is traditionally considered product management. Today, its just as important . . . especially in this web 2.0 world. (eck I said it again)

Instead of regurgitating her research (which you should read yourself) this is my take.

Online, we are all 16 year old teenage girls. Put your self in the shoes of your users. This is your first day of high school. You are new, you dont know any one. How do you know who are the cool kids and who are the crowd to stay away from? (feedback or rating system, leveling) How do you become popular? (participation, collecting, flaunting - good ole online materialism) . How do you fit in while standing out? (completing a set, virtual scarcity, customization) . How do you achieve your status and let everyone know about it (leader board, stars, feedbacks again). How do you curry the favor of the in-crowd? (exchanges, gifting). How do you find a sense belonging? (groups, social networks) How do you fight back? (foums, comments). How do you know what to wear (customization, widgets). What activities and clubs to joing? (games, puzzles, treasure hunts, . . . and MMOG!)

How do you become the Queen Bee? The one all the girls want to be and all the guys want to be with? . . . give your user a way to become the Queen Bee and flaunt it!

Exploiting Leveraging insecurity and vanity of the user is key to building a vibrant and succesful online community. The line between e-commerce and community has already been blurred. The line between web app and community will go next. Pretty soon community and MMOG will blur itself too . . . and websites will become one big goo of cat-fighting teenage girls

This is a familiar story. . . you work at a consumer internet or software company and you are trying to figure out the important features for a product you are launching. You go out and survey a large number of your target customer base and comes back with a top 10 list. Given the scarce development resources, you decided to build the product with the top 5 most important features.

you built it . . . and no one stuck around . . . despite a mentions in all the cool blogs and tens of thousands of dollars in google adword spending . . .

So what went wrong?

Its because competition ALWAYS occur at the margins.

In many industries, espcially online given the low barriers to entry, hyper competition has commoditized almost any “idea” that one can come up with. (how many social network for high school sports can there be?). There are so much competition, only the good one sticks around in the first place. EVERYONE will have the 5 features your customers said they wanted. As such all products and websites begin to look the same to the end user . . . until you dig a little bit deeper . . .

The littles things are what matters. At the margin is where the winner and loser are determined.
The marginal features will determine the success and failure of a product: a simple registration process, a neat little feature, the ability to customize workflow . . . whatever it might be . . . It is the things that are ranked 20+ in importance (and sometimes not even mentioned at all) that will drive true differentiation. But there are so much noise and variance once you get down to the end of that list that it is almost impossible to know whether the data is significant or driven by random chance.

And this is why it is such a crap shoot to build a winning product. EVERYONE will tell you that good breaks are needed for a car . . . but close to NO ONE will tell you the 20 little things you need to outsell BMW.

A product manager’s job is becoming less of a science but an art. Experience matters, gut matters . . . . a holistic approach to understanding the customer experience matters . . . Knowing how to crunch numbers is a BASIC requirement to become a product manager . . . it takes so much more to be “good” than it used too. Ironically, we too are competing at the margins.

I dont write about payments much since lots of my work and research are somewhat confidential (for my job) and; more importantly, I’m really a student of the industry than a guru (not that it ever stopped me from BS’ing before) . . . However as I read and talked more to people in the industry it was clear to me that the future of mobile payments in the US is in somewhat of a holding pattern.

There are several architectural issues that needs to be resolved before wide spread adoption by consumers can take place. Even more importantly, without the emergence of a dominant architecture, the current consumer experience for mobile payments is not a marked improvement over what is the current “NON-MOBILE” payment experience . . . without a magnitude imrpovement in convenience or a catalyst (aka killer app), mobile payment adoption will be hard to realize.

What many entrepreneurs and VC’s fail to realize is that EXISTING payment methods are already “mobile.” What Mobile Payments really should stand for is MOBILE PHONE PAYMENTS as well as what happens to digital payments when ubiquitous real-time connectivity becomes a reality. IP connection and the browser used to tether computer users to the desktop, but not anymore with advent of IP/browser/applications on mobile phone. This is a quantum leap improvement in accessibility and context. However, cold hard cash as well as the good old Visa/MasterCard never tethered the end user to anything in the first place. The wallet is as mobile as it get! . . . Mobile payment’s value proposition will never be mobility, but the increase value through better acceptance, authentication, and control.

There are really two major mobile payment architectures in competition for the future. (Hybrids are possible but harder analyze.)

Scenario One

NFC technology matures (its very close). The payment card (Visa/Master) merges into the cell phone at the physical layer - ie your payment card sim chip is in the mobile phone.

1. Customers buys something
2. Customer “waves” his phone at a terminal
3. terminal asks customer to type in PIN at terminal (or not)
4. terminal uses its existing connection to association rails to authenticate and authorize payment
5. payment complete/fail

This is almost embarassingly simple. Besides step 2, there is almost no difference between this use case and the existing use case for payment cards. And there lies why this architecture will gain the most traction DESPITE the fact that it got a late start. The incremental value proposition is simply speed . . . and the incremental disruption to existing pratices & infrastructure are almost none. And thus the barrier to adoption are the lowest - new terminals and new cards needs to be distributed but the associations have allocated huge budgets to make this happen. Under this architecture, the opportunity for startups to disrupt incumbents are low. The banks, associations, and merchant networks will retain most of the value created.

The only evolution I can see is that Carriers gets in the game and create applications and sofwares that integrate the sim chip at the system and application layer, not just the physical layer. Thus, you can check your balance/credit limit in real time, keep a history of your transactions, etc.

Scenario Two

Almost all independent mobile payment companies are taking this route (Paypal, OboPay, MobileLime etc). Instead of relying on merchants, their terminals, and thus the existing VISA/MC association rails for authentication and payment, they route the transaction through the mobile phone, OTA IP connection, and the Carrier back onto their own merchant processing network (circumventing the association).

Here is the customer experience

1. Customer buys something
2. Phone recieves merchant ID via a few possible methods
- text message using merchant phone # as ID
- NFC gathers merchant ID from a passive terminal (no connectivity whatsoever)
- customer enters ID via an application interface
3. Phone requests/customer PIN
- dial back
- text message
- application prompt
4. The mobile uses its IP connection to authenticate and process the transaction

There are a huge amount of infrastructure that needs to built here. BUT even more important, the change in customer behavior is HUGE. . . . The odds of this architecture winning in the short term are low . . .

BUT I can see why this is a much more exciting vision of the future and a broader set of value can be created. For one, payment connectivity is no longer enaslaved to the merchant. Anyone with a mobile phone can accept payments. ACCEPTANCE (not payment) mobililty for digital money moves from “store to store” to “people to people” (Its like payment cards finally became like cash , ie it is fungible at the people level) Furthermore, the integration of the authentication through the phone opens up creative applications for controling spending, savings, and any financial activities.

EVEN MORE IMPORTANTLY from the business model perspective, this architecture gives startups an opportunity to disrupt the incumbents and create/launch their own merchant network where they can make 50-200bps on the transaction volume. Who wouldnt want to be the Visa/MC/Amex of the next millennium?

Eventhough the 2nd architecture has gain a lot of funding and publicity, I believe the first architecture will be the first to gain adoption. It is certainly not ground breaking, but its the neccessary next step. A version 2nd architecture (which enables breakthrough accpetance and ubiquity) will come but only with the blessing of the existing payment associations and not in its current form as implemented by the existing slew of startups. Paypal is the only player with an outside shot of pulling this off because of their existing merchant relationships and consumer adoption. All others needs to revaluate their architecture and give up the hope to create a merchant network. (atleast not in the first 5 years). Instead, they need to learn to play better with the payment associations, embrace NFC, focus on building mobile applications, and earn the trust of carriers. They need to learn to leverage the hundreds of millions Visa/MC/Amex have earmarked to upgrade its terminal, network, and cards instead of fighting an uphill battle.

In that model, a modified scenerio 2. Mobile phone will become payment as well as a acceptance tool . . . it will be able to act as a terminal as well as a authorization ID. (through NFC/RFID/Infared/bluetooth or whatever communication device method is out there between two phones). Visa/MC/Amex etal will not go away, but a startup might gain enough traction (Paypal) to join them. Their rails/processing systems will migrate to the handset with the help of Carriers who controls the physical layer (mobile phone hardware + airwaves). Applications will be written for the mobile phone to transform “mobile phone + payments cards” beyond simple identification tools (what is a credit card if you take out the plastic?) into innovative method for controlling spend, evaluating credit, managing savings, and whatever entrepreneurs can think of next. . .

(My initial title for this post was “Yahoo Panama Working?” I just changed it to “YHOO is a SCREAMING BUY!” after I did some quick number crunching.)

Ran across this press release from Comscore which claims Yahoo! Sponsored Search Ads
Click-Through Rate increase 5% and to 9% in the last few weeks.

Using the week ending February 4, 2007 as a baseline for sponsored search click-through rates (i.e. total clicks on sponsored search ads divided by total searches) before the ranking model launched, comScore studied the two subsequent weeks of click-through data to evaluate the impact of the new ranking model. comScore’s data indicate that for each of the two weeks subsequent to the launch (ending February 11, 2007 and February 18, 2007), Yahoo! Sites experienced a noticeable lift in its sponsored search click-through rate. The week .ending February 11 saw a 5-percent increase, while the week ending February 18 showed a 9-percent jump.

Furthermore,

Another anticipated result of Yahoo!’s new ranking model is a shift in composition of total click volume from algorithmic to sponsored. The “sponsored click composition” metric (i.e. sponsored clicks as a percentage of total clicks) is critical in understanding Yahoo!’s success in improving both monetization and user experience. qSearch data show positive gains in this area, with sponsored clicks representing 10.6 percent and 11.1 percent of total click volume in the weeks ending February 11 and February 18, respectively. These data represent increases of 0.5 and 1.0 points in the weeks following the new ranking model launch.

So the good news first. . . it appears that CRT for sponsored search results almost doubled WITHOUT take significant share away from algorithmic/natural search results.

Let me do some simple math to confirm. . .

PANAMA Week I

.106 = sponsored clicks / click
+.05 = sponsored clicks / search query

+.05/.106 = Incremental 0.49 total clicks per query

PANAMA Week II

.111 = sponsored clicks / click
+.04 = sponsored clicks / search query (.09-.05=.4)

+.0.4/.111 -> Incremental .36 total clicks per query

So in two weeks, the clicks per query has increase by a total .85 clicks.

To give some background, total clicks per query can approximate to quality of the results (not globally especially when the # is large it might actually point to having bad results, but its safe to assume that we are at a local optimal since Yahoo does have a decent search engine). Futhermore, the rule of thumb is that in general search engines generate 2 clicks per query. To increase clicks per query by close to 50% is freaking amazing.

Even more importantly, I was afraid that sponsored ads was taking click throughs AWAY from organic results which would mean that the effect is short lived and that the numbers would point to LONG TERM dissatisfaction with Yahoo’s organic search algorithm.

BUT to my amazement, the total # of clicks per query has increased showing that overall clicks per query has increased significantly thus organic search clickthrough has maintained (or even improved) because of the change in sponsored search algorithm (aka Panama).

These are all great news for Yahoo. I’m gonna put some money in YHOO now. . .

(CAVEAT, I’ve NEVER made ANY money in stock investments so dont listen to my ramblings)

Online forums around brands (aka fan sites) - everything from Tyra Banks to Bratz are big business. Anyone that has searched for information on google or yahoo will realize that forums represents a disproportionate portion of the search results due to the large amount of dynamic & fresh content generated. All if not most are moneitzed through YPN or Adsense - Google ’s Adsense is generating around $4B a year for Google and probably $8-$12B in total for millions of webmasters. To realize how big this “digital underground” is all we need to do is to take a stroll on sitepoint or webmasterworld. (Monetizing the longtail of websites if you well!)

Foremanski has some good comments from the business perspective.

There is, however, something more going on here from search marketing perspective. . .

1. Yahoo can give preferential treatment to these sites over independent fan sites and forums. Try a Maria Carey search on Yahoo and you will notice the Yahoo Artist page as the top result. This will happen to a lot of top search terms going forward.

2. Even Google will give these sites higher rank compared to independent sites because they are hosted on a yahoo domain which has high PageRank (wii.yahoo.com for example)

3. Having the search term in the subdomain (wii.yahoo.com) could potentially increase search ranking as well (somewhat debatable)

4. Conversely, this could be considered a form of cyber squatting (a subset of the copyright issue widely mentioned) given the brand name in the subdomain

5. Yahoo will for sure expand this down its list of search terms and eventually extract more and more value and advertising dollars from webmasters who runs these fan sites and forums.

Time will tell but this could be a turning point for the large community of niche sites having their way until recently on drawing internet traffic away from larger commercial website. . . Yahoo is fighting back and so are other media companies (note the recent spat of “social networking” initiatives from media companies)

Yesterday, Amazon launched endless.com (coverage at techcrunch and techbeat). Obstensibly its an obvious response to the $1B that Zappos.com is taking in a year (close to 35% of the total shoe market) . But even more strategically, its an increasing sign that the first generation internet giants has reached some sort maturity and limits to scale that traditional marketing techniques like brand extensions (ie, Coke -> Diet Coke) are beginning to become popular if not standard strategic growth options.

Furthermore, as much as the web 2.0 crowd (we? me?) would like to believe, mashups are not invented by the internet generation. Using internal assets and re-aligning them to create new products our services was invented when Henry Ford (finally) launched Model A.. And continutes today, the Chrysler Crossfire uses the same chasis as the Mercedes SLK, another example. (What is interesting is that mashups has moved form assembling internal assets to mixing external and internal assets through loose couplings and arms length relationships). For endless and ebayexpress, the main assets that are being leveraged are the inventory (both share a subset of inventory from the mother site).

I blogged about this a few month back and it appears to me that Amazon is hitting the same scalability issues.

For giants like eBay, Yahoo, Google, and even Facebook and Myspace (today). . . the MARKETING FACT OF LIFE - SEGMENTATION has slowly reduced the value of network effects. By definition, network effects is a mass market play. It is in opposition to the concept of niche marketing, niche product, for niche segments - ie the better you can target your product or service to a particular niche the more likely he or her will chose your product over a competing generic solution. These giants cant no longer band-aide new site wide features and functionalities hoping to attract new segments to their website as USAGE TIME, SCREEN REAL-ESTATE, USABILITY limits the feature creep. This admission that network effects is no longer the dominant driver of their business - that segmentation is - is widely seen but rarely discussed . .

It further amazes me how the mentalities of a startup can blind someone from making the right business decisions (better, faster, cheaper /= better for the customer). For some reason, the “why’s” of endless and ebayexpress were not obvious to the pundits of the tech world. . . but for someone that works at Gap, P&G, or a HomeDepot it must seem very obvious.

Every brand has its limits, even Amazon and ebay. Ofcourse, Amazon decided to completely remove itself from the endless brand while ebay tried an extension to target new segments of users.

One size fits all is only true for baseball caps (and even that is questionable). Beyond the brand, the flexibility to create custom search/discovery experience within a category can create a significantly better conversion rate AS WELL AS re-enforce the brand itself. In the physical world, the Apple Stores and its shopping experience are a clear extention of the Apple brand.

“Enhancements” are not always better. Ajax might be cool, you might love all the drag and drop functionality of the latest website profiled on TechCrunch. But you and I are a sample size of one. For incumbent companies, by definition through the size of its userbase, late adoptors are the largest segment. In the end, the feature needs to be rigorously tested with a large enough sample size to really know if “enhancements” are good for the bottom line. . . not just kinda cool.