Underground

Online company reviews are high stakes. Top reviews on sites like Glassdoor and Google can get thousands of impressions each month and are major drivers of brand perception. Employers know this. And when I come across multiple 5 star reviews left with no cons, or a Pulitzer worthy essay from a former intern, I become suspicious. These reviews start to resemble 30 under 30 lists: so artificially constructed that you begin to question their credibility in the first place. The scrutiny around company reviews is well documented; some companies file lawsuits worth over a million dollars to reveal anonymous reviewers that complain about their jobs. Whilst it's the flashy lawsuits that make the headlines, there also exists an underground economy of company reviews operating quietly every single day. In this underground economy, some companies pay over $150 to freelancers to try and get a negative review removed. If they want “better” results, they go to the plethora of Online Reputation Management services (ORMs) in the United States that can charge retainers worth thousands of dollars. The supply of positive reviews exists too. My research led me to find companies, including a prominent Y-Combinator backed startup, that solicit fake positive reviews from online freelancers to improve their rating. Many of these mercenary fake reviewers, often based in South East Asia, make a full time living doing this, netting over $2,000 per month. Some of these run such sophisticated operations that they’ve even created their own pricing tiers (e.g $35 per original review, $20 to post an already created review from an email address), a la SaaS offering. Others operate on a contingency fee agreement model, where they only get paid if they’re able to take a negative review down. The underground economy of company reviews is well and truly alive. And today we’re going to find out how it operates. Note: For more content like this, subscribe to my newsletter. In a couple of weeks, I'll be releasing my guide to writing a killer resume. Adding reviews The barriers to entry for adding fake reviews are much lower than for getting reviews removed, so that’s where we’ll start. To write an employer review, all you really need is the ability to create an email address. For most sites, you don’t need any proof of employment (say like a company specific email address). I went on a gig marketplace site and posted a pretty vague post related to wanting to find out more on how to improve a company’s online presence. Within minutes of posting a gig, my inbox was flooded with proposals: https://preview.redd.it/tj6462grpe0b1.png?width=3064&format=png&auto=webp&s=b10eef64f3fdc9d1a23168a62637d7e366ed0d1b After a bit of chatting, I narrowed the scope of their services and summarized their rates into the table below: |Channel|Cost|Timeline|Model| |:-|:-|:-|:-| |Freelancer #1|$10 per review|Monthly|Unlimited| |Freelancer #2|$35 per original review, $20 per already created review|Monthly|Unlimited| |Freelancer #3|$25 per review|Monthly|Unlimited| |Freelancer #4|$25 per review|Monthly|10 reviews| |Freelancer #5|$20 per review|Monthly|Unlimited| |Online Reputation Management Agency|$300 subscription|Monthly|8 reviews| Let’s dive a bit deeper into the services that Freelancer #5 offered. Freelancer #5 explained to me he had been writing reviews for one particular company for the past 4 months now. Each month he wrote them 10 reviews. https://preview.redd.it/t2j8ihqfqe0b1.png?width=2684&format=png&auto=webp&s=e94a7cbb76598faaac0036f273b1c09a64c56121 In another message, he tells me he’s offering the same services to 5 other companies. Doing some quick math: 5 companies x 10 reviews per company x $25 per review = $1,250 per month Considering the average person in Pakistan earns $150 per month, that’s not bad change at all. One of the companies that he’s offering his services to includes a Y-Combinator backed startup. I won’t name the company, but here’s what its average Glassdoor review rating distribution looks like: https://preview.redd.it/c7bq8nqhqe0b1.png?width=2420&format=png&auto=webp&s=1f323f06b9305d8781cbb86b897bdc47249fe0cb 5 star reviews account for over 77% of the company’s total reviews. Obviously, no one is buying fake reviews that make them look bad. But here’s the thing: freelancers are getting quite smart when it comes to writing reviews that don’t look too fishy. They tend to do this by spacing the reviews out (so that they don’t come in “spikes” – more on this later) and they also make sure that they’re not always leaving the “cons” section blank. Don’t get me wrong, if you come across this company’s reviews, it’d be pretty easy to tell they’re quite strange. In fact, I can’t even post some screenshots here because it’d give the company away immediately. But it would be challenging to conclude that the above company is buying reviews just by analyzing review volume and distribution without actually reading some of the reviews. The same company is also buying reviews on Google Reviews. Sidenote: I got curious about how he’s been writing 50 reviews from 50 different emails per month. Would he actually create 50 different email addresses? And what about the IP address – doesn’t Glassdoor flag multiple reviews from the same IP? One of the freelancers answered my question: https://preview.redd.it/tu98uzkjqe0b1.png?width=2572&format=png&auto=webp&s=b249cc2e3f463c7864a27e70d95f95ff71859bf9 Moving on – another company that seems to buy fake reviews seems to be having some more trouble. Approximately a month after a freelancer linked me to fake reviews he had written for this company, all five reviews that he had linked me to had been removed: https://preview.redd.it/tf36r8clqe0b1.png?width=3116&format=png&auto=webp&s=6c62524c5219120a74ee6fb33f8a63c9b0e8b74f Based on this Glassdoor webinar from 2018, “if it is found that a user has created multiple email accounts to submit reviews, then ALL submissions from that user are deleted” – so likely Glassdoor’s content moderation team flagged one of the initial reviews and the same freelancer who was writing reviews for that company had all the fake reviews deleted. So far, it looks like the key to an effective fake review creation strategy lies in: * Spacing the fake reviews out * Writing each review from a different IP address (i.e benefit of being part of a team) * Using language that isn’t an obvious giveaway On that third point: the reality is that many of these freelancers’ first language is not English. As an experiment, I turned to everybody’s favorite new toy, ChatGPT, and asked it to write me a positive Glassdoor review: https://preview.redd.it/8on2901nqe0b1.png?width=3164&format=png&auto=webp&s=b34873d55f4d6725756feddb82774cefdf4c1e95 And I’d say that the above answer was better than 95% of the fake reviews I came across. Removing reviews The process for removing an employer review usually works like this: 1. You identify one or multiple reviews that you want removed 2. You verify whether the review violates the site's Guidelines, or whether there’s something else about the review(s) that could get it removed. 3. You file an appeal to get it removed. As an example, Glassdoor’s Review guidelines can be found here. Mainly, they forbid mentioning anyone by name who’s not an executive and revealing proprietary or confidential information, amongst a host of other things. Sounds simple enough right? Well, according to one of the freelancers I messaged: ​ https://preview.redd.it/56rvj8voqe0b1.png?width=2036&format=png&auto=webp&s=74a5b81f541a58758ff1b791f4fda9a6f6109b28 After some research, I summarized the different vendors and prices in the table below: ​ |Channel|Cost|Timeline|Model|Self reported success rate| |:-|:-|:-|:-|:-| |Freelancer #1|$100 per review|3 days|Contingency Agreement Model|100%| |Freelancer #2|$30 per review|7 days|Contingency Agreement Model|100%| |Reputation management service #2|$450 per review|21 business days|Contingency Agreement Model|Unknown| |Reputation management service #3|$1000 per review|Undefined|Contingency Agreement Model|100%| |Reputation management service #4 Plan 1|$550 per review|5-6 Weeks|Contingency Agreement Model|50-75%| |Reputation management service #4 Plan 2|$300 Subscription + $100 per each review removed|Monthly service|Subscription plan|50-75%| |Freelancer #3|$20 per review|Undefined|Pay regardless|Undefined| |Freelancer #4|$500 per review|Undefined|Contingency Agreement Model|Undefined| As you can see, unlike the fake review generation market, the prices vary quite a bit for getting reviews removed. At one end, you have freelancers on gig marketplaces that will attempt to remove a review for less than $100. And then on the other end, you have ORMs (Online Reputation Management Agencies) that have multiple employees and more comprehensive packages in place. The one constant seems to be that most companies operate on a contingency agreement model (i.e pay only if review gets removed). Analyzing reviews ReviewMeta is a site that analyzes Amazon reviews and tells you how many are legitimate. The creator of the site, Tommy Noonan, mentions in an interview with NPR that the main giveaway that a product is soliciting fake reviews is: * A large, suspicious flood of positive reviews at the exact same time. For example, a 3 day stretch of time constituting 30% of total reviews. * Phrases and words that are constantly repeated, especially in the section with no cons * Brand monogamists (only review products from one company) Whilst the last two bullets are hard to track, the first can be used to analyze different companies’ reviews and to check if there might be some funky business going on. After a couple of days, I have the ability to track review volume and review ratings over time for any company that I specify: https://preview.redd.it/eknwjoxese0b1.png?width=1653&format=png&auto=webp&s=6bcb0851ed0e9c46f003ba1a2db9adf4e7f2ecae Let the games begin. Voluntary Response Bias One of the biggest challenges that review platforms face is the Voluntary Response bias. Research shows many of today’s most popular online review platforms (e.g Amazon) have a distribution of opinion that is highly polarized, with many extreme positive and/or negative reviews, and few moderate opinions. Think about it: have you ever felt moderately satisfied at your job and thought to yourself, now would be a great time to leave a Glassdoor review? Probably not. On the other hand, if you’ve had a terrible experience or even just had one thing really flip you off, you might be quite likely to leave an angry review. Consider when a company goes through layoffs. You’re going to have a flood of angry reviews coming your way and are likely going to experience a “spike” in reviews. Note: Just like the Wall Street Journal’s methodology described here, I considered there to be a spike if the total number of reviews in the month was greater than three standard deviations above the mean of the surrounding months. Let’s take the company below. Here’s a graph of of their review volume since Jan 2020, including when they announced one of their first round of layoffs in June 2022: https://preview.redd.it/uqbfm3ugse0b1.png?width=3216&format=png&auto=webp&s=5aa04783c11d533775529f9b6cfb6726b141586f In June 2022, approximately 19% of this company's 52 reviews were 1 star reviews (compared to an overall average of around 10%). This is what we could call a statistically significant spike in reviews. It also illustrates how the employees most likely to leave reviews are the ones that obviously had a bad experience (i.e getting laid off). Here’s another company that had a similar spike in negative reviews due to layoffs in November 2022: https://preview.redd.it/6oo2ipfise0b1.png?width=2408&format=png&auto=webp&s=4f48aad5063648b39652c9d5243236f2bedb594f This company had an approximate 20% 1 star review rate (compared to an overall average of 12%) in November 2022, as well as an Avg Rating of 2.96 that month (compared to an overall average rating of 3.73).Unless HR is proactive, their reviews page risks succumbing to an echochamber of negative reviews that can really tilt one way. Note: Glassdoor does state (based on this video from 2017) that about 75% of the reviews on their platform are neutral. Their “give to get policy” has helped in keeping the platform from becoming too polarized. I can understand why HR teams, like the ones that Nader talked to me about earlier, take a proactive stance towards managing their reviews. If they don’t try to control their reputation themselves, then their reputation risks getting controlled by the employees that had the worst possible experience. Goodhart’s Law Goodhart’s law states the following: "When a measure becomes a target, it ceases to be a good measure" Every October, Glassdoor publishes their Best Places To Work ranking. In a report that the WSJ did a couple of years ago, they found large spikes in the number of reviews that some companies (e.g SpaceX, Bain & Co, etc) got in September. The logic here is that some companies try to artificially inflate their Glassdoor reviews right before the October deadline. I decided to revisit some of this analysis with Glassdoor’s 2023 Best Places To Work Ranking. One of the companies I examined is rated as one of the best places to work in 2023. Let’s refer to this company as FunPlaceToWork. Here is how their review volume looks like for all of 2022: https://preview.redd.it/a9bktnhlse0b1.png?width=2516&format=png&auto=webp&s=99af1d5a68c799e274a6daf3723ff2230cc69825 FunPlaceToWork got around 50 reviews in September 2022. Of those 50 reviews, 96% were 5 star reviews. FunPlaceToWork averaged 12 reviews per month up till then in 2022. Also, in the prior six months, the average percent of 5 star reviews received every month was \~75%. Both the spike in volume of reviews and the spike in percentage of five star reviews are statistically significant. I find it strange that Glassdoor’s proprietary algorithm and/or Human Content Moderation team did not find a spike of this nature unusual. If we look at Glassdoor’s eligibility criteria for the award, it’s as follows: https://preview.redd.it/4j779c0nse0b1.png?width=2868&format=png&auto=webp&s=6686f282508375d041befef298a0594f9460eeb8 The goal, according to Glassdoor, is to collect “authentic and unbiased reviews”. Whilst there’s nothing against the rules for asking your employees to leave you reviews, I find the statistically significant spike of reviews at odds with the goal of collecting "unbiased and authentic" reviews (which Glassdoor states is the purpose of the awards). Glassdoor states that an employer is allowed to ask its employees to leave reviews, but that they are not allowed to “coerce” them. Examples of what you can’t do: * Offer incentives like Gift Cards in exchange for positive reviews. * Withholding their reference letter unless they leave you a positive review. * Anything that leads you to require proof for the employee to show you that they wrote a review. It is possible to play by the rules (i.e not break any of the above rules) and to still in my opinion not collect authentic and unbiased reviews. They say that you shouldn’t hate the player but the game – I think FunPlaceToWork played by the rules, won fair and square, and that this is simply a perfect example of Goodhart’s Law. I reached out to Glassdoor ([email protected]) about the above and this is the reply I got: https://preview.redd.it/5019eexose0b1.png?width=4800&format=png&auto=webp&s=0c010a668da1169ad1bb2383fe7f713666171946 Conclusion When I was 22, on an F1 visa with 3 months to find work, I didn’t give a damn about bad reviews. I needed a job and I’d sign any piece of paper you put in front of me. Compare that to someone at the peak of their career, someone with optionality and a multitude of job offers; an “A-Player”, as the experts call it, would absolutely have the luxury of choice and discard a job offer based on bad company reviews. For most people, the impact of online company reviews lies somewhere in the middle. In marketing, there’s a concept of a “marketing touchpoint” - an interaction with the brand over the course of the whole buying journey. Company reviews are one of the many touchpoints a job seeker experiences over their interview process. And with the technology industry booming the past couple of years, companies couldn’t afford to slack on any touchpoints, including this one. After all, when others start to game the system, you’re at a disadvantage if you don’t. The rewards can be quite high. Certainly higher than just trying to be as transparent as possible. HR leaders are often more incentivized to inflate their metrics than to get honest feedback. Fake review writers have bills to pay. ORMs know that companies are desperate. And the platforms, well, aren’t always paying attention. The result is a potluck of interests that leads to an underground economy. One that ends up hurting the job seeker. *** Whew. That took a while (about 3 months in fact). Thanks for reading. For more content like this, subscribe to my newsletter. It's my best content delivered to your inbox once every 2 weeks.