How to Become a Search Engine Evaluator [Make Up to $20/hr]

Did you know that you can work indirectly for Google/Microsoft and make thousands of dollars per month? Did you know that Google/Microsoft hires thousands of independent contractors from all over the world to help refine their search results?

What if I told you that I am one of them and I make thousands per month working for them? And you, my friend can be one of us as well, working indirectly for Google/Microsoft, and making thousands per month. Read on.

First things first, the TERMINOLOGY. 

These unsung heroes of Google/Microsoft  are known by different names. 

  • Search Engine Evaluator
  • Search Quality Raters
  • Web Content Evaluators
  • Web Search Evaluators
  • Internet Analysts
  • Internet Assesors
  • Personalized Internet Assesors
  • Internet Safety Evaluators
  • Google Raters
  • Raters

Google likes to simply call them raters and for the sake of brevity, let’s just call them raters in this article. 

Now the obvious question that comes to mind is “do these so-called raters even exists?” Do Google/Microsoft even hires independent contractors? Is it some sort of a conspiracy theory?

Let me remove that doubt off your mind. Watch this video and listen carefully at 1:00

If you listened carefully at 1:00 of the above video, you will here Google CEO Sundar Pichai mention the word “rater” and how they use them to evaluate results.

So that should eliminate your “what if raters don’t exists?” thoughts.

Ignore what they talked about, just focus on the “rater” part of it. 

What is a Search Engine Evaluator?

Search Engine Evaluators aka raters evaluates search results – mostly webpages, videos, images, maps, etc and see how helpful/useful they are to the queries issued by the user and rate them as per a set of given parameters as outlined in the General Guidelines.

General Guidelines is the Bible for raters, it is the gospel with which they got to live by. It is a 176 page pdf file that each rater will have to peruse, use it to pass the qualification test and then reference it daily to evaluate the tasks after hitting production.

Illustration of what they do: Supposing you type “weather in Tokyo” in your smartphone. The result would be like this:

Now in terms of search engine evaluation – 

  • “You” the person issuing the query would be known as – USER
  • The search term “weather in Tokyo” would be called as  – QUERY
  • The result enclosed in the blue rectangle would be called as – RESULT BLOCK

Based on the General Guidelines, we would be rating that result on its usefulness to the user query.

Initial Requirements

To be a search engine evaluator aka rater, here are the initial checklists

  • You must be fluent in written and verbal English ( or any language in which this position is available in your country. 
  • You must be living in your country for the last 5 consecutive years.
  • You must own and use a smartphone.
  • The majority of the task types will require both a desktop/laptop and a smartphone.
  • Gmail must be your primary email account.
  • Active daily user of Gmail and other forms of social media.
  • You must have familiarity with current and historical business, media, sport, news, social media and cultural affairs in India.
  • Access to and us of a broadband internet connection and associated computer and software to perform the work, all provided at your own expense.
  • Experience in use of web browsers to navigate and interact with a variety of online content.
  • Payoneer account – a must to receive payments.

Best Search Evaluator Jobs:  

  1. Appen
  2. Lionbridge
  3. Raterlabs
  4. iSoftstone

Here is a brief account of the companies that hires raters.

1. Appen Connect

appen connect sign up page

Appen Connect is a company based in Pleasanton, California. I’ve been with them since June 2016.

It was previously known as Leapforce before its acquisition by Appen in late 2017. It has since been re-branded as Appen Connect

Apart from evaluator positions they hire for multiple projects like – social media evaluation, surveys, smart speakers project, transcription and so on. 

But their main and the oldest project is the evaluator/rater project. 

Please don’t confuse Appen Connect with Appen – which is the parent company of Appen Connect formerly called as Leapforce. If you want to work as an evaluator/rater, you should be signing up with Appen Connect and not with Appen. 

Here is where you would sign up – Appen Connect Sign Up

2. Raterlabs

Raterlabs was a sister company of Leapforce before its acquisition by Appen. It was specifically created by Leapforce for Americans.

With the acquisition of Leapforce, Raterlabs too is now part of the greater Appen family.

Please note that this company offers jobs only for Americans. They exclusively hires Americans as part-time employees. You can sign up HERE. 

3. Lionbridge

This is a MNC based in Waltham, Massachusetts that has 6000+ employees and hires thousands of independent contractors from all over the world. 

They hire for various positions including the evaluator positions. You can sign up HERE.

4. iSoftstone (teemwork.ai)

This is another well -known company based in China. Their US offices are located in New York, Waltham, MA and also in Kirkland, WA. 

They have over 6000+ employees and hires thousands of independent contractors from around the world.

Please note that they have been re-branded themselves as Teemwork.AI as is visible from the sign up page above.

Their primary client is Microsoft and they hire for web search evaluation positions and online ads evaluation position as well.

You can sign up HERE.


Search Engine Evaluator Jobs – FAQs

Here are a few frequently asked questions, that you might want to ask.

1. Is this position available only in English?

Absolutely not ! 

Google operates in multiple languages around the world and hence this evaluator/rater position is also available in multiple languages.

If you are in Spain, this position would be opened in Spanish, if you are in Germany, this  position would be in German, and so on. 

Raters are hired from almost all countries of Europe, North America, South America, most of Asia including India and from few select countries in Africa.

2. Can I work from anywhere in the world?

Yes and no.

As long as you are within your country, you can work from anywhere within your country.

However, you can’t work from another country. Although you could use VPN to work from outside of your country, it is not recommended. It is against their policy.

3. How do I get paid?

You get paid via Payoneer, bank transfer. 

Payoneer is the preferred method of payment for Appen and Raterlabs. Sign up for Payoneer HERE and get $25 as soon as you are paid at least $1000 to your Payoneer account.

4. When do I get paid?

You get paid under NET-30 terms or NET- 45 terms.

Note:Net 30” is a credit term used in business to signify that the full amount a client owes is payable within 30 days, including weekends and holidays, upon goods shipment or job completion. The same goes for “Net 45“, where the payment are made within 45 days.

Appen, Raterlabs & iSoftstone pays under NET-30 terms while Lionbridge pays under NET – 45 terms. 

Appen and Raterlabs normally pays you by 14th of each month – meaning for the work that you have done on January (say), you will be paid by 14th Feb, although they can technically take the entire month to pay you.

Lionbridge payment is a bit different. For the work that you’ve done in January (say), you will be paid by March 15/16/17th. They take almost full 45 days to pay you after your invoice has been approved. 

5. Will I be deemed as an employee?

Except for Raterlabs, all companies will hire you as independent contractors.

Raterlabs hires Americans as part-time employees with all the benefits. Please note that Raterlabs is only open for Americans and not for anyone else.

6. Is this a long-term project?

Yes it is ! As long as Google lasts, there would always be a need for human raters to feed raw, human data to Google’s algorithm so that they can refine their search algorithm.

I know people who have been working as an evaluator for like 8+ years. 

I myself will hit 4 years of rating this June (2020). Been at it since June 2016. I documented everything in my blog.

7. How many hours can I work per week?

It depends on the company you choose to go with and your evaluator level.

 Appen Connect

If you are at Level 1, you can work for a maximum of 2 hours per day.

At Level 2, you can work for a maximum of 4 hours per day.

At Level 3, you can work unlimited hours per day.

How many hours you get to work of course depends on the tasks availability.

 ➡ Raterlabs  

Since you are hired as a part-time employee with them, one has to give a minimum of 15 hours per week.

The maximum hour that one can put in is 26 hours per week.

 ➡ Lionbridge

You get to work for a minimum of 10 hours per week and a maximum of 20 hours per week.

But you do get occasional bonus hours per week.

 ➡ iSoftstone 

They require a minimum of 10 hours per week. 

They may increase your per week hour up to 20+ hours limit depending on the quality of your work.

8. Are these tasks available all the time?

No. US Holidays, weekends, Christmas, New Year, etc means that the task flow will be less.

But one can easily hit 20 – 40 hours per week. 

There were weeks where I hit 60+ hours/week!

9. How much do they pay per hour?

Payment differs from company to company. 

Here is a quick comparison of pay rates per hour.

 AppenRaterlabsLionbridgeiSoftstone
India$3.5N/A$6.79$6
US$10-$13.5 (Depends on the state)$10 – $13.5 (Depends on the state)$11 – $13.5$11 – $13.5
UK$13 – $14N/A$13 – $14$13 – $14
EUUp to $14N/AUp to $14Up to $14
Mexico$8 – $9N/A$9$9
Others $4 – $20 (Japan’s rate is high)N/A$4 – $19 (Depending on the country)$4 – $19 ((Depending on the country)

Leapforce used to pay us Indians $7/hr before. But after its acquisition by Appen and after being re-branded as Appen Connect, they have reduced the pay rate per hour to $3.5 for new Indian raters.

10. How much money can I make?

You can make anywhere from $500 to $4000 approx depending on where you live. 

I personally made $2007 in a single month being an Indian, although we are just paid half of what the Americans or Europeans are paid*

11. Sign Up Bonus for US candidates 

If you are an American, and if you sign to work for Raterlabs, you get $250 bonus.

Here is how you will be paid the bonus amount of $250.

  • If you complete 20 hours within the first 2 weeks after you hit production, you would be paid $100.
  • Another $150 for 20 hours in Week 3 and Week 4 after hitting production.

That’s a total of $250 in bonus on top of your regular pay per hour.


Now, let’s discuss the Pros and Cons of being an Evaluator:

PROS

  1. Boss Free – You don’t work under a direct supervision of anyone. You set your own hours and you decide when you work – morning time or evening time.
  2. Location Independence – You can work from anywhere in the world as long as you are within your country.
  3. Help Better The Internet – You are indirectly helping Google to better their search results. You help them refine their algorithm and show better, relevant results to users.
  4. Make Decent Money – You make decent money in your free time. One can easily make $500 – $4000 depending on where you are located.
  5. No first come first served tasks – If you have done website testing projects in Usertesting and other similar websites, you would see that you have to rush to grab the task before it’s gone or good ( or bad). It is not like this here. If there are tasks, you would get to work it even if you don’t rush.

CONS

  1. Not an employee – Except for Americans, you cannot be an employee of these companies that hires evaluators. You are just an independent contractor. But can I work as an evaluator for a long, long time? Hell yeah! I’ve been with them for almost 4 years now ( as of Jan 2020), and I know people who’ve been with them for like 8-9+ years.
  2. Constant quality checks – Another drawback of this work is that there is a monthly quality check where you are supposed to maintain a certain minimum score. Failure to maintain the minimum score for multiple times may result in removal from the project. However, with multiple training materials, feedback, quizzes, etc, one can easily maintain the minimum score.
  3. Tasks unavailability – There may be some months where the tasks flow would be less. You may struggle to even hit 10-15 hours per week. Also there are no tasks during weekends, US holidays, etc. But normally, one can easily hit 20-40 hours per week.
  4. Self Time-tracking – You will have to track your worked hours on your own. This is another drawback of this work. 

But the overall pros far outweighs the cons of working as an evaluator. I am the living proof of it. I regularly make decent money,I have been with them for almost 4 years now.

How to become a search engine evaluator aka rater?

  • The first step to becoming a search engine evaluator is applying at one of these four companies:
    1. Appen (I work here)
    2. Raterlabs (Only for Americans)
    3. Lionbridge.
    4. iSoftstone.
  • I would highly recommend you to apply for all four search engine evaluator companies and go with the one that responds the fastest.
  • Take their qualification test (more on that later).
  • Create a Payoneer account.

Note: You can’t work as an evaluator for Appen or Raterlabs or Lionbridge at the same time. It is because the client of all the three companies is Google. However, one can work for one of the above three companies (Appen/Raterlabs/Lionbridge) and iSoftstone at the same time. It is because iSoftstone’s client is Microsoft.

  • And, then finally sign the paper works.

How to Qualify for the Rater Test?

To work as an evaluator, you need to pass a very tough qualification test.

In this article I will only be talking about qualification test for Appen, Raterlabs and Lionbridge. It is because I work with Appen and the qualification test is the same for all the three companies mentioned above.

I won’t be talking about iSoftstone’s qualification test.

So the qualification test consists of three parts –

  1. Theoretical Test – It consists of 32 multiple choice questions taken directly from the General Guidelines.
  2. Practical Test I – This is called the Page Quality test and it comes from the first part of the General Guidlines pdf. 
  3. Practical Test II – This is called the Needs Met test and it comes from the second part of the pdf .

This qualification test is so tough that hardly 1 in 100 get through it, and work as an evaluator. How do I know? Because I have referred thousands of people to Appen. I have the numbers of people that I’ve referred to, and the number of people that have actually gone on to pass this test.

But worry not! You’re in for a secret sauce! I have created an in-depth Search Engine Evaluator Course’ that’ll help you excel and qualify for the test. Awesome, right? Avoid mistakes that people normally do while sitting for this test, and ace the test! I highly recommend you to check out the course right now. Use the coupon SWATI10 in checkout for $10 off (limited students only).

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.