Google AND operator

Hi s_federici,

The reason the second phrase returns more results is that you are telling Google to also look for the word "and."  The search algorithm changes slightly and more results come back and the order may be affected as well.

Google defaults to searching for all words given to it, but it doesn't say AND is useless...it says it isn't necessary.  

Try searching for something like:

"want to eat"

and then

"want eat"

and you will see drastic changes in the number of results based on the word "to"

This seems very reasonable to me. So I looked to the very first pages I got by asking for "agriturismo AND toscana". The two first pages didn't show any HTML. So I went to the third page http://www.teseotur.com/toscana/agriturismo/. I looked inside the HTML and also saved the page to my PC and then looked at the saved files. I couldn't find any "AND" inside it. Could you please check this same address too and let me know where I am wrong?

Thanks

Sorry, the http://www.teseotur.com/toscana/agriturismo/ is (right now) the FIFTH page in the result list.

abbracciANDo
CliccANDo
bANDiera

It might be doing a partial match on those...

I see. But, from what is stated in Google search help, it shouldn't. From what they say, only exact matches should be taken into account. Indeed, from Google help (English) I read (http://www.google.com/help/basics.html):

"[...] when appropriate, it will search not only for your search terms, but also for words that are similar to some or all of those terms. [...] Any variants of your terms that were searched for will be highlighted in the snippet of text accompanying each result."

So, they say that if you look for "dietary" also "diet" will be looked for. But the two worde must be similar.

Instead, in the Italian help, they state that only exact matches are looked for, and they also suggest to look for both singular and plural forms.

Sorry, if I knew more I'd be working for Google.  For what it's worth, I never believe the help docs :)

I see. But I'd need some clear information about google's AND operator (even if this is not an operator).

You're getting to a level of detail that people may not be able answer, as they would have to have access to Google's programming.

What I need is the answer to my question. Nothing more. Guesses are welcome, but if they are not supported by evidence then I need other guesses. Maybe, eventually, we can get to an acceptable guess.

You might want to try asking these folks:

http://groups.google.com/group/Google_Web_Search_Help

if you haven't already

I posted to the group. Thanks for the suggestion. I'll keep you posted.

Please do and we'll get this closed out as answered by you.  I'm super curious now...

By enclosing the search phrase in quotes, you're creating a string for which exact matches will be found. If you search for toscana AND agriturismo in quotes, you'll get just that. Since I don't know what that is, I'll use other words as an example: soccer and stadium.

If I search for soccer and baseball, I will get pages where soccer and baseball appear in the text. The words soccer and baseball may be separated by two paragraphs, but they will be there.

If I search for "soccer and baseball" I will get results where those three words (soccer, and, baseball) appear together in that order. For example: Americans play soccer and baseball.

PS. The search for soccer and baseball (no quotes) will return everything that "soccer and baseball" (with quotes) does and then some.

The "soccer and baseball" (with quotes) query is more specific.

Sorry, I guess I got you confused by using quotes to enclose the search terms in my question to EE. I didn't use quotes in my searches, they were in the question only to be more clear. Speaking of search engines, your comment is certainly legitimate

Here's ONE of the reasons why you get different results when you include the word "AND" in your search...

The accuracy of the number of results when searching for foreign words is not all that good.  

For instance if you search for toscana AND agriturismo:

http://www.google.com/search?num=100&hl=en&newwindow=1&safe=off&q=toscana+AND+agriturismo

You will see a link at the top of the search that says, Search for English Results only.  If you click that, you should get a result count of 223,000 instead of 1,730,000.

Remove the AND from the search:

http://www.google.com/search?q=toscana+agriturismo&num=100&hl=en&newwindow=1&safe=off&lr=lang_en&sa=X&oi=lrtip&ct=restrict&cad=9

And you will still get a result count of 223,000

But there's yet another reason...

Google's searches are cached and weighted... and the statistics quoted at the top of the page are just estimates.

You might find it interesting that searching for toscana agriturismo will bring you a different amount of results than  agriturismo toscana.  

It's been said that the number listed in the search statistics is derived by pigeons (http://www.google.com/technology/pigeonrank.html)  But in reality they do say "ABOUT", because that number is never accurate as it changes by the minute, and can't ever really be exactly correct.

If you want to see how the searches for toscana agriturismo have been fluctuating, check out:
http://www.google.com/trends?q=toscana+agriturismo&ctab=0&hl=en

If you want a comprehensive overview of how Google works, and what it all means, please see:
http://www.googleguide.com

Jeff
TechSoEasy

I have noticed exactly what you described but have not really investigated it.  I have always thought it a bug or at least caused by some special use of AND by Google.  I agree with the statement about the Help; the info is good but often "generalized" and so not completely accurate.  I have found the groups very helpful in the past so let us know if you get info there.

I will look into this and see if I can find some of the hard facts you want for your answer but I have no special connection to Google (darn) and am afraid that a definite and clear answer will not be found.

I can tell you that Google uses an "implied AND" so they don't want or expect you to use AND to join terms and narrow the search.  There is an exception with the + character when you want to seach for characters that would usually be ignored but that isn't directly related to your question.  Using and in a phrase (surrounded by quotes) will not have the effect you describe either.  It is just when you make a query that would normally not narrow with more terms (i.e. implied OR).

I hope this helps to get you started.  I will look more and let you know if I find something new.  If you want some suggested reading on this then look at the links below.  I have found the info on that site interesting and accurate.

http://www.googleguide.com/category/query-input/
http://www.googleguide.com/interpreting_queries.html

Let me know if you have a question.

bol

I have no explanation, but here is something that could be clue:  
Note that there are more "Sponsored Links" for the query that does not have AND

One would think that the sponsored link list is based on the keywords of the query.

=--=-=-=-=-=-=-
One unsubstantiated theory:  Perhaps the AND makes google ignore some variations of words for instance, without AND, it recognizes
   tuscany
...and/or...
   l'agriturismo

...but with AND, it does not.  I could not find proof of this theory, but what the heck, nobody seems to have any real idea.

=-=-=-=-=-=-=--==-=-=-=-=-=-=-=-=-=-=-=--==-=-=-=-=-=
By the way:  My searches yield
   1,730,000  -- with AND
   1,670,000  -- without AND

There is less than 4% difference.  In the English version of google, it says
   "of about 1,730,000..."
and I submit that "within 4%" is another way to say "about"  :-)

Just as a further demonstration that the number of results is NOT at all a precise number:

www.google.co.uk:
about 1,660,000 for toscana AND agriturismo
about 1,670,000 for toscana agriturismo

www.google.it:
su circa 1.680.000 per toscana AND agriturismo
su circa 1.630.000 per toscana agriturismo

www.google.fr:
sur un total d'environ 1 670 000 pour toscana AND agriturismo
sur un total d'environ 1 680 000 pour toscana agriturismo

:-)

TechSoEasy

Even more odd is the change in the order of the results.  Of course that might just be caused by the fact that the second term is now the third term.  The order of the terms certainly effects the results.  Adding AND, even if Google ignores it for the most part, could be lowering the importance of the word on the second word.

bol

I'm now studying googleguide and waiting for an answer from the google group. I'll keep you all posted. Thanks to everyone for your help.

I went through googleguide but apparently didn't find any mention to something related to my question. I hope the the google group will be able to drive my search to the (or maybe just AN) answer.

Here ya go... a great resource for all kinds of opinions on Google's Search Counts:
http://blog.searchenginewatch.com/blog/050128-134939

Jeff
TechSoEasy

looking at...

So, this is the simple answer? Google's counts are simply unreliable? It is true that I do not expect to be able to scan 2.280.000 results, but the number of results it is something meaningful to me. If the count is wrong of about 4% or so, this is not bad. But what if the count is wrong of hundred thousands? Is it possible? Or google's counts estimate has an upper-lower bound as for the expected error?

>> I went through googleguide but apparently didn't find any mention to something related to my question. <<

Sorry if it wasn't clear but that was just meant to be a site with good info and pages about queries and the "implied AND."  I didn't mean imply that it answered your question directly.

>> Google's counts are simply unreliable? <<

They aren't exact so that would mean at least somewhat unreliable.  The number isn't meant to be exact and proof is Google using "of about" when displaying the number.  The fact the number is always rounded to the thousands is also proof of it.  Given the rounding it is very likely the "count" will be off by at least thousands if not a few thousand.  The larger the results the more likely the error range (or maybe tolerance, I don't know what term would be best in this case) on the exact count will be higher.  If you do a real specific search then using AND doesn't affect the results at all.  For example I did (b0lsc0tt AJAX) and (b0lsc0tt PHP) and got the same results with or without AND between the terms.  The count of results is exact in that case too.

From my experience with Google and searching with it I really think a key to the change in the number and the order of results is the fact that AND lowers the importance of the second term.  The order of the terms will affect the results and it is important to list the more specific or relative terms first for better results.  It is still odd that the reported count would increase but that I am sure is related to the fact that Google doesn't use an AND as an operator in the query; it isn't meant to join terms.

Let me know if there is a question about this.

bol

"but the number of results it is something meaningful to me"

Well, actually if you're looking for reliable search statistic results to improve your own site's visibility, that's not the place to do it.  Each search engine has tools that will help you maximize results.

Google's:  http://www.google.com/intl/en/webmasters/
Yahoo's:  http://developer.yahoo.com/search/siteexplorer/
Live's:  http://help.live.com/help.aspx?project=wl_webmasters

There are also plenty of other third-party tools and sites which can provide search engine statistics and help.  If that's what you're looking for, then please advise and I can make some recommendations.

Jeff
TechSoEasy

Thansk Jeff, but I'. interested in search statistics for other purposes (I'm a researcher in natural language processing). So I need reliable statistics.

So, setting apart reliability,  AND seems an operator that lower the importance by changing the position of the second term (b0lsc0tt). That could be a good reason if true. Do you guys share b0lsc0tt's opinion about that?

It makes as much sense as anything else I've heard here and elsewhere.

"Do you guys share b0lsc0tt's opinion about that?"

No, not really...

www.google.co.uk:  about 1,670,000 for toscana agriturismo AND
www.google.it:        su circa 1.690.000 per toscana agriturismo AND
www.google.fr:        sur un total d'environ 1 710 000 pour toscana agriturismo AND
www.google.com:    about 1,670,000 for toscana agriturismo AND

Basically by moving AND to the third position doesn't change a thing on www.google.com, and has variable changes on the others.  As stated above, you really CANNOT use that number for any kind of reliable research.

I certainly do not know anything about NLP, but I'm sure there are a number of resources that are much more reliable than the "Results" stat line.

Jeff
TechSoEasy

I think you are looking for a bit more info.
Instead of using the word "AND" as an operator you need to do it like this "bread" AND "milk"
or bread + milk
you can also use an or operator "bread" OR "MILK"  
also
"BREAD" NOT "MILK"  
or "bread" - "milk"
it's essentially BOOLEAN algebra.

Hope this helps

this is a good link
http://www.google.co.uk/intl/en/help/refinesearch.html#or

Everyone,
 you'll see this answer on both the original question and the old one as the answer was not listing in my profile and I thought I need to put it on the old question.
I'm not trying to get double points or anything

ID:22944944Author:s_federiciDate:11.07.2007 at 08:14AM PST | Points: 500

the one that this references

As far as I can read from google help, using + shouldn't add much to the search. "+"  is only useful if you look for common words. And "sardergna" and "toscana" are not.

I know of other operators, but I would like to understand the reasons for the strange behaviour of Google that I have described in the question of this thread.

I do wonder why you insist on calling the behavior "strange".  Google readily admits that those numbers aren't supposed to be totally accurate.  They are just a way for Google to flaunt their capabilities in order to impress users.  It's a marketing gimmick.

I already had provided you links to a source that has a number of articles on the subject, but perhaps you didn't read any of those?

Here are a couple that you should read:  

http://www.searchengineshowdown.com/features/google/inconsistent.shtml
http://www.sifry.com/alerts/archives/000320.html
http://searchenginewatch.com/showPage.html?page=2162421

As you can see, these aren't new... this issue has been around since before Google even existed:
http://searchenginewatch.com/showPage.html?page=2166151

So... as I stated a number of times in this thread already... if you need statistics for research you will not find them at the top of a Google search result page.

Jeff
TechSoEasy

By the way... that first link above is really a great source for you to learn more about this:
http://www.searchengineshowdown.com

Jeff
TechSoEasy

Nevertheless, I believe that the asker has a valid point:  Whatever method Google uses to determine that value, one would THINK that it would be the exact same for both search strings -- after all, that's what they claim every time you use the word AND.  

So the mystery does not disappear by pointing out that the values are inexact and approximate.

It's still not a mystery to me...  

Google's results are mix of indexed, unindexed, "other file types", and daily re-indexed pages.
(source:  http://www.searchengineshowdown.com/features/google/dbanalysis.shtml)

There are thousands of web pages that change every single hour, so it's certainly likely that the cached results from one set of words will be different than another.  Remember as well, Google states this about the use of "AND":

          Automatic "and" queries

          By default, Google only returns pages that include all of your search terms.
          There is no need to include "and" between terms.

The don't say that they will ignore the "word" but rather that they won't treat it as an OPERATOR.  They do have a list of "stopwords" which may or may not include "AND".  But here's the kicker...  (and something I mentioned above)... when doing a general search, you are searching ALL languages unless you limit to a specific language... and the word "AND" is NOT a stopword in many languages... including Italian:  http://www.ranks.nl/stopwords/italian.html.

Jeff
TechSoEasy

To demonstrate my point about language differences, take a look at this:  http://snipr.com/1tszu
(you may have to click the "compare using the same engines" button at the top to get it to display right.

Jeff
TechSoEasy

Using AND as suggested in http:#20303901 will not "fix" this and isn't suggested by Google.  Unless you want to include the word and in the search you should leave it out.  Putting the other terms in quotes and leaving AND out won't make a difference.  If you really want to seach for the word and then it would be good to put it in quotes.

Also, using + in the query isn't the same as AND (which is assumed remember, it should not be used in the query).  Using the plus symbol (+) like in the comment I referenced is to have Google not dismiss a character or common term.  The example in the comment above isn't a great example of its use.  Milk isn't a term that would be dismissed and the plus should be next to the character or term.

There a number of uses of plus.  For example if the search needed to include the word 'the' then you would use +the.  Otherwise, as Google will mention on the results page, the term 'the' is ignored as common.  When searching for something with The in the title or name using the plus and the term will improve results.

The use of the minus should also be next to the term.

Let me know if there is a question about this.  I hope it clarifies the info posted above.

bol

from an early thread you say that you put in "agriturismo AND toscana".  try to find the page that came up then.  I can't seem to find it.  Can anyone else?

Sorry if I don't answer to every single post, but I'm trying to figure out from your comments what is that weird mechanism that google uses so that different search string that they APPARENTLY state to be the same, give different results. Maybe caching is an answer. But I would like to be able to have good evidence.

As for last wittyslogan's question: I can find the same page (www.piedicosta.com/italiano.htm) at the eighth positions today in my Italian search results (at the 2nd in the English-US search results).

As for the language dependence suggestion: in the English-US Google I get

agriturismo AND toscana: 2,250,000
agriturismo toscana: 2,220,000

Instead, In my Italian Google:

agriturismo AND toscana: 2,360,000
agriturismo toscana: 2,340,000

So, apparently AND means "more". Is it all cache related?

PS: It seems perfectly reasonable to me that the same search string gives different results in different languages. But the string with or without AND should give, in my very personal view, the exact same results when looked at in the same language version of Google.

but google states that AND in capital letters is an operator as long as the search terms are in inverted commas.

By the way you are looking for stuff on an italian version of google it seems.  I understand that "e" is AND in italian.  Is there a different word for AND in the field of electronics and mathematics in italian?

"google states that AND in capital letters is an operator as long as the search terms are in inverted commas"

Really? Where???

"Is there a different word for AND in the field of electronics and mathematics in italian"

No, we use the same word as English speaking countries do, i.e. AND. Indeed, if you use "e" in the Italian Google, it (apparently) takes it too into account. Even if, I think, it should be considered as a "common word". You get a warning only if you use AND.

Google does not state that AND is an operator.

But I do think it's somewhat of a language issue at best because check out what happens when you search for it in English on www.google.com:

Results 1 - 100 of about 171,000 for Tuscany Agritourism
Results 1 - 100 of about 171,000 for Tuscany AND Agritourism

But even moreso, Google will sometimes consider "agriturismo toscana" as a single term and when you put the AND in the middle it stops it from being able to do that.

As an example:
Results 1 - 100 of about 1,610,000 for agriturismo toscana
Results 1 - 100 of about 1,650,000 for toscana agriturismo

So, even without the AND, two words in a different order are going to return a different result.

The thing still is that you haven't exactly stated why this information is even important.  Because you stated, "(I'm a researcher in natural language processing). So I need reliable statistics."  and then said, "setting aside reliability..."

So which is it?  Because I've stated a number of times, that if you want reliable statistics, the number returned at the top of a Google (or any other search engine's) Search results is not the place to get it.

Jeff
TechSoEasy

I see a mismatch in some opinions. Wittyslogan says that "google states that AND in capital letters is an operator as long as the search terms are in inverted commas", whereas TechSoEasy says that "Google does not state that AND is an operator".

As for the difference we all can see in results when ordering the search terms in a different way, this is clearly stated by Google ("Keep in mind that the order in which the terms are typed will affect the search results" at http://www.google.it/intl/en/help/basics.html).

Last, as for TechSoEasy's request of clarification: currently I'm not interested in google's number of results for my research purposes (at least, not only). I asked this question as I want to know why google is apparently so relaxed to make statements that -at my eyes- simply don't appears to be true.
There is a clear difference to me between stating that "There is no need to include "and" between terms" (http://www.google.it/intl/en/help/basics.html) and "YOU SHOULD NOT include "and" between terms, AS YOU WILL GET UNPREDICTABLE RESULTS".

Is this true? Google results are unpredictable? Or (still) is it simply a cache problem?

I'm stil bothering you guys as I still can't get an answer from "Google Web Search Help" group (http://groups.google.com/group/Google_Web_Search_Help-UsingWS/browse_thread/thread/f49f71b02999a791/2157e1095f981880?lnk=gst&q=%22google+and+operator%22#2157e1095f981880 and http://groups.google.com/group/Google_Web_Search_Help-Troubleshooting/browse_thread/thread/2e91b5bd49c58709).

Here's another reasonable hypothesis:

There are exactly the same number of matches... but the displayed *estimate* is inexact.

The reasoning behind this theory:
Google displays an inexact (*rounded*) estimate because it would take some computing resources to de-dup the result lists and display an exact number.  Or perhaps some pages are in the process of being re-indexed at the moment of the query so tallies can't possibly be exact to the unit magnitude.

They decided that a few percent difference in a number that large would not affect the overall quality of service.

After all, they will only display the highest 1000 matches.  So matches numbered from 1001 through 172,000 (actually somewhere between 171,500 and 172,499) are pretty much irrelevant.

"I want to know why google is apparently so relaxed to make statements that -at my eyes- simply don't appears to be true."

Overall it doesn't really matter because you won't be able to ever see more than 1,000 results, (http://www.google.com/support/bin/answer.py?answer=484)
so any estimated number above that is just an indexing feature so that you will know your search terms are returning "A LOT" of answers and perhaps you should add some words to your search to narrow it down a bit.

I think that you'd find that number is much more accurate when it's smaller (less than 1,000), but there will always be a bit of variance based on which of Google's datacenters returned your results, whether or not "supplemental results" are included, etc.

I have explained this in many different ways already... (most specifically here:  http:#20305128 ) and if you don't understand what I've said, then please advise.  

Jeff
TechSoEasy

The reasoning is plausible. Do you guys think then that it is impossible to know which is the true end of this story? Should we stay in the realm of plausibility?

Further to the discussion I was sure that Google stated that AND was an operator.  However I can't find any reference but...just try this experiment.

look for:
and;
AND;
"and";
"AND"

under any circumstances these should be the same.  but they are not.
now if you go to the google help on:
http://www.google.co.uk/intl/en/help/basics.html#keywords
and look for capitalization it states that it looks at all words as lower case.
If these figures are all different then that statement is false.

>> Do you guys think then that it is impossible to know which is the true end of this story?

Yes.  This is the sort of thing that only a Google insider would know for certain and they invariably work under non-disclosure agreements.  If one speaks up, there's no way to know if he's authentic or if (assuming he is) his info is current and accurate.

In short, the best anyone can do is theorize intelligently.  

Finally, since (as TechSoEasy has pointed out in various ways multiple times) the question is basically moot (though I might add, except for satisfaction of idle curiosity), there isn't much value in persuing it further.

-- Dan

To answer TechSoEasy: I had read the pages you suggested. But knowing that there are problems is not something I need. I can see that there are problems. I would like to know if there are solutions to my particular problem. Or if I have (again) to remain in the real of plausibility. If no one knows (or is able to discover) why google behaves this way, I will stick with this.

Anyway, reading again your suggested link, make me think better on one of the problems described, namely the following one:

"HIDDEN QUERY TERMS: This can happen with any search engine. Sometimes query words do not show up in some results or in the cache. This can be due to the query word matching a link on another page that points to the URL that is listed on the search results page. [...] apparently, Google is [also] matching on words within redirected URLs.

Maybe this can be the reason. The word "and" shows up in page that links the pages shown in the results (even if those pages do not contain "and"). However, that would mean that "and" IS a search term... and it shouldn't. But, as wittyslogan points out, maybe it is:

and: 10.090.000.000
AND: 10.170.000.000
"and": 10.180.000.000
"AND": 10.130.000.000

... and, more importantly, no ALERT about AND being unnecessary (thanks to Go-d-ogle!)

Look, if this question is intended to rant about Google you've placed it in the wrong category.  It should be in the Lounge Zone for something like that.  

You've asked a question and it's been answered appropriately.  Either close it out and or request that it be deleted in the Community Support Zone.  Either way, I'm done unless you need further clarification on what I've already stated.

Thanks!

Jeff
TechSoEasy

similar problem with OR.

So there is a question as to how the AND and OR word works.

Now we are almost veering into philosophy here but to be fair if you were trying to reverse engineer a search engine these subtleties might be useful.  

If we assume that AND is an operator but ALSO a word does that start to change the logic of the query?

And how does "and" differ from and?

That would be a marvellous exam question.

There is no question as to how AND and OR work in google search unless you choose to ignore what google says about how they work.

http://www.google.com/intl/en/help/basics.html has been posted time and again in this topic but the asker and wittyslogan appear unwilling to accept what it says at face value.

Google is a search engine with a proprietary algorithm and index weighting, and its index cache databases are geographically dispersed and are not 100% in sync.  It's not a database of a fixed number of never-changing reference texts.  Its results will change moment to moment, and from geographic location to geographic location, depending on which server you chance upon, so yes, it is unpredictable to a large degree.  Get over it.

If there is truly a linguistic research aspect to this question, I suggest that perhaps a question specific to that research, and not centering on google search, might be more appropriate.

Have the same searches been done using other search engines?  Are they consistent with their results no matter how your search terms are worded?

"The question of 'why?' has been answered to the greatest degree that is can be"

Well, the answers I got could be divided in three groups:
1. google is not reliable: do not further investigate it
2. google uses cache: this is why you get different answers
3. google AND behaves weirdly and it shouldn't

As I said, I can stick to asnwer 1, but the fact that it is not shared by everybody make me thing that there is something inherently wrong in the way Google handle its operators.

Even answer 2 could be ok, but I keep getting answers that do add something to it, with further evidence that I consider pretty useful.

Answer 3 has been tentatively answered by some of the experts. And I really appreciate it.

So I really don't see why this question should be closed, as the question is not of a philosophical nature at all and, more importantly, I keep getting answers. Interesting answers to me.

And, please, don't consider this question a question centered on linguistic research. It is not.

"Have the same searches been done using other search engines?  Are they consistent with their results no matter how your search terms are worded?"

Sure. If you would like to try for example "www.altavista.it", you will get here too different results. But in the last version of altavista.it help you won't even find any mention to operators such as AND or OR. Only the advanced search form is clearly described.

Finally, I didn't get any opinion from you guys about what I described in my last post, namely:

"HIDDEN QUERY TERMS: This can happen with any search engine. Sometimes query words do not show up in some results or in the cache. This can be due to the query word matching a link on another page that points to the URL that is listed on the search results page. [...] apparently, Google is [also] matching on words within redirected URLs."

What do you think? Does it make sense?

Thanks for all the help you are giving to me.

s_federici,

I'm a little puzzled by the way you closed this.  Glad to see you did but it might be nice to know why.

bol