techborg

In what must be a stunning development across the web, AOL in its foolhardiness (depends on how you look at it) has released a research paper showing data of “top searched queries” on AOL by 650000 users resulting in a data set of nearly 20 million search terms.

According to AOL:

500k User Session Collection
———————————————-
This collection is distributed for NON-COMMERCIAL RESEARCH USE ONLY.
Any application of this collection for commercial purposes is STRICTLY PROHIBITED.

Brief description:

This collection consists of ~20M web queries collected from ~650k users over three months.
The data is sorted by anonymous user ID and sequentially arranged.

The goal of this collection is to provide real query log data that is based on real users. It could be used for personalization, query reformulation or other types of search research.

Why is this important? Because AOL uses Google as their search engine. Which means anyone laying their hands on this data potentially has nearly 20 million queries to work with to come up with list of top keywords people are searching for!

Imagine the possibilities with this - anyone can manipulate this data with the search terms and make websites for those queries. Its obvious that Google is now going to be majorly pissed at AOL for leaking this data. But whats more concerning is the way AOL has released this keyword list, with not enough concerns about any privacy. (Links below on this).

I will be keeping a tab on this story as it develops and people come up with more and more data analysis of this data, and general views on this subject.

More resources on this huge development:

Forums.Digitalpoint.com is carrying a thread on this topic, and naturally everyone is stunned on being able to access the goldmine.

Plentyoffish.wordpress.com  has already begun to analyze the search data and is posting the results of his analysis. Some of the queries are downright disturbing. Specific posts include “Aol data shows users planning to commit murder”, “AOL data showing Myspace growing SEO spam” “Myspace killing dating sites”

Head over to http://www.gregsadetsky.com/aol-data/ to find links to download this huge 500 Mb behemoth file.

Techcrunch is on top of the story as well, and echoes my sentiments as to why this is a huge concern as well:

The most serious problem is the fact that many people often search on their own name, or those of their friends and family, to see what information is available about them on the net. Combine these ego searches with porn queries and you have a serious embarrassment. Combine them with “buy ecstasy” and you have evidence of a crime. Combine it with an address, social security number, etc., and you have an identity theft waiting to happen. The possibilities are endless.

Adios for now. More updates later.
Update: 8.7.2005

Reuters now reports that AOL is facing backlash from people. Why do I feel this will turn out to be a public embrassment for AOL - are lawsuits next?

Meanwhile, an AOL spokesperson has posted an apology on http://plentyoffish.wordpress.com

All –

This was a screw up, and we’re angry and upset about it. It was an innocent enough attempt to reach out to the academic community with new research tools, but it was obviously not appropriately vetted, and if it had been, it would have been stopped in an instant.

Although there was no personally-identifiable data linked to these accounts, we’re absolutely not defending this. It was a mistake, and we apologize. We’ve launched an internal investigation into what happened, and we are taking steps to ensure that this type of thing never happens again.

Yeah - a good screwup, but a boon for MFA’s I would believe. I will post a analysis soon on this.

In what must be a stunning development across the web, AOL in its foolhardiness (depends on how you look at it) has released a research paper showing data of “top searched queries” on AOL by 650000 users resulting in a data set of nearly 20 million search terms.

According to AOL:

500k User Session Collection
———————————————-
This collection is distributed for NON-COMMERCIAL RESEARCH USE ONLY.
Any application of this collection for commercial purposes is STRICTLY PROHIBITED.

Brief description:

This collection consists of ~20M web queries collected from ~650k users over three months.
The data is sorted by anonymous user ID and sequentially arranged.

The goal of this collection is to provide real query log data that is based on real users. It could be used for personalization, query reformulation or other types of search research.

Why is this important? Because AOL uses Google as their search engine. Which means anyone laying their hands on this data potentially has nearly 20 million queries to work with to come up with list of top keywords people are searching for!

Imagine the possibilities with this - anyone can manipulate this data with the search terms and make websites for those queries. Its obvious that Google is now going to be majorly pissed at AOL for leaking this data. But whats more concerning is the way AOL has released this keyword list, with not enough concerns about any privacy. (Links below on this).

I will be keeping a tab on this story as it develops and people come up with more and more data analysis of this data, and general views on this subject.

More resources on this huge development:

Forums.Digitalpoint.com is carrying a thread on this topic, and naturally everyone is stunned on being able to access the goldmine.

Plentyoffish.wordpress.com  has already begun to analyze the search data and is posting the results of his analysis. Some of the queries are downright disturbing. Specific posts include “Aol data shows users planning to commit murder”, “AOL data showing Myspace growing SEO spam” “Myspace killing dating sites”

Head over to http://www.gregsadetsky.com/aol-data/ to find links to download this huge 500 Mb behemoth file.

Techcrunch is on top of the story as well, and echoes my sentiments as to why this is a huge concern as well:

The most serious problem is the fact that many people often search on their own name, or those of their friends and family, to see what information is available about them on the net. Combine these ego searches with porn queries and you have a serious embarrassment. Combine them with “buy ecstasy” and you have evidence of a crime. Combine it with an address, social security number, etc., and you have an identity theft waiting to happen. The possibilities are endless.

Adios for now. More updates later.
Update: 8.7.2005

Reuters now reports that AOL is facing backlash from people. Why do I feel this will turn out to be a public embrassment for AOL - are lawsuits next?

Meanwhile, an AOL spokesperson has posted an apology on http://plentyoffish.wordpress.com

All –

This was a screw up, and we’re angry and upset about it. It was an innocent enough attempt to reach out to the academic community with new research tools, but it was obviously not appropriately vetted, and if it had been, it would have been stopped in an instant.

Although there was no personally-identifiable data linked to these accounts, we’re absolutely not defending this. It was a mistake, and we apologize. We’ve launched an internal investigation into what happened, and we are taking steps to ensure that this type of thing never happens again.

Yeah - a good screwup, but a boon for MFA’s I would believe. I will post a analysis soon on this.