How crazy is it to analyse variables with 50000 levels?

Luke Lake (Dept of Immigration): Should enormous numbers of categorical values in a single column be analysed by data mining? Abstract: With Greg Ridgeway, author of the R package classification program, Gradient Boosting Machine, consent, I changed the number of allowed categorical values in a column from 1024 to 80000. Although it works and produced some amasing and not so amasing results, the issue remains, Why do it?. This seminar will explore some of the mathematical, statistical, computer and business reasons for and against. For example, a business reason to do it is to mine available data, DIAC has data that has over 50000+ unique categorical values in a column. An algorithm limititation is overcome. The 45G RAM machine being used can not handle 50000 values being turned into dummy variables using the dummies R package. But changing the number of the categorical values from 1024 to 80000 in the same package then the machine can process the data. I will use the C++ code in GBM Bernoulli distribution as a straw thingo to show how the categorical data is processed. This hopefully will lead to discussion on the above mentioned issues.