Friday, 25 December 2015
Friday, 31 July 2015
Run a User Recommendation on Mahout
For a beginner like me, the following steps were useful for running user recommendation algorithm of Mahout.
1- I downloaded the file from http://grouplens.org/datasets/movielens/, the directory contains many files the one which I thought will be useful for me was ratings.data of size 568.3 MB, contianing the fields userId, movieId, rating, timestamp. Mahout's recommenders expect interactions between users and items as input. Every line of the file has the format userID,itemID,value. Here userID and itemID refer to a particular user and a particular item, and value denotes the strength of the interaction (e.g. the rating given to a movie).
(Perform the following steps after login as hduser (user for hadoop cluster))
2- I removed the last field of time stamp from the file (as it was not required for the current recommendation) using the following command and saved it in .csv file
cut --complement -f 4 -d, ratings.data >ratings.csv (To remove 4th column- timestamp from the file)
3- Now I created directory in hadoop file system to store the ratings file using the following command
hadoop fs -mkdir /mahout_data/
4-Now I copied the downloaded file of movie recommendation to hdfs using the following command
hadoop fs -put /home/hduser/mydata/ml-latest/ratings.csv /mahout_data/
5- go to the mahout directory cd /usr/local/mahout/bin/ and issue the following command to run :( The output file should be unique and JAVA_HOME should be properly set)
./mahout recommenditembased -s SIMILARITY_LOGLIKELIHOOD -i hdfs://localhost:9000/mahout_data/ratings.csv -o hdfs://localhost:9000/ratings_test/ --numRecommendations 25
-i hdfs://localhost:9000/mahout_data/ratings.csv - Denotes the input file
-o hdfs://localhost:9000/ratings_test/ -denotes the output file .
recommenditembased - Means we are creating itembased recommendation not user based recommendation, there is a difference between itembased and user based recommendation, a user based recommendation finds similar users , and see what they like, item based recommendation see what the user likes and find similar items, Mahout's item-based recommendation algorithm takes as input customer preferences by item and generates an output recommending similar items with a score indicating whether a customer will "like" the recommended item.
Choosing a similarity measure for use in a production environment is something that requires careful testing, evaluation and research. For our example purposes, here I used Mahout similarity classname called SIMILARITY_LOGLIKELIHOOD.
6- It will run for a couple of minutes you can see your output from web interface as well
7- You can check the output file it will contain two columns: the userID and an array of itemIDs and scores.
References :
http://mahout.apache.org/users/recommender/intro-itembased-hadoop.html
http://grouplens.org/datasets/movielens/
http://info.mapr.com/rs/mapr/images/PracticalMachineLearning.pdf
1- I downloaded the file from http://grouplens.org/datasets/movielens/, the directory contains many files the one which I thought will be useful for me was ratings.data of size 568.3 MB, contianing the fields userId, movieId, rating, timestamp. Mahout's recommenders expect interactions between users and items as input. Every line of the file has the format userID,itemID,value. Here userID and itemID refer to a particular user and a particular item, and value denotes the strength of the interaction (e.g. the rating given to a movie).
(Perform the following steps after login as hduser (user for hadoop cluster))
2- I removed the last field of time stamp from the file (as it was not required for the current recommendation) using the following command and saved it in .csv file
cut --complement -f 4 -d, ratings.data >ratings.csv (To remove 4th column- timestamp from the file)
3- Now I created directory in hadoop file system to store the ratings file using the following command
hadoop fs -mkdir /mahout_data/
4-Now I copied the downloaded file of movie recommendation to hdfs using the following command
hadoop fs -put /home/hduser/mydata/ml-latest/ratings.csv /mahout_data/
5- go to the mahout directory cd /usr/local/mahout/bin/ and issue the following command to run :( The output file should be unique and JAVA_HOME should be properly set)
./mahout recommenditembased -s SIMILARITY_LOGLIKELIHOOD -i hdfs://localhost:9000/mahout_data/ratings.csv -o hdfs://localhost:9000/ratings_test/ --numRecommendations 25
-i hdfs://localhost:9000/mahout_data/ratings.csv - Denotes the input file
-o hdfs://localhost:9000/ratings_test/ -denotes the output file .
recommenditembased - Means we are creating itembased recommendation not user based recommendation, there is a difference between itembased and user based recommendation, a user based recommendation finds similar users , and see what they like, item based recommendation see what the user likes and find similar items, Mahout's item-based recommendation algorithm takes as input customer preferences by item and generates an output recommending similar items with a score indicating whether a customer will "like" the recommended item.
Choosing a similarity measure for use in a production environment is something that requires careful testing, evaluation and research. For our example purposes, here I used Mahout similarity classname called SIMILARITY_LOGLIKELIHOOD.
6- It will run for a couple of minutes you can see your output from web interface as well
7- You can check the output file it will contain two columns: the userID and an array of itemIDs and scores.
References :
http://mahout.apache.org/users/recommender/intro-itembased-hadoop.html
http://grouplens.org/datasets/movielens/
http://info.mapr.com/rs/mapr/images/PracticalMachineLearning.pdf
Mahout Installation
I wanted to try Data mining on big data, so I tried installing Mahout for that, here are the steps which I followed for successful installation of Mahout.
Prerequisite to install Mahout is - JDK, Maven and Hadoop cluster.
Prerequisite to install Mahout is - JDK, Maven and Hadoop cluster.
- sudo apt-get install maven
- Download the latest distribution of mahout from the site http://www.apache.org/dyn/closer.cgi/lucene/mahout/
- unzip and copy this to the desired location
- issue ls to check the packages inside it
- cd /usr/local/mahout/distribution
- sudo mvn install (Install maven 3.0.1 or above for Mahout .20 distribution else it will throw some error)
- Your installation is complete if you see the following screen
Tuesday, 7 July 2015
Resolve the installation problem of "rmongodb"
Being a new user of R as well as MongoDB, I wanted to make a mongoDB database connection with R but had to struggle before I could successfully establish the connection.
The version of R what I was using was 3.0.1, being old version whenever I was trying to install install.packages("rmongodb"), I was getting some error. Ultimately I had to upgrade the version of R using the following step and then I was able to install rmongodb package.
- sudo gedit /etc/apt/sources.list
- add the following line as I am using the version 14.04.2 (use the command lsb_release -c to see the name)
- deb http://cran.cnr.berkeley.edu/bin/linux/ubuntu trusty/
- gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
- sudo apt-get update
- sudo apt-get upgrade
Now use the following command to install mongodb package and connect to MongoDB
- install.packages("rmongodb")
- library(rmongodb)
To connect to local mongoD- mongo <- mongo.create()
Will keep on posting as I proceed towards using R, Shiny and MongoDB.
Thursday, 21 May 2015
Resolve the connection problem in mongoDB
I was trying to set up admin password in mongoDB by using admin database, but something went wrong and I started getting error " couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145exception: connect failed".
After giving some trials I rectified using the following commands :
Step 1: Remove lock file.
sudo rm /var/lib/mongodb/mongod.lock
Step 2: Repair mongodb.
sudo mongod --repair
Step 3: start mongodb.
sudo start mongodb
or
sudo service mongodb start
Step 4: Check status of mongodb.
sudo status mongodb
or
sudo service mongodb status
Step 5: Start mongo console.
mongo
Reference : http://stackoverflow.com/questions/12831939/couldnt-connect-to-server-127-0-0-127017/17793856#17793856
Tuesday, 10 March 2015
Install Microsoft Office on UBUNTU
I was trying to install Microsoft office 2K7 on my ubuntu 14.10, last time I installed on Ubuntu 12, simply by using wine windows program loader, but this time I was getting some error "newer windows needed". So I had no other option but to try something else .
I installed using playonlinux
sudo apt-get install playonlinux
sudo apt-get install winbind
start playonlinux
You will see the following screen
I installed using playonlinux
- Click on Install
- Choose Micro soft office version which you want to install
- Choose the location/ correct path
- Enter the licence key the installation will start
Thursday, 15 January 2015
Thursday, 25 December 2014
Heap size problem in weka
I was trying to load 51 MB of data on weka, while trying to load data I was getting heap size error, I did the simple trick by changing the
MEMORY="456m" in the file /usr/bin/weka which was 256 M previously.
Now it works!!Thursday, 20 November 2014
PostgreSQL user can not connect to server after changing password from PgAdmin
I changed postgres user password and after that I was stucked as I was not able to login, tried all the tricks by changing and re-setting password, but finally following solutions worked for me.
Error connecting to the server: FATAL: password authentication failed for user " "
FATAL: password authentication failed for user
The expiry date was set wrong by default and that needs to be changed,
- You can check these expiry dates with:
- SELECT usename,valuntil FROM pg_user;
and if they're wrong, reset them with:
- ALTER USER username VALID UNTIL 'infinity';
Now you are done, restart pgAdmin again and start working with the particular user and password.
Monday, 20 October 2014
Tuesday, 5 August 2014
Start with MongoDB
Ubuntu includes its own MongoDB packages.
Here are the few simple steps I followed to install mongodb on Ubuntu 14.
Here are the few simple steps I followed to install mongodb on Ubuntu 14.
- Import the public key : The Ubuntu package management tools (i.e. dpkg and apt) ensure package consistency and authenticity by requiring that distributors sign packages with GPG keys. Issue the following command to import the MongoDB public GPG Key: sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
- Reload the local package database: sudo apt-get update
- Install latest version of mongodb : sudo apt-get install mongodb-org
- Issue the command to start mongod: sudo service mongod start
This was the basic installation of Mongodb on ubuntu. Will keep on posting as I explore ....
References
References
- http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
Thursday, 22 May 2014
Wednesday, 21 May 2014
Configure IP of CentOS on virtual box
Once you installed CentOS on virtual box and it's not binded with any IP after issuing the command ifconfig.
Troubleshoot by issuing the following commands
Troubleshoot by issuing the following commands
ifdown eth0
ifup eth0
yum update
It will solve the problem.
Reference :
http://stackoverflow.com/questions/20816133/pycurl-error-6-couldnt-resolve-host-mirrorlist-centos-org
Wednesday, 2 April 2014
Troubleshoot- if alt-tab does not switch windows in UBUNTU
If alt-tab is not switching your screen, here is the solution .
Your problem with window manager is solved !! :)
Reference
http://askubuntu.com/questions/135685/alt-tab-does-not-switch
- Open the terminal and install compizconfig-settings-manager
- sudo apt-get install compizconfig-settings-manager
- Or search for
compizconfig-settings-managerin Software Center and install from there. - Then go to system tools > preferences > compizconfig
- Go at the very bottom where it says windows management , Put a checkmark in application switcher.
Your problem with window manager is solved !! :)
Reference
http://askubuntu.com/questions/135685/alt-tab-does-not-switch
Friday, 17 January 2014
Install rabbitvcs, SVN client simillar to tortoiseSVN on Windows
If you are a UBUNTU fan and looking for a svn client, rabbitvcs is a solution. Although I faced some issue while trying to install directly from repository, the nautilus was not showing on right click, after some trial, finally I could set up and installed from the source code by following the steps mentioned below
- sudo apt-get install python-nautilus python-configobj python-gtk2 python-glade2 python-svn python-dbus python-dulwich subversion meld gconf-edit
- sudo apt-get install python-nautilus python-configobj python-gtk2 python-glade2 python-svn python-dbus python-dulwich subversion meld
- wget http://rabbitvcs.googlecode.com/files/rabbitvcs-0.15.2.tar.bz2
- tar jxvf rabbitvcs-0.15.2.tar.bz2
- cd rabbitvcs-0.15.2/
- sudo python setup.py install --install-layout=deb
- cd clients/nautilus-3.0
- sudo cp RabbitVCS.py /usr/share/nautilus-python/extensions/
your are done now !!
Thursday, 26 September 2013
Cron Job to take a daily backup of Database - Postgres
Cron job are used to schedule commands to be executed periodically. You can setup commands or scripts, which will repeatedly run at a set time.
PATH TO POSTGRES/bin/pg_dump -U USERNAME DBNAME > PATH FOR BACKUP FOLDER/$(date '+%Y%m%d')-bkp.sql
0 17 * * * /PATH TO SCRIPT/scriptname.sh
- To edit your crontab file, type the following command at the UNIX / Linux shell prompt:
- Syntax of crontab (Field Description)
1 2 3 4 5 /path/to/command arg1 arg2
OR
1 2 3 4 5 /root/backup.sh
Where,
1: Minute (0-59)
2: Hours (0-23)
3: Day (0-31)
4: Month (0-12 [12 == December])
5: Day of the week(0-7 [7 or 0 == sunday])
/path/to/command - Script or command name to schedule
For example if you want to take a daily backup of your postgres database you can follow the following steps:
- Make entry for user USERNAME for database DATABASENAME as following in the file pg_hba.conf
- local DATABASENAME USERNAME trust
- * make sure that the user has select permission on the database.
- Write a script as following (vi scriptname.sh )
PATH TO POSTGRES/bin/pg_dump -U USERNAME DBNAME > PATH FOR BACKUP FOLDER/$(date '+%Y%m%d')-bkp.sql
- Create a cron job (Take a daily backup at 5 pm)
0 17 * * * /PATH TO SCRIPT/scriptname.sh
Tuesday, 24 September 2013
About Patents search - India
If you want to search the current patents filed in India, you can search using diffrent keywords on the following website
http://ipindiaservices.gov.in/patentsearch/search/index.aspx
It provides an exhaustive list of granted patents, published applications, Application status etc.
For other details you can also check
http://ipindia.nic.in/ipr/patent/patents.htm
Happy Patenting :)
http://ipindiaservices.gov.in/patentsearch/search/index.aspx
It provides an exhaustive list of granted patents, published applications, Application status etc.
For other details you can also check
http://ipindia.nic.in/ipr/patent/patents.htm
Happy Patenting :)
Thursday, 18 July 2013
Android on virtual box
If you want to run Android on you PC, you can install Android ISO in Virtual box. Once installed you can use Android enviornment on your machine install apps on it.
You need to install virtualbox first, which can be downloaded from https://www.virtualbox.org/wiki/Linux_Downloads (I tried it for UBUNTU 12.04)
Then download Android x-86 ISO from http://www.android-x86.org/download
I have tested it for the following settings
- Operating System: Linux – Linux 2.6
- Memory Size: 512 MB
- Hard Drive Size: 3 GB or More
Here is the snap shot of adding Android on virtual box.
After Installation is complete, reboot it go to devices > CD/DVD Devices remove disk from virtual drive so as to boot normally inplace of getting booted from ISO.
Wednesday, 26 June 2013
"xournal" an excellent pdf editor for UBUNTU
I was looking for a pdf editor in linux and found xournal. This is excellent pdf editor and easy to use.
you can install it by command
you can install it by command
- sudo apt-get install xournal
After it is installed open the pdf file which you want to edit, after you are done you can also export it into pdf.
Thursday, 13 June 2013
My experience - "Member of the panel discussion" at an International Conference
It was two day Conference on 12th-13th June 2013, at Sofitel Hotel, BKC on "The Global high on Cloud summit". It was related to cloud computing and we have recently started a research activity on "Mobile Cloud Computing", so our group head suggested to attend this and get an insight about how the industry is adopting cloud and what issues and challenges are they facing.
On the morning of 13th the session started and I entered into the conference room, looking for a comfortable place to get seated and settled, the moderator of panel discussion had already initiated a panel discussion and this time members were not preplanned, as a Prof of class he started calling up the delegates randomly, my colleague and me were sitting at the last seat, in the age,experience and designation we were the junior most barely having 6+ years of total experience and months of experience in the field of cloud computing, while mostly the CIO's, CTO's, Executive Directors, MD's and all exalted dignitaries that everyone aspire to get added with his name, were present from renowned Industries like IBM, Oracle, Hero Moto Corp, Maruti Suzuki India Ltd, Essar, Sify etc .
Suddenly the moderator asked me to join the panel at the stage "oh No" I whispered as it used to be in the MBA classes (no not me ;)), but this was not the class, I could not say NO I am not prepared or how can I join with u all luminaries, I had to suppress my nervousness and get up with confidence.
I got seated at the Dias with 5 other panelist the Director New India Insurance, Executive Director – IT, JM Financial Services Ltd etc etc. Fortunately they did not asked me to initiate the discussion and one of the member started with what challenges are they facing, which are the issues need to be addressed and issues related to continuity, disaster recovery, availability and security while two of them were talking I got some time to prepare my speech, had a deep breath and gear up. As soon as the second guy finished the moderator said "Let us have the privilege to hear something from the lady ".
After my introduction I started with "As you rightly said about the mobile reaching right up-to the "Paanwala", I would like to continue from there, the mobile has reached to masses but computer has not reached yet, we are looking forward for a solution that can bridge this gap and this is possible with the help of "Mobile Cloud Computing", So the vision is to allow everyone to use their mobile as if they are using a computer system and the solution should be intelligent enough to identify the kind of device, bandwidth, network connectivity etc and scalable enough to match the capacity of the cloud, dynamically call the services of the cloud on to the mobile and on the top of that it has to act like a plugin so as to become convenient for the user to install on his device, be it a low end mobile or high end tablets, For example I should be able to access all the features of Microsoft Power point from my mobile/ tablet itself in the way it works on desktop/laptop" . I brought a lot of questions to be answered and was counter argued by the other members as most of them were having their own cloud set up and they were trying to relate with the client server or thin client kind of architecture for mobile, then I had to convince them how different and usable it would be from client server, definitely a market need to be identified for this but every concept has some niche at the beginning.
Don't know what shape this idea is going to take in future but definitely I am happy that I was able to express my self, could remember something which my group head used to discuss and those exhaustive and difficult research paper from Berkley university had left some impression into my mind :)
On the morning of 13th the session started and I entered into the conference room, looking for a comfortable place to get seated and settled, the moderator of panel discussion had already initiated a panel discussion and this time members were not preplanned, as a Prof of class he started calling up the delegates randomly, my colleague and me were sitting at the last seat, in the age,experience and designation we were the junior most barely having 6+ years of total experience and months of experience in the field of cloud computing, while mostly the CIO's, CTO's, Executive Directors, MD's and all exalted dignitaries that everyone aspire to get added with his name, were present from renowned Industries like IBM, Oracle, Hero Moto Corp, Maruti Suzuki India Ltd, Essar, Sify etc .
Suddenly the moderator asked me to join the panel at the stage "oh No" I whispered as it used to be in the MBA classes (no not me ;)), but this was not the class, I could not say NO I am not prepared or how can I join with u all luminaries, I had to suppress my nervousness and get up with confidence.
I got seated at the Dias with 5 other panelist the Director New India Insurance, Executive Director – IT, JM Financial Services Ltd etc etc. Fortunately they did not asked me to initiate the discussion and one of the member started with what challenges are they facing, which are the issues need to be addressed and issues related to continuity, disaster recovery, availability and security while two of them were talking I got some time to prepare my speech, had a deep breath and gear up. As soon as the second guy finished the moderator said "Let us have the privilege to hear something from the lady ".
After my introduction I started with "As you rightly said about the mobile reaching right up-to the "Paanwala", I would like to continue from there, the mobile has reached to masses but computer has not reached yet, we are looking forward for a solution that can bridge this gap and this is possible with the help of "Mobile Cloud Computing", So the vision is to allow everyone to use their mobile as if they are using a computer system and the solution should be intelligent enough to identify the kind of device, bandwidth, network connectivity etc and scalable enough to match the capacity of the cloud, dynamically call the services of the cloud on to the mobile and on the top of that it has to act like a plugin so as to become convenient for the user to install on his device, be it a low end mobile or high end tablets, For example I should be able to access all the features of Microsoft Power point from my mobile/ tablet itself in the way it works on desktop/laptop" . I brought a lot of questions to be answered and was counter argued by the other members as most of them were having their own cloud set up and they were trying to relate with the client server or thin client kind of architecture for mobile, then I had to convince them how different and usable it would be from client server, definitely a market need to be identified for this but every concept has some niche at the beginning.
Don't know what shape this idea is going to take in future but definitely I am happy that I was able to express my self, could remember something which my group head used to discuss and those exhaustive and difficult research paper from Berkley university had left some impression into my mind :)
Subscribe to:
Posts (Atom)





































