John Berns

Installing Solr on Ubuntu Jaunty 9.04

Solr install success on Ubuntu Jaunty 9.04

I just installed Apache Solr on Ubuntu Jaunty 9.04. I am not a Java or Tomcat person, so it was all rather new and a confusing, there was no good install guide and I hit a few walls, so here are some pointers that should get you to a working install.

I am using the command-line install method, so it’s works for desktop and server editions. You can skip the apt-gets and use Synaptics Package Manager to install the packages if you prefer.

1) Install Java

sudo apt-get install sun-java6-bin sun-java6-demo sun-java6-jdk sun-java6-jre

I believe the default install of Jaunty installs openjdk java, so you will want to make sure that Sun Java is chosen:

sudo update-alternatives --config java

You will prompted to pick a version as the default:

Select the Sun Java version

Edit /etc/profile to sent the JAVA_HOME environment variable:

JAVA_HOME=/usr/lib/jvm/java-6-sun
export JAVA_HOME

2) Install Tomcat

sudo apt-get install tomcat5.5 libtomcat5.5-java tomcat5.5-admin tomcat5.5-webapps

Check that Tomcat is running:

sudo service tomcat5.5 status

Grant yourself Tomcat Manager and Admin permissions; edit /usr/share/tomcat5.5/conf/tomcat-users.xml and include the following:

<user username="myname" password="mypassword" roles="admin,manager"/>

3 Install Solr

sudo apt-get install solr-common solr-tomcat5.5 libxpp3-java

(I kept getting “java.lang.NoClassDefFoundError: org/xmlpull/v1/XmlPullParserException at java.lang.Class.forName0(Native Method)” in the Solr admin page; a little searching led me to find that the problems was that libxpp3-java was not installed. Why it’s not a required package, I don’t know!)

4) Restart Tomcat and Test Solr

sudo service tomcat5.5 restart

Now you can go to the Tomcat web admin page at http://localhost:8180/ and login with the credentials you supplied above.

Go to http://localhost:8180/solr/ and do a test query to confirm it’s working.

Of course you will get zero results, you have no content indexed yet.

That should be enough to get started!

Cal Henderson “Why I Hate Django” Djangocon 2008

I confess: I am a tech talk junkie. I often have some tech talk running on my desktop computer while I am working on my laptop, or watch one on my mobile phone while I am stuck in Bangkok traffic.

But this… this tech talk is a masterpiece. It’s geeky. It’s techy. It’s educational. It’s sarcastic. And, if you are a geek, it’s side-splitting funny. I laughed, I cried… I learned.

Warning: Ruby developers: you will be made the brunt of many jokes.

Highlight: Non-Ruby developers: Smug Ruby developers will be the brunt of many jokes.

Even if you are not into Django, it’s a great talk on scaling web apps and it’s f’ing funny too.

And Cal, if you are ever in Bangkok… I am buying, let’s go out for a drink: on me.

Tidy Up Your XML in Vim

You have some nice, compact XML, the kind without whitespace or newlines, like this:

Not easy to read. OK for computers to crunch–but if you want to get an idea of the document’s structure at a glance or do some editing–well, you might want it formatted a little nicer.

So let’s use Tidy to clean it up–all from the comfort of your Vim session…

(You could also use xmllint–your choice–depends on what you have installed and what you prefer.)

You can execute a shell command from Vim by typing:

:!

Then to tidy up your XML run tidy with the following options:

tidy -mi -xml -utf8

“-im” means indent and modify the original file; “-xml” means, you are dealing with XML and “-utf8″ is optional, it means “use UTF-8 for both input and output” so you’ll get readable characters instead of encodings if you have umlats or grave symbols.

You can see all of tidy’s formatting, encoding and other options by typing

tidy -h

So the whole command we type into Vim is:

:! tidy -mi -xml -utf8

You will drop to shell while the command executes:

Hit “return” when you are done and then Vim will ask if you want to reload the file:

And there you go, beautifully formatted HTML to read!

Enjoy !

When Social Networks Violate Trust: Yaari.com = Social Network Virus + Spam Engine

I just got taken for a ride and I am pissed.

I got an invite from an Indian friend that I respect, inviting me to join Yaari.com, what purports itself to be an Indian social networking site. He’s a very savvy guy and generally on the forefront of what’s hot and cool in Indian tech. So I thought “hey–must be something here, I should check it out.” After all, India tech is hot and I like to stay on top of what’s happening.

So I signed up. They asked if I wanted to check my GMail contact list for friends on Yaari.com. I thought: I have a lot of friends in India, most are serious web geeks, let’s see who’s here.

When I went to add my GMail password I paused for a moment. The conversation inside my head went something like this: “Give my email password to a site that I never heard of… that’s a bad idea! But then again I have checked my Gmail contacts against other sites buddy lists before and I never had a problem. Well, it was X that invited me and I trust him… so, I guess I will trust this site.”

And that usually works.

Usually.

But as it turns out, Yaari.com appears to be a site that is crafted to do nothing more than dupe unsuspecting people into giving up their friends email addresses; a social virus posing as a social network to harvest emails for spammers.

So, usually you can trust invites from your friends, but what happens if your friend was duped into trusting an untrustworthy site and you, in turn, trust what you think is his recommendation? Well…. it’s not really his recommendation and that’s not trustworthy, is it? The chain of trust was compromised somewhere along the line.

So, what happened was, that my entire Gmail contact list was spammed with invitations to join Yaari.com WITHOUT MY CONSENT. My trust had been compromised and they took advantage of that and then used my reputation to spam my friends.

And my friends signed up.

And the cycle repeated.

After this whole mess, I had a friend point out that if you Google Yaari you see that they are a scam. A bit late, I am sad to say.

I feel terrible that I fell for this. I feel worse that my trust was used to compromise other people.

Sadly, it seems to be a new enough scam that it works and works well. This is likely to be a new frontier for scammers. Expect to see more exploits like this springing up.

They could have done worse: they could have hijacked my Gmail account. That could have been a disaster. I guess I should consider myself (relatively) lucky that the worst that seems to have happened is that I suffered embarrassment and put my friends through some inconvenience. Not to say that’s a trivial matter, but I think how much I depend on my Gmail account and I shudder to think what could have happened.

But for all of you that read this the lesson to be learned is this: it’s not enough to trust the person that you get an invite from on a social network, you MUST VERIFY YOU CAN TRUST THE NETWORK IT WAS SENT OVER.

I should have seen that. My hesitation is clicking the button was the little voice inside my head telling me what I already knew. My scam sensor went off and I chose to ignore it.

There is a first time for everything…

Some Tips for Secure Social Netwoking

• Be more aware of the dynamics of trust on the Internet. You must think of trust not as a person or a name, but as a chain of events and each link in the chain must be trustworthy
• Never give out your passwords to any site–trusted or not. Yeah, we already know it–but social networks have conditioned us to behave otherwise.
• Google and email providers need to make a great effort to educate users not to give up their password for any reason and provide alternate ticket-based access to contacts that can be authorized–much like Yahoo is doing.
• Browsers should be flagging sites like Yaari.com as phishing scams. They are potentially as dangerous as banking scams; access to email passwords can divulge tremendous amounts of user data and can give scammers the ability to reset passwords–which is even better then the password itself

Google Chrome Changes the Game: The Browser as Platform

Google announced their new browser named “Chrome” today. Check out the blog post and, better yet, the Google Chrome comic book.

The platform wars just moved from the OS to the browser and Google took a commanding lead.

Rather than re-write everything, allow me the luxury of re-purposing my Tweets:

• Chrome is all open source. They have made the world their R&D department. Brilliant.
• Chrome just increased the importance of Javascript dramatically. They launched a platform where Javascript is the dominant language.
• Google just moved the development platform to the browser. The OS just took a backseat
• If MS was afraid that Google had them in the Search Engine Market, they should be shitting themselves about now about the browser market.
• Chrome will set the bar for what people will expect in a web browser.
• Google didn’t have to reinvent the OS; they just had to build the best browser that could run on any OS.
• Automated testing against google’s vast index of web pages is a stroke of brilliance for stability testing a browser.
• Chrome should solve the biggest annoyance I have with my browser: better memory management so I don’t have to restart my broswer 3X daily.
• Chrome is privacy-oriented. That’s a good thing.
• The name is very toungue-in-cheek: chome is a refernece to the UI for an application, Google wants Chrome to be the UI for the user’s web experience.
• This is the biggest architectural leap in computing in a long time.

Google has just moved to the forefront of the Browser wars and will force the competition to keep up. They have the brand recognition, industry leverage and exposure to get their browser installed in a LOT of computers.

Oh–an it’s also built on the same engine that the Android mobile browser will run on.

Mark my words: Google just changed the game on the web. Chrome is the lever they use to move the world.

Quick Video on Building Flex Apps

I just saw this brief video on how to build a Flex app on the Adobe website. It’s pretty impressive!

How to Dis-Organize a Barcamp

Barcamp on the brain. I have been doing nothing but barcamp stuff this week. I labored all day to think of a great topic, but I was always being interrupted by barcamp business.

So I finally thought: if I can’t come up with a good topic, why not present about how to (dis)organize a barcamp at Barcamp?

Mind you, this is my thoughts about barcamp organization. Other barcamps are organized differently. No two barcamps are alike. No two barcamp organizers have the same style. Your mileage may vary. Drink plenty of fluids. Use caution when operating heavy machinery.

View my presentation on SlideShare

How to Mount a VMWare (Linux) Partition in Ubuntu (Hardy 8.04)

I recently had a problem starting a VMWare VM (the VM ran CentOS) and I needed to get some data off the virtual machine. Since the VM would no longer boot, I decided to try to mount the partition under Linux and copy the data off. It was a little trickier than I had anticpated so I tought I would share my solution.

vmware-mount.pl Doesn’t Get You All The Way There

vmware-mount.pl does just what is says it will do: it mounts VMWare partitions on your computer, IF the target computer can mount the partition type. The problem with Ubuntu (Hardy 8.04) was that it would not mount an LVM partition.

Look at vmware-mount.pl for moreon how to use this command.

But vmware-mount.pl was still useful in helping me find out the partition type and creating a block device from the .vmdk file that I could use other tools to mount.

Change to the directory where you have the virtual machine files. There will be one (often more) .vmdk files in that directory and these files are the virtual drives that hold the data for your VM.

cd /path/to/your/vm/

Now let’s see what is inside these files; we want to know what partitions there are so we can figure out which one has the data we are searching for.

sudo vmware-mount.pl -p name-of-the-vmware-virtual-drive.vmdk

That will output something like:


--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter
Version: 1.0 build-80004
Copyright 1998 VMware, Inc. All rights reserved. -- VMware Confidential
--------------------------------------------

Nr Start Size Type Id Sytem
-- ---------- ---------- ---- -- ------------------------
1 63 208782 BIOS 83 Linux
2 208845 16563015 BIOS 8E Unknown


So we have two partitions in this vmware virtual drive. Just by looking at it I can tell that the big partition, partition number 2 is the one that has all my data. But notice the “Id” is “8E” and the “Sytem” is “Unknown.” Well “8E” is the ID for LVM file systems and that’s a problem; let’s try and mount it and see what happens:

Create a mount point:

sudo mkdir /mnt/vmdk

Now let’s try to mount it.

The command syntax for mounting a .vmdk partition is

sudo vmware-mount.pl name-of-the-vmware-virtual-drive.vmdk [partition number] /mnt/point

So if we want to mount partition 2 on /mnt/vmdk we’ll try:

sudo vmware-mount.pl name-of-the-vmware-virtual-drive.vmdk 2 /mnt/vmdk

And the results we get are something like this:


jb@jb-ubuntu:~/Desktop/fuxored-disk$ sudo vmware-mount.pl CentOS\ 4.4-000007-cl1.vmdk 2 /mnt/cent
[sudo] password for jb:

Unable to retrieve the filesystem of the partition (the partition type is BIOS
and the partition Id is 8E). Please file an incident with VMware at
http://www.vmware.com/forms/Incident_Login.cfm by copying this error message.
jb@jb-ubuntu:~/Desktop/fuxored-disk$


It created a block device that can be read, but the partition didn’t mount as we can see by this message:


Client: The partition is now mapped on the /dev/nb0 Network Block Device.
mount: unknown filesystem type 'LVM2_member'


So we have a block device of type “LVM2_member.” Let’s try to find a way to mount that!

Luckily, Kurt Wall did a nice tutorial to how to mount a fedora logical volume in Ubuntu that solves our problem.

If you type in the commands he has listed on that page, one after the other, you will have a mounted partition and problem solved!

Fixing Compiz Fusion on ATI Radeon Mobility 9800 Under Ubuntu Hardy (8.04)

**** OK, this works–but it’s not feasible. I found xserver-xgl to be VERY slow and had screen rendering issues. ****

Take this for what it’s worth–and that does not seem to be much.

When I originally installed Ubuntu Hardy on my HP Pavillion zt3000 laptop Compiz worked perfectly. Much to my dismay, after I re-installed, Compiz Fusion would not re-enable, even though I had the restricted drivers installed and enabled. (Selecting System -> Preferences -> Appearance -> Visual Effects -> Extra returned a “Desktop effects could not be enabled” error.)

I tried running “compiz” from the command line and one line of the output caught my eye:

checking for xgl not present

That gave me enough search terms to search on Google and hit paydirt on UbuntuForums.

The solution was simple: Go to System -> Administration -> Synaptics Package Manager and install “xserver-xgl.”

After I logged back out and back in, I enabled Compiz ( System -> Preferences -> Appearance -> Visual Effects -> Extra) and everything worked.

Tackling Scalability Issues

If you a techie kind of person and if you have ever had ambitions to build a large-scale website, you have thought to yourself “how can I make this scale really big?”

Well, the guys at YouTube have been through the drill. Here is a great video by Cuong Do, one of the founders of YouTube and the guy in charge of making YouTube scale, about how they managed to scale YouTube.

Yeah, there are some architectural decisions you can make up-front that will make scaling a website easier, but, after watching this video about the scalability challenges YouTube faced, I realized that:

1. Scalability is all about managing bottlenecks. You watch for them forming, you deal with them before they become a problem.
2. Improving performance in parts of the system that are not bottlenecks does very little good.
3. The problems you get are almost never the problems you plan for.
4. Plan, but don’t over plan. See what happens and adapt.

I have become a huge fan of the Google Tech Talk videos and tech videos in general on Google Video and YouTube. I have started to download them and load them on to my iPhone (how to do that is a future blog post itself) and I find myself watching them whenever I am in the subway, in a taxi or have a few minutes to kill. They are a great way to get a healthy dose of new knowledge fast and if you have them on the iPhone, you can get that dose in those random periods of downtime that would otherwise be wasted time.