News feeder

March 18, 2012

A couple of weeks ago I found out about a search function for linux calledSphinxsearch that supported MySQL (and XML piping), needless to say I got interested on how it worked with large amounts of data. So the first thing I did was to generate a database filled with random data, and then indexed the database, the test didn’t really work out because I couldn’t tell if the random garbage was what I actually searched for or not.

Instead I built an XML/RSS parses, and went out to look for feeds, which I then parsed through and saved every 5 minutes with crontab.

I built an Ajax powered front end for the application which outputted the latest 100 news as well as support two types of searches, one which only takes out results from the latest 24 hours and one for global search.

I started to use my news feeder for more than just testing and built a mobile version as well.

Sadly, I can’t show you this awesomeness, because I’m hosting it on my home internet connection, and I don’t want the connection to be indexed by spider bots. But instead I’ll give you some screenshots.