1. Is the software the company needs already created and will sufficiently do the tasks they need at an affordable cost to the client/company?
2. If there’s no software that’s a 100% fit either with affordability or satisfaction of the functionality needed, can one be purchased and perhaps customized?
3. If the customization of existing software (either open source or purchasing a developer’s license from a pre-built software company) isn’t available, can it be coded by taking pieces of opensource code and either pairing it together with more open source code or with custom code?
4. If all else fails, break out the checkbook because it’s going to cost you.

Now this is extremely simplified, but it is a basic 4 steps of why you should or shouldn’t be developing your own software for your company’s needs.  There’s other factors not mentioned here such as, possible retail licensing of your software, partnership revenues, etc.  but you get the idea.

So back to the issue…data mining software.  It’s a nightmare to code and quite honestly there’s not a whole lot of good ones out there when it comes right down to it.  Even as a coder, I find the “coding” versions of some of these systems way too complicated than they need to be, and hence not worth my or my clients’ time.

I’ve compiled some thoughts on the following systems, please feel free to comment on your own experiences or if you’ve tried something better….I’m all ears.

Selenium HQ

Not bad software and as a matter of fact if you like Firefox plugins, this one may work for you.  The free price tag on it is equally appealing for certain.  I actually used this software before in its earlier stages, and not much has changed except they have added a full coding module to the system so you can basically run it from a number of coding systems such as asp.NET, PHP, Java, etc. which makes it very appealing to the coder in me, but honestly when it comes to data mining, who wants to spend that amount of time coding something that most likely will change the next time a site you are gathering data from, changes it’s layout or format.  Sorry Selenium….not interested.

Mozenda

Now this company has the right idea when it comes to revenue generating models that work.  This site has both a web client and desktop client (Windows Only right now…grrr) but has a rather snazzy looking interface and really does a heck of a job with giving you free pages (that’s how it tracks your usage/cost) just by going through their pretty comprehensive, but not time consuming tutorials.  I really like the way the data repeaters know that you are looking for results data on this page, but when you click on the title of the product or the image, you can then tell it that you are now going to a product detail page and to continue to grab data for that same row.  It’s pretty intuitive, quick and the free version which gives you about 5,000 pages to churn through, it’s not bad.

Drawbacks are of course the cost.  It can get really pricey for companies who need a lot of data on a daily basis, and their engine for the desktop ran a bit slow at times which was a bit annoying.  Overall, I think this product is perfect for a company who has a small to medium sized budget for data acquisition and has a computer savvy person who would be driving the Mozenda engine, but definitely is more advanced than just your basic MS Office computing user.

Needlebase

This is the last product for this blog post, but it’s the one I’m currently exploring as we speak.  When I first came to their site, I said….ugh….this looks terrible.  However, after doing some research I came to find that it was recently acquired by Google and was made free (for the time being) for any google/gmail/chrome/blah blah blah….registered user.  The interface and data modeling is very difficult to get a grasp on if you’re not familiar with nodes and types and how data models generally come together.  However, if you’re like me where you understand this type of computing, however need a quick and easy way to grab sources of data through a gui interface that allows you to select the various pieces of data on the actual page you are looking to grab data from (Mozenda also does this btw as does Selenium’s FF plugin), but do it a bit more efficiently than the others…you may want to give Needlebase a try.

Ok back to Needlepoint….I mean Needlebasing….I hope this helps a few of you in your searches for robust, easy to use Data Mining software.

-Doug Justice is the CEO & Chief Superhero of Justice Solutions LLC.  He is an expert in over 10 different programming languages and development methodologies and performs analysis of company development teams, new business concepts, and web marketing potentials.

Well, we’ve been doing this for quite a while and I felt as though it was time to let other folks know about it since there are a lot of opportunities for businesses to save on some soft employee costs, reduce overhead, and increase productivity and data.

A Quick Word About Why You Should Hire a Web Development/Design Company
This is where web development companies are worth their weight in gold.  Yes, I know there’s plenty of cookie cutter do it yourself websites out there, and they are great and very affordable.  But when you have an idea that is unique, or a situation that is unique, you really don’t want what is out there available for everyone else.  I mean, honestly…if Dell used a cookie cutter build your own computer plugin, would they really be all that different from the other 100 web stores that offer the same thing at approximately the same cost?  No, you’d probably go to the place that was unique, professional looking, and looked as though they actually spent some money building a better mousetrap.  Ok…so there’s my two cents why you should actually use us….on to the good stuff.

Why You Would Need an Expert in Data Mining and Screen Scraping
Let’s say you are a company in Arizona who has an idea to put together a site that puts together all of the latest reduction in homes for sale prices.  Now there’s a bunch of websites out there that you’ve contacted and said you’re going to promote this site and obviously push the business to the various real estate agents involved.  Now comes the fun….gathering this data on a daily basis and putting it all into your application.  That’s where Justice Solutions comes in.

Grab Data, Massage Data, Import Data, Display Data
So now you hire Justice Solutions to do this seemingly impossible task.  We now take the web addresses of these various sites and create an application that will go out to each one, identify the way each site displays its data, grab it (even if it’s on multiple pages), and then perform some massaging scripts that will convert any text data into price data, etc.  We then import that data into your master database which will then be used to display the data to your website users in a unique format specific to your website.

Why It’s So Difficult
The reason this is usually such a daunting task is because if you’re not used to it, trying to identify data patterns and other things that go into dealing with data from multiple sources can be quite overwhelming.  Also, you have to be able to automate all of it so it happens quickly and without much user intervention.

We’ll Toot Today
So if you or a person you know has an idea like this, or an existing business that would benefit from this type of application, please refer them to us.  We gladly pay referral fees and would be happy to help in easing the fears of this commonly thought of, but rarely successful, business model.  Contact info@justicesolutionsllc.com for more information.

Until next time…happy coding…and mining….

Doug.