Posted on

5 tips on how to keep your data clean

clean-up
We spend so much time and investment in data gathering which, despite our best efforts, corrodes over time.

To retain data quality, you should make sure you have the correct automated and manual processes and procedures in place.

Here are five tips that we have for you to consider:

  1. Ensure that a clear set of data standard instructions are defined and exist on how data is to be stored e.g. postcode to be stored without whitespace
  2. Encourage your staff to check and update a customer’s data each time they are in contact with them
  3. Validate all data entry points before saving new data or when updating existing data
  4. Invite your customers to maintain their own data via a secure web portal
  5. To reduce data entry errors, use drop-down lists in forms as much as possible

The trick is to make sure that all your processes and procedures stay current and try and automate as much as possible.

Posted on

Data migration experts

We would like to make it official and state that we, the DataWand Wizards, feel for data migration experts. Their job isn’t made easy given the volumes of data that we now store and the lack of standards with easy import/export routines in software applications and we’re talking about those applications with a database at their core such as Oracle, SAP, Sage, Quickbooks and Microsoft Access.

There are a number of data migration experts and consultants out there and we would like to make their jobs easier by saving them time and headache by reducing the number of bespoke scripts and manual re-work that’s undertaken. We hope that DataWand becomes one of those tools (dare we say wand?) in their toolkit.

Take the example of when IBM won the London Congestion Charging contract from Capita in 2008 and started operating it in 2009. We wondered if their data migration experts could have used DataWand to ensure that the data was as good as possible before importing it into their new system? But,  IBM do have a vast software library (like Softek) and resources that they can call upon. Who knows, maybe DataWand will be used next time around.

Anyway, we’re looking forward to hearing from data migration exports and consultants so we can lend you a wand. 🙂

Posted on

What is ETL?

It’s a geeky term that stands for Extract, Transform and Load and used in order to manage databases. Apart from us mentioning ETL on this page, we’re quite keen to avoid this jargon as we like the term data transformation better.

ETL is really all about using software data tools to get data (a dataset) from one or more databases, manipulating it using rules or actions e.g. reformat, merge, split, de-dupe the data and then uploading the transformed results to another database. You could of course update the original database if you needed to.

ETL’s non-geeky equivalent terms include data migration, data conversion, and data mapping, which to some readers could be seen as still geeky!

ETL comes in handy as it eliminates the need to do repetitive data transformation task manually. Instead of spending hours manually transforming the data the alternative options are either to write some bespoke software coding or use a data tool which are available as a ready-to-use software package.
These data tools make the job of transforming data easier and besides saving time, these data tools will support different database formats (Oracle, SAP, mySQL) and different data types (CSV, XML, flat, fixed).

These data tools are expensive but powerful applications which you install on your local machine. Both the source and destination dataset structures will be shown to the user and the graphical display will show lines mapping the data across to the different structure, hence the term data mapping.

Data mappings can be integral to your processes as some data tools will also allow a data mapping to be scheduled so that the rules can run on datasets based on a schedule which will automatically extract live data, transform it using the data mapping rules and making sure that it gets to its destination (the transformed dataset).

All-in-all a very simple concept and terminology for what is a really complex task as data comes in all shapes and sizes.

And just in case you’re wondering… Yes, DataWand would be classed as an ETL tool though its different from traditional ETL tools as the DataWand Wizard tool is available online and it manipulates the data based on the actions you’ve chosen and then displays the transformed data results immediately which you can then email to yourself and upload it to a database if you wanted to.

Posted on

Where art thou data conversation?

We’ve spent this afternoon searching Twitter to try and join in conversations ‘all about data’ .

But we’re having a difficult time finding that right conversation to join in. Everyone’s talking about big data (the latest buzz word these days) and business intelligence (BI) but we don’t specialise in either of those. Nor are we hard core data cleansers (comparing name, address and contact details against a master all-knowing database), at least not yet.

We are sure that there’s a market niche for DataWand out there because if your data isn’t quite in the right structure or needs to be extracted, merged, split, reformatted, enriched or de-duplicated, that’s when we can wave our magic wands and help out. I think its time to dust off our crystal ball and take a deeper look to find those companies and individuals that are tired of spending hours manually cleaning up data and want to become wizards.

In the meantime, our Twitter search continues… and we hope to conjure out of thin air, the appearance of #DataWand on Twitter.