DataBasin and DataBasinKit 0.9 released

After more than a year of work, finally a new release of DataBasin and its associated framework. This version features a Framework that can be used in threads and the application itself is capable of having concurrent connection classes to SalesForce.com.

On Windows

On OpenBSD

Countless bug fixes (thanks to the bug reports of my colleagues Diego, Moustapha, Matteo + Matteo).

Major new features are:

• threadable DataBasinKit framework
• concurrent, interruptible operations (e.g. select vs. update)
• handle multiple errors as a result of update
• filter new lines when writing CSVs
• countless bugfixes, especially in select-identify corner cases

DataBasinKit allows you native connection to SalesForce.com, allowing your application to integrate SOQL queries (SELECT, UPDATE similar operations), be it on Mac (Cocoa) or NetBSD, FreeBSD, OpenBSD, Linux or Solaris as well as Windows (MinGW).
DataBasin itself allows you to perform standard operations in a quick and agile interface: Extract accounts on Linux without the need of Java. Use the unique select-identify feature.

I am proud that Free Software can connect from a Free Software Operating System to a proprietary system and bridge the two worlds, enabling to do administrative work without being constrained to Java on Windows (read: DataLoader). Thanks to the many developers who continue supporting me in the development and keep these fine Operating Systems and tools alive.

DataBasin 0.8

DataBasin 0.8 is out!

After several months of development and testing, many news:

• DataBasin is now divided in its DataBasinKit framework which is LGPL'd and the application itself
• (CSV writing) Support of empty fields in empty semi-joined objects through query parsing
• (CSV writing) Support for writing fields in exact order as in user query, trough query parsing
• Select Identify supports LIMIT
• Customizable CSV file quoting and separator
• Support for COUNT and aggregate queries
• Object Inspector supports selection of values in cells

The most important news is the DataBasinKit separation.
The most important feature instead is DataBasin ability to parse the SOQL query and thus rearrange the output fields in CSV files not as Salesforce returns them but as the user requested them. The same feature allows related objects (. notation) to be null and retain the correct columns in the CSV file.

DataBasin: more powerful CSV writing for Select and Select-Identify

DataBasin, the clean-room impelmentation of a data extraction tool for SalesForce.com available for GNUstep and Mac just made a big leap forward!

If you have ever done a SELECT in salesforce.com, you might have noticed that the results are not ordered and that semi-joined fields ("." notation) are handled strange because a whole object is returned.

DataBasin extracts the  columns of the CSV file by checking the first row of the dataset and recursing on the field names of salesforce's response. The effects in DataBasin are the following:

• The field order is not preserved
• If a sub-object is queried, all those fields are grouped together in that object
• If more than one field is queried on a sub-object and on certain record this object is missing, the number of columns between records is inconsistent: salesforce.com doesn't return the whole object and DataBasin iterates record after record in the sub-object, but if it it is totally missing, only one empty column will be written, not as many as needed
• Capitalization of the column names doesn't get preserved (compared to the above issues, something really minor)

I added a new option available in write out fields in the order of the query.

This feature implies a more complex under the hood. For each record gotten in the response, the field names are reconstructed as before (e.g. by recursing inside sub-objects) and put in a Dictionary.

The original SOQL query is parsed and the field names extracted, then these names are used as keys for each record dictionary and written out.

• Order preserved, also with sub-objects
• Fields preserved and empty values are written even if whole sub-objects are missing
• Case is preserved

This approach looks fine and comes with a relatively small performance penalty, however it is delicate because during parsing of the query, many naming subtleties of Salesforce.com needs to be inferred, that is, DataBasin tries to infer how the name in the Response will be:

• field aliases
• aggregate names with no alias, which are called progressively Expr0, Expr1
• idiosyncrasies with aggregate functions. A field in a sub-objects is called Object__c.Field__c when in a regular query, but when used in an aggregate query, only Field__c is used, disregarding the join-depth
• COUNT() vs.COUNT(Id)

I hope I have not missed any! in any case, enjoy!

This feature is available now both in Select and Select-Identify, since it is a problem of the CSV Write backend and enhances DataBasin operation usability greatly, bringing it on par with DataLoader.
Since DataLoader performces similar transformation and sometimes produces wrong results, I left this feature as optional with the flag "Write fields in query order".

DataBasin development: logger and semi-joins

I am working on two long-needed features of DataBasin, the open source application to extract and inspect data from salesforce.com

The first feature is rather simple: a log window which shows some informative or debug output usually thrown to the console. While on standard Unix we always have the terminal emulator open and all debugging is very convenient there, on mac and especially on windows it is inconvenient: DataBasin logged every output as an Event in the windows log. Quite painful.

A much more complex, interesting and long awaited feature is that DataBasin now supports semi-joins! In other words, one can reference a field of a referenced object with the dot notation (like: Case.Account.Name) exactly as in the Salesforce.com Data Loader. The objects are recursively scanned and flattened so that they can be written to the CSV file. The column names are generated through the descent of the object tree.
This feature required a much more flexible approach in the CVS writer core. This means that some regression bugs could appear... hopefully not.

This feature applies to all queries, including the Select Identify query tool.

As the parser was being rewritten and the supported API bumped to 25, it is now improved to handle Aggregate Objects, thus one can now issue the following SOQL query:

select count(id), status from case group by status

and get the expected result:

"Status","expr0"
"New","16"
"Closed","31"

I hope these features make DataBasin much more useful!