Importing CSV file data into sqlite3

by Mandar Vaze on July 13, 2009
in Code, Hack, Linux, Open Source, tips

The :en:SQLite logo as of 2007-12-15
Image via Wikipedia

I was trying to import the data from CSV file into sqlite3 database. Ideally this should be very simple task, with following the steps given in the sqlite tutorial. It is a matter of calling the sqlite command with separator argument, followed by an import operation, as listed below.

sqlite3 test.db  "create table t1 (t1key INTEGER PRIMARY KEY,data TEXT);"
sqlite3 -separator , test.db ".import some.csv t1"

Except that main attribute of my CSV file was that it could contain single records with embedded comma. I was hoping that sqlite3 would be smart enough to detect that the fields were enclosed within double quotes and then separate by comma. But I soon realized that only a code specifically dealing with CSV would know about this.  As we can see in the example above, the import is a generic code and as a user I listed comma as a separator.

My Data looked something like this :

"1","data1"
"2","data2,data3"

So like any *nix geek would do, I tried providing double quote and comma as a separator. To my surprise it worked very well. I though separator would take only single character, and I had provided two (three?). Anyway, important thing to remember is to escape the single quote with a backslash (I didn’t try it without the backslash, may be that would work too)

So here is the syntax that worked :

sqlite3 -separator \", test.db ".import mydata.csv mytbl"

Update : Turns out SQLite Manager is much better solution after all.  It is an Extension for Firefox and other apps to manage any sqlite database. Not only it took care of above situation, it also handled empty cells as well where the command line failed with following error message :

line 4: expected 3 columns of data but found 2

Data with missing cells : Notice two successive commas :

"1","data1","data2"
"2","data3,data4","data5"
"3",,"data6"
Reblog this post [with Zemanta]

Comments

One Response to “Importing CSV file data into sqlite3”
  1. Jillian says:

    I know this is old as hell but instead of a comma-delimited csv file it’s better to save your data as a tab-delimited text file and import it with:

    .separator ‘ ‘

    .import links.txt links

    That way you can have commas in your data and since excel spreadsheets don’t allow tabs they’re great for preparing your data and then saving as tab-delimited text files. GL!

Share Your Thoughts