Friday, April 6, 2012

Windows Phone Local Database tip: Exploring DELETE performance and a “Bug Alert”

In the previous instalments in this series of Windows Phone Local Database tips, I have been exploring SELECT, INSERT, UPDATE, CREATE DATABASE and Encryption. Now the time has come to have a closer look at DELETE, and investigate how they can be optimized (if possible). I will look at deleting multiple rows in a single call to SubmitChanges.

To test this, I will be using the code below, and as usual use the Chinook sample database.

//New database from embedded resource
using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
if (db.DatabaseExists())
db.DeleteDatabase();

db.CreateIfNotExists();
}

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{

try
{
//db.LogDebug = true;

var list = db.InvoiceLine.Take(100);

db.InvoiceLine.DeleteAllOnSubmit(list);

System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
sw.Start();

db.SubmitChanges();

sw.Stop();
System.Diagnostics.Debug.WriteLine(sw.ElapsedMilliseconds);
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.ToString());
}
}



Running this code using the InvoiceLines table from the Chinook sample database will result in 100 calls to a DELETE statement like this:


DELETE FROM [InvoiceLine] WHERE ([InvoiceLineId] = @p0) AND ([InvoiceId] = @p1) AND ([TrackId] = @p2) AND ([UnitPrice] = @p3) AND ([Quantity] = @p4)
-- @p0: Input Int32 (Size = 4; Prec = 0; Scale = 0) [100]
-- @p1: Input Int32 (Size = 4; Prec = 0; Scale = 0) [19]
-- @p2: Input Int32 (Size = 4; Prec = 0; Scale = 0) [581]
-- @p3: Input Decimal (Size = 19; Prec = 10; Scale = 2) [0.99]
-- @p4: Input Int32 (Size = 4; Prec = 0; Scale = 0) [1]


Notice that all columns are included in the WHERE statement, to check for concurrency conflicts (you can argue if this is really required when deleting). The 100 deletes runs in 320 ms on the emulator.


Now let us try to add a rowversion column to the InvoiceLines table, like we did for the UPDATE testing, and see if this has any effect.


You can also use the new feature in the SQL Server Compact Toolbox, which will allow you to add rowversion columns to all your tables, for the benefit of UPDATE and maybe also DELETE statements.


image


Let us try to run the same test code again.


Now the SQL statement looks like this, indicating that the query processor is bypassed, unlike in the statements above.


-- CURSOR BASED INDEX DELETE [InvoiceLine].[PK_InvoiceLine] (
--     InvoiceLineId: [1]
--    )
-- EQUALITY CHECK [versioncolumn], [System.Byte[]] = [System.Byte[]]


The time to perform the 100 DELETE statements is now down from 320 ms to under 70 ms, a significant improvement.


BUG ALERT!


If you attempt to take advantage of rowversion columns in an existing database, make sure that the table in question does not have more indexes that cover the primary key columns. If you have that, your SubmitChanges statement will “bomb out” your app.


image


For example the Album table in Chinook has a PK_Album index and PFK_Album index, that both are indexes on the AlbumId column. DROP the PFK_Album index in this case (you can use the new Index context menu option in the Toolbox to do that).


Hope you found this useful.

Thursday, March 29, 2012

SQL Server Compact Toolbox hits 100.000 downloads–new release planned

My SQL Server Compact Toolbox add-in for Visual Studio 2010 and 11 beta has now had more than 100.000 downloads. (Actually, more than 105.000, but who’s counting.)

Thanks to all the users of this tool, and your continous feedback. I always think that the most recent release will be the last, but I am already perparing for the next relase, which will add features like:

- Provision a 3.5 database for Sync Framework
- Ability to Migrate to a SQL Server/Express/LocalDB database in a single click (like what you can do with WebMatrix, but from within Visual Studio)
- Support for SQL Server Compact 3.5 connections with Visual Studio 11 beta (to support Windows Phone users and others)
- Option to add a ConnectionStringBuilder class to Windows Phone projects
- When exporting table data, the tables are now ordered topologically
- The Server based DGML files now respect schemas
- Server date and datetime2 columns are now converted to datetime

And in connection with the MVP Summit I got the opportunity to appear on the Visual Studio Toolbox show on Channel 9, where I demo some of the Toolbox features. Maybe you will discover features you did not know about during the demo.

Thursday, March 8, 2012

Windows Phone Local Database tip: Working with encrypted database files

SQL Server Compact on Windows Phone supports encryption of the database file. By specifying a password on the ConnectionString during encryption and subsequent openings, the contents of the database file will be encrypted.
Beware that the encryption will degrade database performance, due to the extra processing required to decrypt and encrypt data.
Also keep in mind when using encryption, that unless you protect your connection string (the password), anyone that knows it can open the database file.

Some Phone developers (see this forum post, for example) have encountered issues when moving databases from device to desktop and vice versa, when encryption is involved. In this blog post I will try to explore what can and cannot be done in this respect.

From the root context menu in the SQL Server Compact Toolbox, select “Add 3.5 connection” and select Create:

image

Specify a password of “zyx” and click OK. Now we have a connection to a desktop encrypted database file.

Open the SQL Editor, and create a basic table:

CREATE Table Test
(
Id int PRIMARY KEY,
Data nvarchar(10)
);

Now create a Windows Phone 7.1 Application, and add the database file from above as Content:

image

Right click the database connection in the SQL Server Compact Toolbox, and select “Add Windows Phone DataContext…” and click OK, the WP7EncryptContext.cs file has now been added to the project.

Add a reference to System.Data.Linq, and compile the project.

Add a button called “Connect”, and add a click event handler, with this code:

            using (WP7EncryptContext db = new WP7EncryptContext(WP7EncryptContext.ConnectionStringReadOnly + ";Password=zyx"))
{
var list = db.Test.ToList();
}



Now run the app in the emulator by pressing F5 (debug):


image


Obviously the desktop encryption is not compatible with the device database engine!


Now remove the database file from the project, and create the database file in isolated storage – change the click event handler code:

using (WP7EncryptContext db = new WP7EncryptContext(WP7EncryptContext.ConnectionString + ";Password=zyx"))
{
db.CreateIfNotExists();
var list = db.Test.ToList();
var test = new Test();
test.Id = 2;
test.Data = "ErikEJ";
db.Test.InsertOnSubmit(test);
db.SubmitChanges();

}



Now the database will be created in Isolated Storage on the device. Run with F5 again, and no errors will occur. And we have “pre-populated” the database with a single row. Let’s try to move this database file to the desktop, I use Windows Phone Power Tools from CodePlex to do this:


image 


When you add a connection to the file form the device, and click “Test Connection”, you will get this error message:


image


We already know that some version 4.0 features have been made available in the 3.5 Windows Phone database engine, like support for OFFSET/FETCH to allow paging. Has the enhanced version 4.0 encryption also been move to the device?


In the Toolbox, select “Add 4.0 Connection” and try to connect to the database from the device:


image


(Remember to specify the password!)


image
Reply Yes! And now we can browse and inspect both schema and data in the file – keep in mind that this file cannot be moved back to the device.


image


Now, GET the file again from the device via the Power Tools, and add the new copy to the project again, also as Content (as above, but the database file comes “pre-populated” from the device.)


Again, change the button click event handler as follows:

using (WP7EncryptContext db = new WP7EncryptContext(WP7EncryptContext.ConnectionStringReadOnly + ";Password=zyx"))
{
var list = db.Test.ToList();
if (list.Count > 0)
MessageBox.Show("It works " + list[0].Data);

}



Conclusions:


You CANNOT:


- Use a desktop encrypted database on the device
- Open a device encrypted database on the desktop


You CAN:


- Prepopulate an encrypted database (but only on a device)
- Convert a device encrypted database to 4.0 format on the desktop, and browse it.


Credits: Thanks to Marc-Anthon Flohr for prompting me to write this blog post. His overview is nice, although his conclusions are slightly different.

Friday, March 2, 2012

SQL Server Compact 4.0 SP1 CTP1 available

(Nice short name!) SQL Server Compact 4.0 SP1 CTP1 is now available for download: http://www.microsoft.com/download/en/details.aspx?id=29037 – is is also installed with Visual Studio 11 Ultimate beta. Installing this .MSI will replace your current SQL Server Compact 4.0 binaries in Program Files, and change the version to 4.0.8854.1.

No blog post or KB article yet, I will update this post when they become available. I the meantime, I have skimmed though the code in the ADO.NET provider, which reveals the following changes (all is just guesses, and it is beta software, of course):

New connection string keyword: “File Access Retry Timeout” – will reattempt connection open from 0 – 30 seconds, with a default value of 0.

Improved local C++ runtime verification.

SqlCeConnection.GetSchema bug fixes

SqlCeParameter.Add(String parameterName, Object value) has been deprecated.  Use AddWithValue(String parameterName, Object value) instead.

According to this thread, this bug has been fixed: Getting exception "The data area passed to a system call is too small." in SQL Server Compact 4

Monday, February 27, 2012

Migrating databases between SQL Server and SQL Server Compact

In this post, I will try to give an overview of the free tools available for developers to move databases from SQL Server to SQL Server Compact and vice versa. I will also show how you can do this with the SQL Server Compact Toolbox (add-in and standalone editions).

Moving databases from SQL Server Compact to SQL Server

This can be useful for situations where you already have developed an application that depends on SQL Server Compact, and would like the increased power of SQL Server or would like to use some feature, that is not available on SQL Server Compact. I have an informal comparison of the two products here. Microsoft offers a GUI based tool and a command line tool to do this: WebMatrix and MsDeploy. You can also use the ExportSqlCe command line tool or the SQL Server Compact Toolbox to do this. To use the ExportSqlCE (or ExportSqlCE40) command line, use a command similar to:

ExportSQLCE.exe "Data Source=D:\Northwind.sdf;" Northwind.sql

The resulting script file (Northwind.sql) can the be run against a SQL Server database, using for example the SQL Server sqlcmd command line utility:

sqlcmd -S mySQLServer –d NorthWindSQL -i C:\Northwind.sql

To use the SQL Server Compact Toolbox:

Connect the Toolbox to the database file that you want to move to SQL Server:

image

Right click the database connection, and select to script Schema and Data:

image

Optionally, select which tables to script and click OK:

image

Enter the filename for the script, default extension is .sqlce:

image

Click OK to the confirmation message:

image

You can now open the generated script in Management Studio and execute it against a SQL Server database, or run it with sqlcmd as described above.

Moving databases from SQL Server to SQL Server Compact

Microsoft offers no tools for doing this “downsizing” of a SQL Server database to SQL Server Compact, and of course not all objects in a SQL Server database CAN be downsized, as only tables exists in a SQL Server Compact database, so no stored procedures, views, triggers, users, schema and so on. I have blogged about how this can be done from the command line, and you can also do this with the SQL Server Compact Toolbox (of course):

From the root node, select Script Server Data and Schema:

image

Follow a procedure like the one above, but connecting to a SQL Server database instead.

The export process will convert the SQL Server data types to a matching SQL Server Compact data type, for example varchar(50) becomes nvarchar(50) and so on. Any unsupported data types will be ignored, this includes for example computed columns and sql_variant. The new date types in SQL Server 2008+, like date, time, datetime2 will be converted to nvarchar based data types, as only datetime is supported in SQL Server Compact. A full list of the SQL Server Compact data types is available here.

Wednesday, February 8, 2012

Saving images and long strings to SQL Server Compact with Entity Framework 4.1 Code First

This StackOverflow question points out an issue with EntityFramework 4.1, when used with a SQL Server Compact table with image columns. The “image” type is the predecessor to varbinary(MAX), and is used for storing large binary values, sometimes referred to as BLOBs.

I have created a sample Console application with fixes for the related issues.

To re-create the application, install EntityFramwork 4.1, create a C# Windows Console application in Visual Studio 2010, and add .NET references to:

System.Data.Entity
EntityFramework
System.ComponentModel.DataAnnotations

Then add code similar to this:

using System;
using System.ComponentModel.DataAnnotations;
using System.Data.Entity;
//Also added reference to EntityFramework.dll (EF 4.1)
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;

namespace CfTest
{
class Program
{
static void Main(string[] args)
{
using (var db = new StudentContext())
{
var student = new Student { Name = "Erik", Photo = ConvertImageToByteArray(@"C:\Users\Erik\Pictures\erik.jpg"), StudentId = 1 };
student.LongText = new string('x', 6666);
db.Students.Add(student);
int recordsAffected = db.SaveChanges();

Console.WriteLine(
"Saved {0} entities to the database, press any key to exit.",
recordsAffected);

Console.ReadKey();
}

}

private static byte[] ConvertImageToByteArray(string fileName)
{
Bitmap bitMap = new Bitmap(fileName);
ImageFormat bmpFormat = bitMap.RawFormat;
var imageToConvert = Image.FromFile(fileName);
using (MemoryStream ms = new MemoryStream())
{
imageToConvert.Save(ms, bmpFormat);
return ms.ToArray();
}
}
}


public class Student
{
public int StudentId { get; set; }
public string Name { get; set; }

// Required to force Code First to create a ntext column, not a nvarchar(n)
[MaxLength]
public string LongText { get; set; }

// Required to force Code First to create an image column, not a binary(n)
[MaxLength]
public byte[] Photo { get; set; }
}

public class StudentContext : DbContext
{
public DbSet<Student> Students { get; set; }
}

}



And add an app.config with contents like this. The connection string name matches the DbContext name and this causes Code First to magically create a database file in the specified location, with a Students table:



<?xml version="1.0" encoding="utf-8" ?>
<
configuration>
<
connectionStrings>
<
add name="StudentContext"
providerName="System.Data.SqlServerCe.4.0"
connectionString="Data Source=C:\Users\Erik\Documents\visual studio 2010\Projects\CfTest\Students.sdf"/>
</
connectionStrings>
</
configuration>



This is the resulting Students table:




-- Script Date: 11-07-2011 09:43  - Generated by ExportSqlCe version 3.5.1.3

CREATE TABLE [Students] (


  [StudentId] int NOT NULL  IDENTITY (1,1)


, [Name] nvarchar(4000) NULL


, [LongText] ntext NULL


, [Photo] image NULL


);


GO


ALTER TABLE [Students] ADD CONSTRAINT [PK__Students__000000000000000A] PRIMARY KEY ([StudentId]);


GO





This code demonstrates how to convert an image to a byte array, and also highlights the attributes required to work with image columns:

// Required to force Code First to create an image column, not a binary(n)
[MaxLength]
public byte[] Photo { get; set; }


The Photo column must be decorated with the MaxLength attribute.

// Required to force Code First to create a ntext column, not a nvarchar(n)
[MaxLength]
public string LongText { get; set; }



Likewise the LongText column must specify the MaxLength attribute.



public class StudentContext : DbContext
{
public DbSet<Student> Students { get; set; }
}



UPDATE: @DamienGuard, of LINQ to SQL and EF Code First fame, pointed out that a better and more provider agnostic solution is to use MaxLength rather than TypeName = “ntext”.


UPDATE 2: Using [MaxLength] prevents any validation errors, and disabling validation is not required.

Monday, February 6, 2012

Windows Phone Local Database tip: Exploring multiple UPDATEs and rowversion impact

According to the Local Database best practices, it is recommended to use a rowversion column in your local database table, if you are performing many updates. This blog post will examine the performance and changes to UPDATE handling that are caused by the presence of a rowversion column.

But first, how to add a rovversion column: If you are using code first, this simple column declaration will suffice:
[Column(IsVersion=true)]
private Binary _version;

If you are using database first, simply add a rowversion column like so:
ALTER TABLE [Artist]
ADD [Version] rowversion NOT NULL

Let us run the following test code which updates 100 rows with and without a rowversion column. I am using a prepopulated Chinook database, added as en embedded resource to the project. See this blog post for details on how to do this.

//New database from embedded resource
using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
if (db.DatabaseExists())
db.DeleteDatabase();

db.CreateIfNotExists();
}

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{

try
{
db.LogDebug = true;

var arists = db.Artist.Take(100);

foreach (var artist in arists)
{
artist.Name = Guid.NewGuid().ToString();
}

System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
sw.Start();

db.SubmitChanges();

sw.Stop();
System.Diagnostics.Debug.WriteLine(sw.ElapsedMilliseconds);
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.ToString());
}
}


Now lets us see what the SQL statements executed look like without a rowversion column on the table:




UPDATE [Artist]


SET [Name] = @p2


WHERE ([ArtistId] = @p0) AND ([Name] = @p1)



As you can see, a standard UPDATE statement is executed, including a WHERE clause referring to all columns in the table, in order to trap concurrency exceptions. The time to do 100 updates is about 320 ms.



Now let us add the rowversion column to the Artist table, recreate the DataContext so it contains the Version column, and run the same code.



Now the SQL statements look like so, indicating that the query processor is being bypassed:



-- CURSOR BASED INDEX UPDATE [Artist].[PK_Artist] (

--     ArtistId: [99]


--    )


-- EQUALITY CHECK [Version], [System.Byte[]] = [System.Byte[]]


-- [Name] <= [5ad7e0d2-925e-44f7-bcd7-c3bca32de745]


-- AUTOSYNC [Version] <= [System.Byte[]]



The time to do 100 updates is now about 110 ms, a significant improvement from 320 ms. So if you expect to do UPDATEs on your local Windows Phone database, add a rowversion column to your tables!