Everything SQL Server Compact: 2011

Thursday, December 22, 2011

Windows Phone Local Database tip: Exploring INSERT performance–5 power tweaks

Josué Yeray Julián, (blog in Spanish here), has done some interesting tests on INSERT performance, both on the emulator (like I started doing in my previous blog post) and on two real devices, an HTC Mazaa and Nokia Lumia 800. (Sadly, I do not own a real device – yet!)

In summary, the following parameters can improve performance for doing INSERTs of 500.000 rows:

1: Set “Max Database Size” on your connection string to an expected size in advance (default is 32 MB, max is 512 MB on Windows Phone)

2: Increase “Max Buffer Size” to a value higher than the default of 384 KB. On the tested real devices the max of 4096 worked best.

3: Run the INSERT process in a background thread.

4: Remove any Version columns (if only INSERTSs matter, not UPDATEs)

5: Use InsertAllOnSubmit instead of InsertOnSubmit

These tweaks reduced the total processing time on the Nokia Lumia 800 from 29:20 to 16:04!

But - keep in mind that you can use Database first development with the SQL Server Compact Toolbox add-in and then include the database with your application package. You can also use the SqlCeBlukCopy API to insert 1.000.000 rows (twice as many as tested) in 6 seconds on the desktop!

You can read the Microsoft Translator versions of his 2 blog posts here and here.

Sunday, December 11, 2011

Windows Phone Local Database tip: Batch INSERT performance

You could imagine a scenario, where one of the tables in your local database need to be periodically updated. For Windows Phone Local Database, there is no support for Sync Framework or similar technologies, so you may have to simply add a bunch of rows to your table, based on data retrieved from a web service. This blog post explores how SQL Server Compact on the device does INSERTs, and how batch INSERTs can be tweaked for optimum performance.

Let’s start by using the DebugWriter class from my previous post, and examine what INSERT statements look like. For this test, I have included a copy of the Chinook database with the phone application as an embedded resource, so the test table already contains 275 rows. Then use this code to view INSERT statements.

private void button1_Click(object sender, RoutedEventArgs e)
{
    CleanDatabase();
    using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
    {
        db.LogDebug = true;

        var artist = new Artist();
        artist.Name = Guid.NewGuid().ToString();

        db.Artist.InsertOnSubmit(artist);
        db.SubmitChanges();
    }
}

private static void CleanDatabase()
{
    using (var store = IsolatedStorageFile.GetUserStoreForApplication())
    {
        store.Remove();
    }

    using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
    {
        db.CreateIfNotExists();
    }
}

This results – surprisingly – in the following output:

-- CURSOR BASED INSERT [Artist]
-- [Name] <= [2807fb28-49d7-4a25-8350-e4e40245934a]

-- AUTOSYNC [ArtistId] <= [276]

This indicates that in order to perform INSERTs to the Local Database, the query processor is bypassed (no “INSERT” SQL statement is shown). Instead, the LINQ to SQL implementation on the device uses the SqlCeResultSet and SqlCeUpdateableRecord classes (or equivalent), that allows you to access the database tables without the overhead of the query processor, similar to what I use in the SqlCeBulkCopy library for performing fast INSERTs in SQL Server Compact databases on the desktop. So for INSERTs, the device implementation is already optimized in that respect.

Let’s then look at the speed of “many” INSERTs, in this case 10000. The following code implements this test:

CleanDatabase();

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
    List<Artist> artistList = new List<Artist>();

    for (int i = 0; i < 10000; i++)
    {
        var artist = new Artist();
        artist.Name = Guid.NewGuid().ToString();

        artistList.Add(artist);
    }

    db.Artist.InsertAllOnSubmit(artistList);
    System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
    sw.Start();
    db.SubmitChanges();
    sw.Stop();
    System.Diagnostics.Debug.WriteLine(sw.ElapsedMilliseconds);            
}

On my emulator, this takes about 2500 ms for each run. So 2.5 seconds to insert 10.000 rows – no too bad. Let’s see if there are any knobs available in the connection string parameters that will allow us to tweak this. The connection string parameters available for Windows Phone Local Database are documented here.

Inspired by this article by Joao Paulo Figueira, we could try to increase the Max Buffer Size from it’s default value of 384 K to 1024 K.

This can be done in the connection string like this:

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString + ";max buffer size=1024"))

And lo and behold, the same process of inserting 10000 rows now takes 1.6 seconds on my emulator, a 36% improvement! (Your mileage may vary, of course).

Note: Increasing above 1024 buffer size had no positive effect.

Hope you found this useful.

Speaking of Data on Windows Phone, I will be reviewing the Windows Phone 7.5 Data Cookbook soon, you can download a sample chapter on Isolated Storage from here.

Monday, November 28, 2011

Windows Phone Local Database tip: Viewing the SQL generated by LINQ to SQL

It may sometimes by useful to be able to see the SQL statements generated by an ORM, in order to understand better what is going on, and why things perform the way they do. With LINQ to SQL on desktop, you can assign any TextWriter object to the DataContext Log property, and this will allow you to log to a file or the console.

On Windows Phone, this gets a little bit complicated, as files must be stored in Isolated Storage, making them hard to get at, and there is no Console. This MSDN blog includes a TextWriter implementation, that writes to the console of a attached debugger, for example the Visual Studio Debug output Window. This class (which I have renamed to DebugWriter) will be part of the DataContext generated by the next SQL Server Compact Toolbox release. So now you can add the following while debugging your LINQ to SQL statements:

db.Log = new DebugWriter();

We can now try some various LINQ queries against our database to find out how they have been implemented in SQL (I am using the excellent Chinook sample database, available for download here):

db.Album.Where(al => al.AlbumId == 10).SingleOrDefault();

And the output in the Debug output window:

db.Album.FirstOrDefault();

Becomes:

SELECT TOP (1) [t0].[AlbumId], [t0].[Title], [t0].[ArtistId]
FROM [Album] AS [t0]

db.Album.Where(al => al.Title.StartsWith("For those%")).FirstOrDefault();

Becomes:

SELECT TOP (1) [t0].[AlbumId], [t0].[Title], [t0].[ArtistId]
FROM [Album] AS [t0]
WHERE [t0].[Title] LIKE @p0

db.Album.Skip(50).Take(25).ToList();

Becomes:

SELECT [t0].[AlbumId], [t0].[Title], [t0].[ArtistId]
FROM [Album] AS [t0]
ORDER BY [t0].[AlbumId] OFFSET @p0 ROWS FETCH NEXT @p1 ROWS ONLY

Notice that the last statement uses OFFSET and FETCH, a syntax only introduced in SQL Server Compact 4.0, but apparently retrofitted into the version 3.5 engine running on Windows Phone.

Happy SQL debugging!

Monday, November 14, 2011

SQL Compact 3rd party tools

This is an attempt at a comprehensive list of SQL Server Compact Microsoft and 3rd party tools, both commercial and non-commercial. Please let me know if I have missed anything, and I will add it.

Microsoft supplied tools

Visual Studio Community 2017 free – when used with the SQL Server Compact / SQLite Toolbox: SQL CE 3.5, SQL CE 4.0

Visual Studio 2008 Professional (SQL CE 3.5)

Visual Studio 2005 Standard (SQL CE 3.1)

SQL Server 2008 R2 Management Studio Express (free, SQL CE 3.5 SP2)
Link: http://www.microsoft.com/en-us/download/details.aspx?id=30438
SQL Server 2005 Management Studio Express SP2 (free, SQL CE 3.1)
Link: http://www.microsoft.com/en-us/download/details.aspx?id=15366

New or updated entries are noted with (new) or (updated) ”Last updated” indicates when the tool was last updated by it’s author.

Non-commercial (free) tools

My GitHub Tools

Name: SQL Server Compact Toolbox (updated)
Link: https://github.com/ErikEJ/SqlCeToolbox
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 29th June 2010
”Add-in for Visual Studio 2071/2015/2013/2012 (for 3.5 and 4.0). Standalone app (for 4.0 and 3.5)
Adds several features to help your SQL Server Compact development efforts:
- Scripting of tables, including data.
- Script of entire schema, optionally with data, both of SQL Server Compact and SQL Server 2005 or later databases, including SQL Azure.
- Import to SQL Compact 3.5 from a server database or CSV file.
- Basic, free form query execution.
- Generation of DGML files for visualizing table relationships
- Database Schema Diff Script
- Detect database file version
- Fix broken connection definitions”

Name: SqlCeScripting.dll (updated)
Link: https://github.com/ErikEJ/SqlCeToolbox
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 29th June 2010
”Scripting library for inclusion in your own application, enumerates SQL Compact objects and allows scripting of these.”

Name: SqlCeCmd (updated)
Link: https://github.com/ErikEJ/SqlCeCmd
SQL CE Versions: 3.5, 4.0
Last updated: 2015
Platforms: Win32, Win64
”Administer SQL Compact databases directly from the command line, without having Visual Studio or SQL Server Management Studio installed. Only requires NetFx 2.0 and SQL Compact installed. Command line options similar to sqlcmd. Runs scripts generated by ExportSqlCe.”

Name: ExportSqlCe (updated)
Link: https://github.com/ErikEJ/SqlCeToolbox/
SQL CE Versions: 3.0, 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
”This command line utility helps you better manage your SQL Compact development efforts. If allows you to script schema and data from any SQL Compact database file to a .sql file, which can be used in any context. The utility is developed in C# with Visual Studio 2008. You can use the resulting script for documentation, programmatic scripting of SQL Compact database files, or migration of schema and data to SQL Server (Express) 2005 or later.

Name: Export2SqlCe (updated)
Link: https://github.com/ErikEJ/SqlCeToolbox
SQL Server Versions: 2005, 2008, 2012, 2014
Last updated: Current
Platforms: Win32
Added to list: 4th March 2010
”A command line utility to script an entire SQL Server 2005 or later database with or without data to a SQL Compact T-SQL script.”

Name: SqlCeBulkCopy (updated)
Link: https://github.com/ErikEJ/SqlCeBulkCopy
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: .NET library, .NET CF library
Added to list: 4th March 2010
”.NET Library for loading data fast (doing bulk inserts) into a SQL Server Compact database file. Attempts to mimic the SQLClient SqlBulkCopy API. Includes support for BULK INSERTing any IEnumerable”

Query and database management tools

Name: CompactView
Link: http://sourceforge.net/p/compactview/home/Home/
SQL CE Versions: 3.1, 3.5, 4.0
Last updated: 2015
Platforms: Win32, Win64
Added to list: 14th November 2011
CompactView is a viewer for Microsoft® SQL Server® Compact Edition (SQLCE) database files (*.sdf). CompactView can open database files of versions 3.1, 3.5 and 4.0

Name: SQL Compact Query Analyzer
Link: http://sqlcequery.codeplex.com/
SQL CE Versions: 3.1, 3.5, 4.0
Last updated: 2013
Platforms: Win32, Win64
Added to list: 3rd July 2011
Features:
- Execute SQL Queries against a SQL Server Compact Edition database
- Table Data Editor to easily edit the contents of the database
- Supports SQLCE 3.0, 3.1, 3.5 and 4.0
- Execute multiple SQL queries (delimited by a semi colon ;)
- Display query result as XML
- Shrink and Compact Databases
- Generate Schema and Data Scripts
- Display database and schema information

Name: LINQPad
Link: http://www.linqpad.net/
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 14th December 2010
Linqpad allows you to query various data sources via LINQ, including SQL Server Compact 3.5 and 4.0. See this blog post.

Name: Database .NET
Link: http://fishcodelib.com/Database.htm
SQL CE Versions: 3.1, 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 20th September 2010
“Database .NET is a simple and intuitive database management tool for multiple databases. Making Local and Remote Database Administration Easier, With it you can Create databases, Design tables, Edit data, Export data, Generate scripts and run SQL queries.”

Name: SQL CE Browser
Link: http://ericwillis.com/notes/2009/12/sql-ce-browser-v-110/
SQL CE Versions: 3.5
Last updated: 2010
Platforms: Win32, Win64 (?)
Added to list: 4th March 2010
”Open regular and encrypted/passworded databases for viewing or editing. Write and work with any number of queries at once via tabs. SQL syntax highlighting and SQL syntax verification. Explore columns, keys, and indexes“

Name: Mini SQL Query
Link: http://minisqlquery.codeplex.com/
SQL CE Versions: 3.5
Last updated: 2014
Platforms: Win32, Win64 (?)
Added to list: 4th March 2010
”Mini SQL Query is a minimalist SQL query tool for multiple databases (MSSQL, MSSQL CE, SQLite, OLEDB, MS Access/Excel files etc). The tool uses a generic schema engine taking advantage of the ADO.NET providers. Mini SQL Query is also easily extended with plugins.“

Name: SQL Server CE Query Tool
Link: http://cequery.codeplex.com
SQL CE Versions: 3.5, 4.0
Last updated: 2011
Platforms: Win32, Win64 (?)
”CEQuery is written in C# with the help of Visual Studio 2008 platform. Some of the features of this tool are described below:
User is able to create CE database. If the schema is provided, the tool is able to create the tables and columns. User is able to overview an existing CE database with its table and column structure in tree format. User can design a query by dragging and dropping tables and columns from the database tree. User can manipulate [Insert/Update/Delete] records in a selected table. User is able to generate scripts from a single table or of whole databases. [Both for SQL CE or SQL Server]. User is able to open multiple SQL Server CE files simultaneously. User is able to convert a SQL Server database to a SDF file with schema and data. User is able to change password of a SQL CE DB. The tool is now capable of handling the database with no password.“

Name: SeasonStar Database Management(SSDM)
Link: http://ssdm.codeplex.com/
SQL CE Versions: 3.5?, 4.0
Last updated: 2011
Platforms: Win32
”With support for all major database formats, top-class export functions and advanced feaures like Linq Data Context availabe, SSDM is probably your best choice for DB management. It's even free software!”

Name: SQL CE Database Editor
Link: http://sqlcedatabaseeditor.codeplex.com/
SQL CE Versions: 3.5
Last updated: 2009
Platforms: Win32
“Small lightweight application to browse and edit Microsoft SQL CE server databases. Does not require SQL CE server to be installed and can run custom queries.”

Name: DataBoy
Link:http://databoy.codeplex.com/
SQL CE Versions: 3.0, 3.5
Last updated: 2008
Platforms: Win32
“Small footprint, portable and simple database query tool. Support SQL CE 3.1, SQL CE 3.5 and SQL Server.”

Other tools and libraries

Name: Lyare.SqlServerCe.MultiQuery (new)
Link: https://github.com/lokiworld/Lyare.SqlServerCe.MultiQuery
SQL CE Versions: 4.0
Last updated: 2014
Platforms: Win32, Win64
Added to list: 2nd March, 2014
“An extension to the Microsoft SQL Server Compact Edition client library, which simulates the support of the multiple statements in a single command.”

Name: SQLCE SCRIPTER
Link: http://sqlmgmt.com/sql/SQLCE-Scripter
SQL CE Versions: 3.5, 4.0
Last updated: 2011
Platforms: Win32, Win64
Added to list: 14th November, 2011
“Create SQL schema scripts for tables, indexes and referential relations. Create data scripts so that CE data can be moved to SQL Server or other data store. User can select specific tables only. User can select Script Schema, Script Data or both.”

Name: SQLCE Code Generator
Link: http://sqlcecodegen.codeplex.com/
SQL CE Versions: 3.5, 4.0
Last updated: 2012
Platforms: Win32, Win64
Added to list: 27th March, 2011
“Contains a stand alone GUI application and a Visual Studio Custom Tool for automatically generating a .NET data access layer code with Unit tests for objects in a SQL Server Compact Edition database. Also generates a Windows Phone DataContext”

Name: OpenNETCF.ORM Framework
Link: http://orm.codeplex.com/
SQL CE Versions: 3.5
Last updated: Current
Platforms: WinCE, Win32, Win64
Added to list: 27th March, 2011
“The OpenNETCF.ORM library is a lightweight object-relational mapping framework designed for use with the .NET Compact Framework.The framework is extensible to allow any type of backing store. An initial implementation using SQL Server Compact is provided.”

Name: SQL Metal Open Source User Interface
Link: http://sqlmetalosui.codeplex.com/
SQL CE Versions: 3.5
Last updated: 2008
Platforms: Win32
“This project provides an user interface for SqlMetal.exe tool.
SQLMetalOSUI allows users to generate code and mapping files for their SQL Server Compact Edition databases. SQL Server Compact Edition 3.5 is fully compatible with LINQ to SQL but the designer does not. This project will be focused to reach this initial purpose.”

Commercial tools

Name: MS Compact Maestro (new)
Link: http://www.sqlmaestro.com/products/mssql/compact_maestro/
SQL CE Versions: 3.5, 4.0
Last updated: 2013
Platforms: Win32, Win64
”MS Compact Maestro is a top-class SQL Server Compact management and development tool.”

Name: SDF Viewer
Link: http://www.flyhoward.com/SDF_Viewer.aspx
SQL CE Versions: 3.0, 3.5, 4.0
Last updated: Current
Platforms: Win32
“The simple way to create, view, edit or export the data in your Windows Mobile or Pocket PC device's SQLCE database, from the convenience of your Desktop. Run SQL commands on your SQLCE databases, and view the results instantly.”

Name: RemoteSqlCe
Link: http://www.gui-innovations.com/html/remotesqlce.html
SQL CE Versions: 2.0, 3.0
Last updated: 2007
Platforms: Windows CE, Win32
”SqlServerCE is a great database for Pocket PC’s, but designing, maintaining and querying databases on the PPC, especially whilst developing, can be wearing. We developed this tool to help us when developing for SqlServerCE. With RemoteSQLCe you can completely manage your SqlServerCE database from the desktop. RemoteSQLCE works in the cradle, and over ethernet/bluetooth.”

Wednesday, November 9, 2011

SQL Server Compact 3.5 SP2 now supports Merge Replication with SQL Server 2012

A major update to SQL Server Compact 3.5 SP2 has just been released, version 3.5.8088.0, disguised as a “Cumulative Update Package”. Microsoft knowledgebase article 2628887 describes the update.

The update contains updated Server Tools, and updated desktop and device runtimes, all updated to support Merge Replication with the next version of SQL Server, SQL Server 2012.

For a complete list of Cumulative Updates released for SQL Server Compact 3.5 SP2, see my blog post here.

It is nice to see that the 3.5 SP2 product, with it’s full range of device support and synchronization technologies is kept alive and kicking.

NOTE: I blogged about this update earlier, but it was pulled back. Now it is finally available, and all downloads can be requested. (I have downloaded all the ENU ones, anyway)

Monday, November 7, 2011

SQL Server Compact Toolbox 2.5–Visual Guide of new features

After more than 66.000 downloads, version 2.5 of my SQL Server Compact Toolbox extension for Visual Studio 2010 is now available for download. This blog post is a visual guide to the new features included in this release, many suggested by users of the tool via the CodePlex issue tracker

Properties of selected Database and Table displayed in Properties window

When you navigate the list of SQL Server Compact databases and database objects, the Toolbox now displays properties for a Database and a Table.

The Database properties are the following:

And the table properties are these:

DGML file (database graph) now also contains object descriptions

The database graph (DGML) file has been enhanced to display descriptions of Database, Table and Column objects, based on the documentation feature introduced in version 2.3:

Entity Data Model dialog now allows adding configuration for Private Desktop Deployment to app.config

When using Entity Framework with Private Deployment of SQL Server Compact, some entries in app.config are required (for Desktop application), as described here and here. This is required, as Entity Framework build in the DbProvider interfaces, which requires some configuration entries.

These settings can now be added when creating an EDM as shown below:

Ability to add 3.5 Connections from Toolbox

It is now possible to add 3.5 Database Connections to the Server Explorer and the Toolbox directly from the Toolbox, rather than having to go to Server Explorer, add the connection, and then Refresh the Toolbox. You can now do this without leaving the Toolbox, and the Toolbox will refresh “automagically”.

Improved VS UI Guidelines compliance

The Toolbars, SQL Editor Font, Dialogs (frame, background and buttons) have been overhauled to comply better with my recent discovery of “Visual Studio UI Guidelines”, available for download here. In addition, the Toolbox now follows the chosen Visual Studio Theme (to some extent, anyway).

This is the “new look” for the Toolbox Explorer and SQL Editor:

Other minor improvements and fixes

- Explorer Treeview: ArgumentException when getting connections
- WinPhone DataContext: Split files failed with empty database
- SQL editor: Check if .sqlplan is supported
- SQL editor: Save button was not working
- SQL editor: Results pane not always cleared
- SQL editor: Results as text improved formatting
- SQL editor: Text scrollbar was overlaid by splitter bar

As usual, the full source code for these new features is available on CodePlex for you to re-use or maybe improve!

Thursday, October 27, 2011

SQL Server Compact 4.0 under ASP.NET Hosting– common issues

This blog post covers some of the issues that can be encountered when using SQL Server Compact under ASP.NET, in particular in a hosted environment, where the environment can be restricted in unexpected ways.

In order to be able to troubleshoot issues with ASP.NET hosting of web sites using SQL Server Compact 4.0, it can be useful to understand how Medium Trust works.

Medium Trust only works under .NET 4.0 (ASP.NET 4.0), as it depends on a configuration in the global web.config.

The global web.config is located at C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config or C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config, and contains the following setting, that was configured when .NET 4.0 was installed:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location allowOverride="true">
    <system.web>
      <fullTrustAssemblies>
        <add
            assemblyName="System.Data.SqlServerCe"
            version="4.0.0.0"
            publicKey="0024000004800000940000000602000000240000525341310004000001000100272736ad6e5f9586bac2d531eabc3acc666c2f8ec879fa94f8f7b0327d2ff2ed523448f83c3d5c5dd2dfc7bc99c5286b2c125117bf5cbe242b9d41750732b2bdffe649c6efb8e5526d526fdd130095ecdb7bf210809c6cdad8824faa9ac0310ac3cba2aa0523567b2dfa7fe250b30facbd62d4ec99b94ac47c7d3b28f1f6e4c8"
                />
      </fullTrustAssemblies>
      <partialTrustVisibleAssemblies />
    </system.web>
  </location>

So this was added during installation of .NET 4.0, long before SQL Server Compact 4.0 was released in January 2011. The fullTrustAssemblies element is new in .NET 4.0: http://weblogs.asp.net/asptest/archive/2010/04/23/what-is-new-in-asp-net-4-0-code-access-security.aspx

If this section has been removed from the global web.config file, SQL Server Compact 4.0 will not run under medium trust.

It is possible to configure similar functionality under .NET 3.5 SP1, as described here: http://msdn.microsoft.com/en-us/library/ms174612(v=SQL.110).aspx

Notice that the SQL Server Compact binaries are delivered with two Assembly version numbers, 4.0.0.0 and 4.0.0.1. The 4.0.0.1 Assembly is for Private deployment, but on desktop only, as described here: http://msdn.microsoft.com/en-us/library/gg213826.aspx – it will not work with Medium Trust, as the assembly version registered in the machine.config file is 4.0.0.0, as shown above.

Other issues you may encounter when running ASP.NET and SQL Server Compact 4.0 can be:

- The process identity running your web application must have write access to the App_Data folder or the folder where your database file resides:

http://social.msdn.microsoft.com/Forums/en-US/sqlce/thread/6c1c8798-85d7-4d8f-9908-e9f7dc1c17e4

- SQL Server Compact database files must be located on a local drive, not shared folders:

http://stackoverflow.com/questions/7791345/sqlce-4-ef4-1-internal-error-cannot-open-the-shared-memory-region/7804049#7804049

- If SQL Server Compact is not installed locally, a number of files must be included in the bin folder of your web site:

http://stackoverflow.com/questions/3223359/cant-get-sql-server-compact-3-5-4-to-work-with-asp-net-mvc-2/3223450#3223450

http://msdn.microsoft.com/en-us/library/gg286946.aspx

- If only .NET 4.0 is present on the server, you may be missing the Visual C++ 2008 SP1 Redistributable Package (which is installed only with .NET 3.5 SP1)

I will update this blog post if I notice more “common issues”.

Tuesday, October 11, 2011

SQL Server Compact books

A number of SQL Compact (or a least SQL Compact related) books have appeared on the market.

I will attempt to keep this list of books up to date as new material appears.

“Building Standalone Data-Driven Applications using SQL Server Compact 4.0 and Visual Studio 2010”

“Microsoft SQL Server Compact 4.0 (SQL Compact onwards) is a free SQL Server embedded database ideal for building standalone desktop, web, and mobile applications that need local data storage.”

http://www.c-sharpcorner.com/Ebooks/

“Microsoft SQL Server Compact 4.0 Books Online”

Microsoft SQL Server Compact 4.0 is a free, embedded database that software developers can use for building ASP.NET websites and Windows desktop applications. Download the documentation to learn about SQL Server Compact 4.0 and to build applications for it.

http://www.microsoft.com/download/en/details.aspx?id=21880

“La réplication du fusion avec SQL Server Compact: Implémentation and conseils”

“Free eBook in French about implementation and advise regarding Merge Replication – in many ways similar to the Rob Tiffany book.”

http://www.wygwam.com/Documents/ReplicationFusionSQLServerCompact.pdf

“.NET Compact Framework 3.5 Data Driven Applications”

“Develop data-driven mobile applications from the ground up on top of the Oracle Lite and SQL Server Compact”

http://www.packtpub.com/data-driven-applications-with-.net-compact-framework-3-5/book

“Microsoft SQL Server Compact 3.5 Service Pack 2 Books Online”

This download contains the Books Online for the SP2 version of SQL Server Compact 3.5.

http://www.microsoft.com/downloads/details.aspx?familyid=746C3A6E-FFB1-4C92-93FA-B3BA41FDE681&displaylang=en

“SQL Server MVP Deep Dives”
This book with contribution from 53 SQL Server MVPs includes 2 chapters relating to SQL Compact:
Placing SQL Server in your pocket by Christopher Fairbairn and Mobile data strategies by John Baird
http://www.manning.com/nielsen/

"Enterprise Data Synchronization with SQL Server 2008 and SQL Server Compact 3.5"

An update of the previous book, equally excellent and hands-on.

Available from Amazon

"Microsoft Mobile Development Handbook"

This book covers development using .NET Compact Framework 2.0 in broad terms, and has two chapters relating to SQL Compact:

Chapter 3 describes how to persist data using SQL Compact and how to bind SQL Compact data to UI elements and chapter 7 briefly covers using RDA and merge replication with SQL Compact.

http://www.microsoft.com/MSPress/books/10294.aspx

"Windows Mobile Data Synchronization with SQL Server 2005 and SQL Server Compact 3.1"

I have already recommended the excellent Merge Replication book by Rob Tiffany here:

http://erikej.blogspot.com/2007/10/rob-tiffany-publishes-mobile-merge.html

Available from amazon

"Microsoft® SQL Server 2005 Compact Edition"

This book covers the product in full, including a chapter on Native development, and several chapters on replication. The main perspective of this this book is a DBA approach, with only a single chapter covering development per se. There are also useful chapters on security and performance tuning.

http://www.informit.com/store/product.aspx?isbn=0672329220

"Hitchhiker's Guide to Microsoft SQL Server 2005 Compact Edition"
For desktop developers diving into SQL Compact, the e-book by Bill Vaughn is an excellent introduction, which covers many aspects and pitfalls of developing with SQL Compact edition in a desktop application.

http://www.hitchhikerguides.net/EBooks/5582.aspx

Friday, September 30, 2011

Analyzing SQL Server Compact queries using Visual Studio 2010 Premium/Ultimate

f you are the happy owner of Visual Studio 2010 Premium or Ultimate, there is a hidden tool that allows you to run and analyze queries against SQL Server Compact 3.5 and 4.0 databases. (Support for 4.0 requires Visual Studio 2010 SP1 + the SQL Server Compact Tools update). This blog post will walk through how to access and use this “hidden” tool.

NOTE: If you only have Visual Studio Professional, you can use my SQL Server Compact Toolbox in combination with the free SQL Server 2008 R2 Management Studio Express to perform similar query analysis.

To access the tool, go to the Data menu, and select Transact-SQL Editor, New Query Connection… (The tool is part of the so-called “Data Dude” features)

In the Connect to Server dialog, select SQL Server Compact:

You can select an existing database, or even create a new one. This dialog will automatically detect if the specified file is a version 3.5 or 4.0 file.

Once connected, you can perform functions similar to what you may know from SQL Server Management Studio:

Thursday, September 22, 2011

SQL Server Compact Toolbox available for Visual Studio 11

Visual Studio 11 Developer Preview is now available for testing. As one of the first third party add-ins, a build of the SQL Server Compact Toolbox version 2.4 that supports this Visual Studio Preview version is available via Extension Manager or in the Visual Studio Gallery.

In order to add support for Visual Studio version 11 in an existing add-in, all you need to do is modify the source.extension.vsixmanifest file as shown below:

    <SupportedProducts>
      <VisualStudio Version="10.0">
        <Edition>Pro</Edition>
      </VisualStudio>
      <VisualStudio Version="11.0">
        <Edition>Pro</Edition>
      </VisualStudio>
    </SupportedProducts>

The result of this change is that the add-in can now be installed for several versions of Visual Studio.

I have had to make some changes, as the Toolbox currently depends on SQL Server Compact 3.5 SP2 to store it’s connections, and only SQL Server Compact 4.0 is included with Visual Studio 11. In the Developer Preview the version of SQL Server Compact included is the 4.0 RTM version, so no changes there for now.

To detect which version of Visual Studio you are running, you can use the following code in your Package.cs class:

public Version VisualStudioVersion
{
    get
    {
        var dte = this.GetServiceHelper(typeof(EnvDTE.DTE)) as EnvDTE.DTE;
        string root = dte.RegistryRoot;

        if (root.Contains("10.0"))
        {
            return new Version(10, 0);
        }
        else if (root.Contains("11.0"))
        {
            return new Version(11, 0);
        }
        else
        {
            return new Version(0, 0);
        }
    }
    set
    {
        this.VisualStudioVersion = value;
    }
}

I am currently not bringing forward any 4.0 connections defined in the VS 2010 edition of the add-in. Please let me know if a feature to import these connections to the VS 11 Server Explorer would be useful.

Also, would it be of interest to be able to manage 3.5 databases in VS 11, even though they are not supported in Server Explorer?

As always, please provide any feedback in the comments or via the Codeplex issue tracker.

Friday, September 9, 2011

SQL Server Compact Toolbox 2.4–Visual Guide of new features

After more than 50.000 downloads, version 2.4 of my SQL Server Compact Toolbox extension for Visual Studio 2010 is now available for download. This blog post is a visual guide to the new features included in this release.

Edit Table Data (beta)

The tools that are included with Server Explorer for SQL Server Compact 3.5 and 4.0 already include a feature to edit table data, called Show Table Data:

But the grid has some limitations that I have lifted on the new “Edit Table Data” feature:

- Ability to sort data by clicking a column heading

- Ability to Import/Export/Delete content of image columns

- Ability to locate data in a column (QuickSearch)

Split Windows Phone DataContext into multiple files

You can now select to have the Windows Phone DataContext generated as a DataContext class file, and a class file per table in your database.

Select tables to include in Entity Data Model

You can now select which tables to include in the Entity Data Model created from your SQL Server Compact database. (Thanks to the Extended WPF Toolkit Project)

Explore and script primary and foreign keys

The tree view now lists the primary and foreign keys belonging to a table, and it is also possible to script these individually.

Other fixes

Improved Add-in Update detection (for users behind proxies)
Improved handling of password protected files for non-English SQL Compact runtime
Some icons were not transparent
CreateDataIfExists (Windows Phone DataContext) now returns bool if database was created
Latest scripting library, with fix for missing SET IDENTITY INSERT with multiple files (from large tables)

Monday, September 5, 2011

Useful Windows Phone advice from Nick Randolph

Fellow MVP Nick Randolph (@BTRoam) publishes an excellent blog, Nick’s .NET Travels, often with articles that relate to SQL Server Compact and synchronization technologies, but also very useful articles for any Windows Phone developer, with a practical, hands-on approach. Highly recommended.

He has recently published the following articles:

Windows Phone LINQ to SQL and the INotifyPropertyChanged and INotifyPropertyChanging Interfaces

This article demonstrates the importance of implementing the INotifyPropertyChanging interface on your DataContext classes to improve memory management. As he points out, you should use SQLMetal to generate your DataContext, as this will avoid missing to implement these interfaces. Or even better use the SQL Server Compact Toolbox Visual Studio add-in, as it adds the following features on top of SQLMetal:

1. Removes unneeded/unsupported constructors

2. Adds any [Index] attributes to each table

3. Adds the CreateDatabaseIfExists method

4. Optionally splits the generated files into a file per table (in next version (2.4), currently available in beta)

Change Tracking with SQL Server Compact (LINQ to SQL) on Windows Phone

This article demonstrates how to get started using SQL Server Compact Change Tracking with a Windows Phone SQL Server Compact database, despite the fact that the Change Tracking APIs are not available on Windows Phone. He also gets thrown in how to use the Windows Phone SDK ISETool to move the database from the Phone (Emulator) to your local disk. Good stuff.

Thursday, September 1, 2011

SqlCeBulkCopy, a library for fast SQL Server Compact INSERTS released

Version 2.1 of my SQL Server Compact Bulk Insert Library has now been released. This library exposes an API similar to the SqlBulkCopy API implemented for the SqlClient (working against SQL Server). The library allows you to quickly load data inot a SQL Server Compact database.

New features in this release include:

3 editions of the library:
One for .NET Compact Framework for version 3.5 databases - ErikEJ.SqlCe.NetCF.dll
One for full .NET Framework for version 3.5 databases - ErikEJ.SqlCe.dll
One for full .NET Framework for version 4.0 databases - ErikEJ.SqlCe40.dll

- New overloads of the WriteToServer method allows you to load any .NET list that implements IEnumerable or IEnumerable<T>

- API Documentation is now available in HTML format here. The API documentation was created very easily using the open source ImmDoc.NET utility. All this command line utility requires is XML comments file and the DLL file(s).

- NuGet package available

A NuGet package, that includes the SQL Server Compact 4.0 library is now available via the NuGet Package Manager.

If you need to load data fast from a DataTable, DataReader or .NET List to a SQL Server Compact database file, this is the library for you.

Monday, August 15, 2011

Major update to SQL Server Compact 3.5 SP2 available

A major update to SQL Server Compact 3.5 SP2 has just been released, disguised as a “Cumulative Update Package”. Microsoft knowledgebase article 2553608 describes the update. The update contains the following product enhancements:

Support for Windows Embedded CE 7.0

The update contains updated device components. This expand the supported device platforms to this impressive list: Pocket PC 2003 Software, Windows CE, Windows Mobile 5.0, Windows Mobile 6, Windows Mobile 6.1 , Windows Mobile 6.5 Professional, Windows Mobile 6.5 Standard, Windows Embedded CE 7.0

Support for Merge Replication with SQL Server “Denali” CTP3

The update contains new Server Tools, that support Merge Replication with the next version of SQL Server, codename “Denali”. The replication components also work with Windows Embedded CE 7.0.

For a list of fixes in the Cumulative Updates released for SQL Server Compact 3.5 SP2, see my blog post here.

It is nice to see that the 3.5 SP2 product, with it’s full range of device support and synchronization technologies is kept alive and kicking.

NOTE: Currently, the only download available is the desktop runtime, I will update this blog post and tweet (@ErikEJ) when the other downloads are available.

Thursday, August 11, 2011

Viewing SQL statements created by Entity Framework with SQL Server Compact

Sometimes it can be useful to be able to inspect the SQL statements generated by Entity Framework against your SQL Server Compact database. This can easily be done for SELECT statements as noted here. But for INSERT/UPDATE/DELETE this method will not work. This is usually not a problem for SQL Server based applications, as you can use SQL Server Profiler to log all SQL statements executed by an application, but this is not possible with SQL Server Compact.

This forum thread contains an extension method, that allows you to log INSERT/UPDATE/DELETE statements before SaveChanges is called on the ObjectContext. I have updated and fixed the code to work with SQL Server Compact 4.0, and it is available in the updated Chinook sample available below in the ObjectQueryExtensions class in the Chinook.Data project.

You can now use code like the following to inspect an INSERT statement:

using (var context = new Chinook.Model.ChinookEntities())
{ 
    context.Artists.AddObject(new Chinook.Model.Artist { ArtistId = Int32.MaxValue, Name = "ErikEJ" });
    string sql = context.ToTraceString();
}

The “sql” string variable now contains the following text:

--=============== BEGIN COMMAND ===============

declare @0 NVarChar set @0 = 'ErikEJ'

insert [Artist]([Name])

values (@0)

; select [ArtistId]

from [Artist]

where [ArtistId] = @@IDENTITY

--=============== END COMMAND ===============

This statement reveals some of the magic behind the new support for “server generated” keys with SQL Server Compact 4.0 when used with Entity Framework 4.0. SQL Server Compact is “tricked” into executing multiple statements in a single call.

Sunday, August 7, 2011

SQL Server Compact Toolbox 2.3–Visual Guide of new features

After more than 44.000 downloads, version 2.3 of my SQL Server Compact Toolbox extension for Visual Studio 2010 is now available for download. This blog post is a visual guide to the new features included in this release

Generate database documentation

This feature allows you to create documentation of all tables and columns in your database, in HTML or XML (raw) format, for use with product documentation etc. If you have added descriptions to database, table or column, these will also be included.

From the database context menu, select Create Database Documentation…

You will be prompted for a filename and can choose between HTML and XML format. The generated document will then open in the associated application (for example your browser).

The format of the HTML and XML file comes from the excellent DB>doc for Microsoft SQL Server CodePlex project. You can use the XML file as the data in your own documentation format.

By default, tables beginning with __ are not included in the documentation (this includes the table with object descriptions). They can optionally be included via a new option:

Please provide any feedback for this new feature to the CodePlex issue tracker

Handle password protected files better

When trying to open a password protected file, where the password is not saved with the connection string, you are now prompted to enter the database password, instead of being faced with an error.

Show result count in status bar

The query editor status bar now displays the number of rows returned.

Other fixes

Improvements to Windows Phone DataContext generation, improved error handling to prevent Visual Studio crashes, and the latest scripting libraries included.

Wednesday, July 13, 2011

Using PowerShell to manage SQL Server Compact database files

I noticed that someone asked if you could manage SQL Server Compact from PowerShell, and yes you can, based on the reply here I put together the following sample:

[Reflection.Assembly]::LoadFile(“C:\Program Files\Microsoft SQL Server Compact Edition\v3.5\Desktop\System.Data.SqlServerCe.dll”)

$connString = "Data Source=C:\data\sqlce\test\roads.sdf"
$cn = new-object "System.Data.SqlServerCe.SqlCeConnection" $connString

# create the command
$cmd = new-object "System.Data.SqlServerCe.SqlCeCommand"
$cmd.CommandType = [System.Data.CommandType]"Text"
$cmd.CommandText = "SELECT TOP (100) * FROM Road"
$cmd.Connection = $cn

#get the data
$dt = new-object "System.Data.DataTable"

$cn.Open()
$rdr = $cmd.ExecuteReader()

$dt.Load($rdr)
$cn.Close()

$dt | Out-Default | Format-Table

Sunday, July 3, 2011

Windows Phone / SQL Server Compact resources

This blog post collects links to relevant blog posts and articles about Windows Phone support for SQL Server Compact. You can also follow me on Twitter (@ErikEJ) to get notified of any SQL Server Compact related news.