An alternative to Merge Replication with SQL Server and SQL Server Compact – Zumero for SQL Server

While looking for a migration path for a customer currently using Merge Replication with SQL Server and Windows Mobile, I came across the recently released solution from Zumero, Zumero for SQL Server. As mentioned in my previous blog post, Merge Replication between SQL Server Compact and SQL Server 2014 is no longer possible, and in addition, SQL Server Compact 3.5 SP2 only supports a limited number of client platforms (Windows Mobile/CE and Windows desktop). Microsoft is promoting Azure Mobile Services with Offline support, but for obvious reasons, this solution does not work for on premise databases.

Zumero for SQL Server enables you to synchronize any mobile device with tables on a central SQL Server, using essentially the same components that we know from Merge Replication:

1: Configuration of tables to be synchronized, and added metadata/tracking to those. Before: Using SQL Server Management Studio to create a Merge Publication with articles (tables)
Now: Using ZSS Manager to create a DBFile with Tables

2: An IIS based agent, that exposes a http(s) endpoint for performing the sync process.
Before: Configure Web Synchronization Wizard
Now: ZSS Server

3: Client library for performing sync.
Before: System.Data.SqlServerCe.SqlCeReplication class, part of the SQL Server Compact ADO.NET provider
Now: Zumero Client SDK and SQLite.

using Zumero;

ZumeroClient.Sync(
  "/path/to/sqlite.db",   // full path to local db file
  null,
  "http://MY_ZUMERO_SERVER:8080",
  "test",                 // remote DBFile name
  null,
  null,
  null);

To get started testing out Zumero, simply follow the instructions here: http://zumero.com/get-started/ or start by watching the short, introductory videos here: http://zumero.com/howto/

Notice that Zumero works with any edition of SQL Server 2008 R2 Express or higher/later. Zumero Server is not free, but works with the free SQL Server Express for small scale solutions.

On the client side, the following impressive list client Operating Systems are supported:

Android (native and PhoneGap)
iOS (native and PhoneGap)
Xamarin
Windows, WinRT and Windows Phone 8
Mac OS X
Linux (Ubuntu 12.04+, Fedora 17+)

In my next blog post, I will be doing an interview:  “Hard Talk” with Eric Sink, Zumero founder.

Disclaimer: I am simply a Zumero user, and not affiliated in any way.

Merge Replication with SQL Server Compact 3.5 SP2 and SQL Server 2014 and 2012

Time for a SQL Server 2014 related blog post to celebrate the release of this product. (And to celebrate me entering my 6th year a s Microsoft MVP).

SQL Server 2012

SQL Server Compact based Merge Replication is "supported" by SQL Server 2012, but there are a few "gotchas" that you need to be aware of before implementing this.

This blog post covers these in detail. As you may know, there are 3 components involved in a replication setup: Client, Web Server and Database Server. For each of these components, certain requirements must be met for the solution to work. And each component does not “just work” out of the box.

Client (Windows desktop/server/Mobile)

The client, which can be a Windows desktop or Windows Mobile device, must run a recent build of the SQL Server Compact runtime files. The file version/ServerVersion of the runtime must be 3.5.8088 or later. You can find download locations for the installers of the recent runtime here: http://erikej.blogspot.dk/2010/08/sql-server-compact-35-sp2-downloadable.html

Web Server (IIS)

The SQL Server Compact agent (sqlcesa35.dll) must be version 3.5.8088 or later. You can find download locations for the installers of the most recent Server Agent here: http://erikej.blogspot.dk/2010/08/sql-server-compact-35-sp2-downloadable.html

In addition if you have a separate web and database server, you must install the SQL Server 2012 Database Engine component (and most likely you want to disable the SQL Server service after install) in order to get the required SQL Server COM files installed. 

You must also install SP1 and the latest CU, as a minimum CU5. You can find information about the CUs here: http://sqlserverbuilds.blogspot.dk/

Database Server (SQL Server 2012)

You must install Service Pack 1 and the latest CU (Cumulative Update), as a minimum CU5. You can find information about the CUs here: http://sqlserverbuilds.blogspot.dk/

SQL Server 2014

SQL Server 2014 does NOT support merge replication with SQL Server Compact: http://msdn.microsoft.com/en-us/library/bb500342(v=sql.120).aspx – end of story.

But I am working on a blog post about an alternative solution that “just works” and enables you to Synchronize from any device, including Windows desktop/server, WinRT, Windows Mobile, Windows Phone, Android, iOS, Mac OS X and Linux to SQL Server 2008 R2, 2012 and 2014 (even SQL Server Express). Stay tuned, and follow me on Twitter @ErikEJ and subscribe to my blog RSS feed.

Entity Framework 6 & SQL Server Compact (7) –New features and fixes in version 6.1

Entity Framework 6.1 is now available on NuGet, and this blog post by Rowan Miller, Program Manager for the Entity Framework team, covers the major new features in this release. Most of the new features are also applicable to SQL Server Compact users, for example Code First Model from Database in the EDM Wizard, which is also available with SQL Server Compact if the latest SQL Server Compact Toolbox version is  installed. Rather than repeating the blog post from Rowan, this blog post will look at some of the minor features in EF 6.1 affecting SQL Server Compact. (A few of which I have had Pull Requests accepted)

#824 Designer: DDL Generation Template for SQL CE should separate each statement with GO

This fix allows you to run the script created by the “Generate Database from Model” EF Tools feature in the latest version of the Toolbox, as described here.

#898 Reverse Engineer Code First: Using database name as entity container name causes issues when database name has invalid chars (bad for SQL Compact)

This fix effectively makes “Code First from Database” work with SQL Server Compact, as it was previously broken.

#1322 SqlCePropertyMaxLengthConvention should be updated to reflect changes in the standard PropertyMaxLengthConvention

This fix makes code more portable between database engines

#1852 Migration for EF 6.0 wrong migration script for altering property (required->non required) for SQL CE 4

This fix ensures that NULL or NOT NULL is always specified in migration SQL when changing between [Required] and not required

#1863 Add DbProvider registration to SQL Server Compact NuGet package

This fix enables private deployment of SQL Server Compact, both for use in desktop, web and unit test scenarios (and makes the workaround I describe here obsolete)

#1878 Add SQL Server Compact 3.5 provider + NuGet package

This feature enables you to use SQL Server Compact 3.5 with Entity Framework 6.x, via the EntityFramework.SqlServerCompact.Legacy NuGet package, and includes all the same fixes and features as the SQL Compact 4.0 provider. I blogged about the new provider earlier.

#1962 SQL Server Compact data provider for EF does not support some Entity SQL canonical functions

It turned out, that the current SQL Server Compact data provider only supported the “canonical” functions defined for EF 1.0  – this fix adds support for most applicable functions from EF 4. The following functions are not supported due the SQL Server Compact having a limited number of data types and built-in functions:  StDev, StDevP, Var, VarP, Reverse, CurrentUtcDateTime, CurrentDateTimeOffset, GetTotalOffsetMinutes, CreateDateTimeOffset, CreateTime, AddMicroseconds, AddNanoseconds, DiffMicroseconds, DiffNanoseconds

SQL Server Compact Toolbox 3.7.2–Visual Guide of new features

After more than 260.000 downloads, version 3.7.2 of my SQL Server Compact Toolbox extension for Visual Studio 2013, 2012 and 2010 is now available for download. This blog post is a visual guide to the new features included in this release, many suggested by users of the tool via the CodePlex issue tracker. The focus for this release is a number of features that help improve the experience when using the new support for SQL Server Compact 3.5 (supplied by yours truly) in Entity Framework 6.1 and the related Entity Framework 6.1 Tools.

Run “Generate from Model” script without errors

When using the “Generate Database from Model” option in the Entity Data Model Tools, the tool generates a script to DROP and CREATE the generated objects in the database. If you have tried to run the script until now in the SQL Server Compact Toolbox, the script would fail. A new option “Ignore DDL errors” in the Toolbox SQL Editor will allow you to run the script without any errors:

DDEX install requires VS restart

The simple DDEX providers supplied by the Toolbox for VS 2012 and VS 2013 are installed when the Toolbox is activated, and therefore require VS to be restarted to be used. A notification is now available to notify you of this. You can read more about the DDEX providers in my previous blog post here and here.

(Also, notice the new, “modern” icons in the toolbar of the Toolbox.)

Preserve SQL Server date types

A new option is now available to control scripting of the SQL Server date, datetime2 and datetimeoffset data types. The default behaviour is to always convert these column types to datetime (SQL Server Compact does not have these datatypes). But this can cause data loss (as datetime has limited precision) or overflow errors, as datetime only supports dates from 1753 to 9999, unlike date, datetime2 and datetimeoffset, that all support dates from 0001 to 9999.

Improved About dialog

The About dialog has been improved for readability, and contains a counter for the current number of Toolbox downloads:

Bug fixes

EDMX generation feature failed if only VS 2012 and not VS 2013 was installed.

"Refresh" of tables, including refresh after schema changes, was broken

Scripting API improvements

- improved SQLCE runtime check

- datetimeoffset columns scripted as datetime by default now, not nvarchar()

- sql_variant and hierarchyid handling improved, to avoid getting wrong ordinal for identity and timestamp

- sqlite indexes are now always given unique names

The scripting API improvements are also included in the latest version of my SQL Compact command line export tool and Scripting API, available here.

Feedback

As usual, please provide feedback if you have any feature requests, ideas or encounter any issues (or even bugs!!) by using the CodePlex Issue tracker.

Entity Framework 6 & SQL Server Compact (6)–Entity Framework Reverse POCO Code First Generator

The Entity Framework Reverse POCO Code First Generator is a customizable T4 template, that reverse engineers an existing database and generates Entity Framework Code First POCO classes, Configuration mappings and DbContext It is available directly in Visual Studio via the Tools, Extensions and Updates menu item. It can be used with VS 2010 or later, and the latest version supports Entity Framework 6 and both SQL Server and SQL Server Compact (both 3.5 and 4.0).

This T4 template is similar to the Entity Framework Power Tools “Reverse Engineer Code First” feature, but much more versatile and flexible, and constantly updated by the developer. You can see how this project compares with the Power Tools feature here.

To get started using the template with SQL Server Compact, follow these simple steps:

Install the relevant Entity Framework NuGet package:

• EntityFramework.SqlServerCompact.PrivateConfig (EF 6.0.2 with SQL CE 4.0)
• EntityFramework.SqlServerCompact (EF 6.1 when released with SQL CE 4.0)
• EntityFramework.SqlServerCompact.Legacy (EF 6.1 when released with SQL CE 3.5 – currently available as beta)

Add a connection string to your app.config/web.config:

<connectionStrings>
   <add name="MyDbContext" 
    providerName="System.Data.SqlServerCe.4.0"
    connectionString="Data Source=C:\data\mydb.sdf" />
</connectionStrings>

In Visual Studio, right click project and select "add - new item".

Select Online, and search for "reverse poco". 

Select the "EntityFramework Reverse POCO Code First Generator" template

Give the file a name, such as Database.tt and click Add.

Edit the Database.tt file and specify the connection string name as "MyDbContext" which matches your name in app.config.

Save the Database.tt file, which will now generate the Database.cs file.

There are many options you can use to customise the generated code. All of these settings are in the Database.tt files with comments attached.

SQL Server Compact Toolbox 3.7.1–Visual Guide of new features

After 250.000 (!) downloads, version 3.7.1 of my SQL Server Compact Toolbox extension for Visual Studio 2013, 2012 and 2010 is now available for download (and available via Tools/Extensions in Visual Studio). This blog post is a visual guide to the few new features included in this minor release, which all centre around Server Explorer (DDEX) , including support for Visual Studio 2012 and 2013.

 

Simple DDEX providers for Visual Studio 2012 and 2013

I already blogged about this feature here, it has been extended to also support SQL Compact 3.5. Follow these steps to use the providers with Entity Framework Tools:

- Install this version of the toolbox and launch Visual Studio 2012/2013.
- Open the Toolbox (this will silently run the “installation” of the DDEX provider if required).
- Restart VS 2012/2013.
- Add new Toolbox connection to a new or existing SQL Server Compact 4.0 or 3.5 database file.
- Add EntityFramework.SqlServerCompact or EntityFramework.SqlServerCompact.Legacy NuGet package to project and build project.
- You can now use this connection for Generate Model from Database, Generate Database from Model and Update Model from Database, and with the Power Tools.

With this release, the DDEX support matrix looks like this:

	SQL Server Compact 3.5	SQL Server Compact 4.0
Visual Studio 2010	Microsoft	None
Visual Studio 2010 SP1 + SQL CE 4.0 Tools	Microsoft	Microsoft
Visual Studio 2012	ErikEJ	Microsoft
Visual Studio 2013	ErikEJ	ErikEJ

“Microsoft” means full featured, Microsoft supplied provider
”ErikEJ” means simple, basic support, enough to enable Entity Framework Tools, provided by this version of SQL Server Compact Toolbox.

Currently, the 3.5 DDEX provider does not fully work with the Entity Framework Tools, due to a bug (my bug!) in the EntityFramework.SqlServerCompact.Legacy Entity Framework provider; I am working on getting it fixed.

Scan Solution for sdf files

This new menu item will scan the active Solution for any sdf files, and add those to the Toolbox:

Prefer DDEX

This new option allows you to always use the Toolbox supplied dialogs for connecting to a SQL Server Compact database file, which will allow you to connect to any file, and does not require an .sdf file type. (By choosing not to prefer!)

Bug fixes and enhancements

Add connection will now always use DDEX provider if available
Handle apostrophe in file and folder names when saving connections
Fixed error: The specified table does not exist. (Databases)
Make it clearer that sqlmetal dependent features requires 3.5 to be installed

Please report any issues you encounter and provide feature suggestions and requests here.

Entity Framework 6 & SQL Server Compact 4.0 (4) - Restoring full EF Tools support in Visual Studio 2013

As described in my blog post here, Entity Framework Tools support for SQL Server Compact 4.0 in Visual Studio 2013 is broken, because SQL Server Compact is no longer supported by Server Explorer in VS 2013. The latest release of my SQL Server Compact Toolbox, version 3.7.1, currently available in beta release from here, restores the Entity Framework Tools support for SQL Server Compact 4.0 by installing a simple DDEX provider.

Developing DDEX providers is a bit of a dark art, which has put me off even thinking about it, but a recent discussion on CodePlex drew my attention to the DDEX sample provider included with the Entity Framework source code. In particular the fact that the Entity Framework Tools does not get any information from the DDEX provider apart from the connection information, so all Tables, Columns etc. are enumerated via code in the Entity Framework provider, not the DDEX provider. Quote from the included Word document: “Thus, the information shown in the wizard GUI comes directly from the EF provider. The wizard lets users select which tables, views and stored procedures to include in the model”.

So my “simple” provider enables you to use the Entity Framework Tools, for example “Generate Model from Database”, “Generate Database from Model”, “Update Model from Database” and also the EF Power Tools Reverse engineer feature. (Soon to be included directly in the EF Tools).

Once the DDEX Provider has been installed, (which consists of placing the provider DLL on disk, and add a number of registry entries), you can add a connection to any SQL Server Compact 4.0 database from the Toolbox, and it will be added to Server Explorer:

(If you do not see the label at the bottom, but potentially a warning instead, one of the requirements for using the provider have not been fulfilled)

The requirements for the provider are:
- Visual Studio 2013 Pro or higher (it is not possible to add 3rd party DDEX providers to any Express edition, please correct me if I am wrong)
- The Simple DDEX provider must be registered (may require a VS restart after launching with version 3.7.1 the first time), you can verify via About dialog in Toolbox.
- The 4.0 DbProvider must be properly registered, use the About dialog in the Toolbox to confirm:

Finally, you can of course confirm the presence of the provider in the Server Explorer Choose Data Source dialog:

To connect via Server Explorer, select the “SQL Server Compact 4.0 (Simple by ErikEJ)” data source, click continue, and all you normally would need to enter is the path to the database file:

Notice that this UI does not let you create a new database file, but you can do that from the “Add Connection…” dialog in the Toolbox.

With that in place, you have access to the exact same Entity Framework Tools features that you had in Visual Studio 2012. Happy coding (and please let me know if your encounter any issue with this feature)