Boyan Penev on Microsoft BI » T-SQL

Number of Weekdays Between Two Dates

Boyan Penev — Wed, 04 Aug 2010 04:43:04 +0000

There was an old post here describing some T-SQL code for finding the number of weekdays between two dates, which I wrote. It was working fine, so if you have implemented it you have not done anything wrong. However, Jeff Moden from SQL Server Central has written a post a while ago about this same problem and his implementation is a bit cleaner, and thus I would consider it better than mine. So, here is the link:

http://www.sqlservercentral.com/articles/Advanced+Querying/calculatingworkdays/1660/

Do Business Analysts make good dimensional modellers??

Boyan Penev — Wed, 26 May 2010 08:04:07 +0000

Recently I had the (dis)pleasure of working with Business Analysts, who also thought that they are good in dimensional modelling. so, I had to implement BI solutions (including cubes) on top of their database design. I will show an example (about 95% the same as the actual design), where the idea of letting BAs go into dev territory does not yield the best results:

This “dimensional model” was created by an experienced BA. Some “features” are missing here:
1. The fact table had EffectiveFrom and EffectiveTo dates
2. The relationships between some Dim Tables were 1-1 ?!
3. The Time dim (the only one properly implemented on its own – on the bottom of my example) had columns like: DateTimeName nvarchar(100), DateTimeKey nvarchar(100), YearName nvarchar(100), etc..
4. The Some Tables on the top had nothing to do with the rest (in fact a colleague of mine reckons they are there to fill in the white space on the top of the A3 printout)

Another design, which is better, but still pretty bad showed up after my training on Dimensional Modelling (1hr to go through EVERYTHING, including M2M relationships, Parent-Child hierarchies, Type 2 dimensions, etc):

Obviously, the designer (a developer actually) did grasp some concepts. However, my explanation of a star schema must have been not too clear..

Hope that you had some fun with these two diagrams..and I am sure many developers get in a similar situation, especially when someone else designs their databases. But two points:

1. Ask the BAs to analyse the business and their requirements – not to design the database
2. 1 hour of training on dimensional modelling will not make you an expert

Star-Join Optimisation – Prerequisites

Boyan Penev — Thu, 06 May 2010 04:29:29 +0000

A colleague of mine asked recently: Do we need to have foreign keys between our fact tables and dim tables in order to take advantage of the new SQL Server 2008 star-join optimisation. I decided to ask the question at the MSDN Forums. Just now I got a reply and I thought it may be good to share with everyone:

Charles Wang – MSFT

To use star join, indexes are required on tables. FK constraints are not necessary, but it is recommended that you have FK constraints defined since without it SQL Server must depend on heuristics to detect star schema query patterns. There are some restrictions for heuristics to determine a start join query. And it may not pick up a correct start join query plan under some cases. You can find the detailed information in the section “Star Join Heuristics” of this article, Data Warehouse Query Performance. The following is an extraction of the content:

Many physical designs for data warehouses follow the star schema but do not completely specify the relationships between the fact and dimension tables, as mentioned earlier for foreign key constraints, for instance. Without the foreign key constraints explicitly specified, SQL Server must depend on heuristics to detect star schema query patterns. The following heuristics are applied to detect star join query patterns

1. The largest of the tables participating in the n-ary join is considered the fact table. There are additional restrictions on the minimum size of the fact table. For instance, if even the largest table is not beyond a specific size, the n-ary join is not considered a star join.

2.All join conditions of the binary joins in a star join query have to be single column equality predicates. The joins have to be inner joins. While this might sound restrictive, it covers the vast majority of joins between the fact table and dimension tables on the surrogate key in typical star schemas. If a join has a more complex join condition that doesn’t fit the pattern described above, the join is excluded from the star join. A five-way join, for example, can lead to a three-way star join (with two additional joins later on), if two of the joins have more complex join predicates.

There we go – so we need indexes, while FK constraints would be good to have but not necessary. I also wrote to the CSS SQL Server Team, so if they come back with more information, I will update this post.

SQL Server DBMS Top 1 Wish List

Boyan Penev — Tue, 08 Dec 2009 03:23:00 +0000

As an addition to Teo Lachev’s Top 10 Wishlists (SSAS and SSRS), I would like to contribute only 1 item to a possible SQL Server DBMS wishlist:

1. Source Control.

Not SourceSafe source control, but rather an automated version out-of-the-box, not relying on developers to check in/out. Rather, it should track the changes to the code as they are made, and a full version history should be available directly in the DBMS. It should not be too hard. After all, there is a nice database available, which can store code with its version numbers just like anything else.

This would make a lot of developers’ lives a bit less frustrating.

OK, a SQL code “beautifier” would also be nice, but it is not all that important…

Vote on Connect

Changes in SQL Server 2008 sysadmin group

Boyan Penev — Wed, 17 Dec 2008 10:46:00 +0000

There are some noteworthy changes in the way SQL Server 2008 handles security; apart from the single major improvement – the replacement of the Surface Area Configuration tool by Policy-Based Management.

One thing that surprised me today was that even though I was a Domain Admin and a member of the local Administrators group, SQL Server 2008 refused to let me log in. A login had to be explicitly created so I could access the instance. After some research, I found out that in SQL Server 2008 the local Windows administrators do not get mapped to the sysadmin role. Therefore, it is possible to get locked out of a server instance if there are no sysadmins on it. This is a feature, which separates more clearly SQL Server admins and Windows admins.

A further note on this topic. I would have not lost a small SQL Server war on a Dev environment recently if we were using SQL Server 2008 instead of SQL Server 2005. Now, being a Domain Admin does not necessarily win the battle for SQL Server permissions.

There is a TechNet page describing SQL Server 2008 Security Changes for further reference.

And another one, helping in case all system administrators are locked out.

Spreading Non-Transactional Data Along Time

Boyan Penev — Mon, 08 Dec 2008 12:18:00 +0000

In some cases we need to be able to analyse non-transactional data for discrete periods along a time dimension. An example of such a case is a collection of invoices, which have start and end dates for a period, but are not otherwise connected to a time axis. We may have such invoices with these properties:

Invoice Id
Start Date
End Date
Amount

One of the invoices may be:

Invoice Id: 34821432
Start Date: 2008-10-15
End Date: 2009-03-14
Amount: 15,000.00

and another one:

Invoice Id: 34934221
Start Date: 2008-12-01
End Date: 2009-05-30
Amount: 6,500.00

If the company we are building this for is daily deducting a fee for its services (e.g. funds management, software maintenance, etc.), we may have to be able to spread the amount in smaller periods, like months or days and then aggregate the smaller amounts along a time dimension.

To do this we have to first store the data in a relational table and then write some SQL to do the trick for us.

First, we should create a table valued function which returns all the dates at a specified granularity, such as days, from the Start to the End dates and the count of all the periods in between (in our case is is a count of days):

CREATE FUNCTION udf_Create_Daily_Date_Spread
(
      @Start_Date datetime
    , @End_Date datetime
)
RETURNS @Daily_Spread TABLE (
      Date_Id datetime
    , Count_Of_Days int
)
AS
BEGIN
    DECLARE @Count int
    SET @Count = 0

    IF @Start_Date >= @End_Date
        RETURN

    WHILE @Start_Date <= @End_Date
    BEGIN
        INSERT INTO @Daily_Spread(Date_Id)
        SELECT @Start_Date

        SET @Start_Date = DATEADD(d, 1,@Start_Date)
        SET @Count = @Count + 1
    END

    UPDATE @Daily_Spread
    SET   Count_Of_Days = @Count

    RETURN
END

After having created these functions, we can use the CROSS APPLY statement to create the even spread:

SELECT             Invoice_Id
                        ,Start_Date
                        ,End_Date
                        ,cdds.Date_Id
                        ,Amount/cdds.Count_Of_Days
FROM Invoice_Source inv
CROSS APPLY udf_Create_Daily_Date_Spread(inv.Start_Date, inv.End_Date) cdds

After running the sample data through this code, we will get an even spread for both invoices and we will be able to attach a time dimension to them.

Even though the data size may explode after such a manipulation, Analysis Services provides an excellent way of handling even the largest sets of data. If storage is a problem, we can always choose to break down our data in less periods – instead of days, weeks or months.

Ordering Dimensions: Recursion vs Loops in T-SQL

Boyan Penev — Fri, 21 Nov 2008 00:25:00 +0000

Recently I had to build a function, which returned dimension members from a parent-child relational table. I needed to be able to order the members hierarchically and I decided to use a recursive CTE call.

Usually, we have a parent-child dimension table with structure similar to this:

Dim_Business

Business_Skey int

Parent_Business_Skey int

with a root node with Skey of -3 and Parent_Business_Skey of -3.

Let’s assume we have a hierarchy like this:

All (Id = -3, P_ID = -3)

-GLOBAL (Id = 1, P_ID = -3)

–Europe (Id = 2, P_ID = 1)

—UK (Id = 3, P_ID = 2)

—France (Id = 4, P_ID = 2)

—Spain (Id = 5, P_ID = 2)

–North America (Id = 6, P_ID = 1)

—USA (Id = 7, P_ID = 6)

—Canada (Id = 8, P_ID = 6)

—Mexico (Id = 9, P_ID = 6)

If we do something like:

SELECT Business_Skey

FROM Dim_Business

ORDER BY Parent_Business_Skey ASC, Business_Skey ASC

We will get:

-3 (All)

1 (GLOBAL)

2 (Europe)

6 (North America)

3 (UK)

4 (France)

5 (Spain)

7 (USA)

8 (Canada)

9 (Mexico)

Obviously, this hierarchy is incorrect, because we want to see the leaf nodes under their respective parents.

We can recursively create order, which concatenates the parent ids:

WITH b_ord(bid, bord)

AS

(

SELECT Business_Skey AS bid

, CONVERT(nvarchar(1000), Business_Skey) AS bord

FROM Dim_Business

WHERE Business_Skey = -3

UNION ALL

SELECT Business_Skey AS bid

, CONVERT(nvarchar(1000), bord + ‘|’ + CONVERT(nvarchar, Business_Skey))

FROM Dim_Business db

INNER JOIN b_ord bo

ON db.Parent_Business_Skey = bo.bid

WHERE db.Business_Skey <> bo.bid

)

SELECT *

FROM b_ord

ORDER BY bord ASC

The result of the CTE query is:

-3 -3 (All)

1 -3|1 (GLOBAL)

2 -3|1|2 (Europe)

3 -3|1|2|3 (UK)

4 -3|1|2|4 (France)

5 -3|1|2|5 (Spain)

6 -3|1|6 (North America)

7 -3|1|6|7 (USA)

8 -3|1|6|8 (Canada)

9 -3|1|6|9 (Mexico)

and the order is correct.

Because the code needed to go in a function, invoked by a number of stored procedures, .NET application and various reports, I needed the code to be quick and

light. As some dimensions had a large number of members (50000+), which could grow with time, the code needed to implemented in a careful way. So, I decided

to compare the recursive CTE function to a WHILE loop and a temporary table implementation:

DECLARE @c int

DECLARE @num_of_nodes int

SET @num_of_nodes = (SELECT COUNT(*) FROM Dim_Business)

CREATE TABLE #order(

skey int

, ord nvarchar(1000)

, lvl int

)

INSERT INTO #order

SELECT Business_Skey

, CONVERT(nvarchar(1000), Business_Skey)

, 1

FROM Dim_Business

WHERE Business_Skey = -3

SET @c = 2

WHILE @c > 0

BEGIN

INSERT INTO #order

SELECT Business_Skey

, CONVERT(nvarchar(1000), ord + ‘|’ + CONVERT(nvarchar, Business_Skey))

, @c

FROM Dim_Business db

INNER JOIN #order o

ON db.Parent_Business_Skey = o.skey

AND o.lvl = @c – 1

WHERE db.Business_Skey <> o.skey

SET @c = @c + 1

IF (SELECT COUNT(*) FROM #order) = @num_of_nodes

SET @c = 0

END

SELECT skey AS bid

, ord AS bord

FROM #order

ORDER BY ord ASC

DROP TABLE #order

After comparing the results in Client Statistics and the IO reads, the 1st recursive query performs more than 20% worse than the WHILE loop query. It also trails the non-recursive query in the count of logical reads,read-ahead reads and scan counts.

It seems like in SQL Server 2005 calling WITH recursively does not work as good as coding set-based operations through WHILE loops.

Fun with T-SQL

Boyan Penev — Sat, 15 Nov 2008 02:22:00 +0000

As a tribute to some fairly restrictive .NET design I once fell victim to, I wrote this piece of code:

Use [Database]

Select [Select]

From [From]

Where [Where] = ‘Where’

And [nvarchar(50)] = ‘nvarchar(50)’

And [int] = 1

The CREATE and INSERT statements for the above command:

CREATE TABLE [dbo].[From](

[Id] [int] NOT NULL,

[Where] [nvarchar](50) NOT NULL,

[On] [nvarchar](50) NOT NULL,

[Select] [nvarchar](50) NOT NULL,

[From] [nvarchar](50) NOT NULL,

[int] [int] NOT NULL,

[nvarchar(50)] [nvarchar](50) NOT NULL

)

INSERT INTO [From] ( [Id]

, [Where]

, [On]

, [Select]

, [From]

, [int]

, [nvarchar(50)])

SELECT 1, ‘Where’, ‘On’, ‘:-)’, ‘From’, 1, ‘nvarchar(50)’

Enjoy!

EXECUTE AS someone else – impersonation in stored procedures

Boyan Penev — Tue, 11 Nov 2008 00:55:00 +0000

Recently, I had to implement a stored procedure, which is executed under a service account context. It had to be invoked by an Informatica workflow and its purpose was to clean a cache database table and to update all statistics on a database through invoking the sp_updatestats. As the Informatica user did not have enough permissions to perform these tasks, I had to use the EXECUTE AS in the stored procedure. The EXECUTE AS statement can also effectively be used to limit the execution scope through allowing limited permissions to the user account executing the stored procedure:

CREATE PROCEDURE [dbo].[usp_Update_Statistics]

WITH EXECUTE AS user

AS

BEGIN

clear cache

EXEC sp_UpdateStats

END

Immediately I encountered various problems with the execution of the stored procedure. After some research it turned out that a few preconditions need to be satisfied before we can execute it:

1. TRUSTWORTHY setting on the database needs to be turned on:

     ALTER DATABASE database SET TRUSTWORTHY ON;

2. The owner of both the context database and the master database must be the same. To set this property we need to:

   2.1 Find who is the owner of master:

           exec sp_helpdb

   2.2 Set the context database owner to the same owner:

           ALTER AUTHORIZATION ON DATABASE::database TO user

3. The account under which we EXECUTE AS needs to be a database/server principal. The database and server principals can be found in the sys.database_principals and the sys.server_principals tables.

4. The account needs to be granted impersonate permissions:

     GRANT IMPERSONATE ON USER::user in EXECUTE AS TO user executing the sproc

5. If a server login is specified (instead of a database user), it needs to be mapped to a database user.

Also, if SQL Server is running under a local account/service it is not possible to use the EXECUTE AS statement.

The same statement can be used to execute SQL Server Agent jobs under different context – something particularly useful when trying to run them through Integration Services.