SSIS: How to debug Foreach Loop Container Variable

In today’s blog post, I am going to show you how to debug ForEach Loop Container variable.

Let’s say you fetched a data list from table and want to loop this list to do other tasks item by item. But you want to debug the item value before run other task.

First, Create two variables for Foreach Loop Container:

In the control flow of an SSIS package, add an Execute SQL Task and open the editor. Set the ‘Result Set‘ property to Full result set. Create an OLE DB connection to the database. Add the following T-SQL statement:

SELECT URL as BookListURL,Id as BookCategoryId,PreviousProcessedPageNumber as BookPreviousProcessedPageNumber
FROM URL_BookList

Click the ‘Result Set‘ option to the left. Set the Result Name to 0 and for the Variable Name, select User::BookListURLs. This variable will hold the results returned by the query. Click OK to close the editor.

Next,  add a ForEach Loop container and connect the Execute SQL  task to it. Open the editor and click Collection. In the enumerator drop down menu, select ForEach ADO Enumerator. In the ADO Object Source Variable drop down menu, select BookListURLs and for the Enumeration Mode, select Rows in the first table.

Set up Variable Mappings then click OK to close the editor:

To debug looping item value, you need to add a Breakpoint:

Now, execute the package. The package will stop and a watch window should open at the bottom of the screen. If it doesn’t, click Debug > Windows > Watch 1. In the watch window, add the names of the variables you want to monitor. Keep in mind that the variable names are case sensitive here.

Powershell script sending email (Gmail)

I have a simple powershell project to do file copy and sync task. Can I get email notification if task has issue, like error message or copied file name. The answer is yes, we can us Net.Mail.SmtpClient to send out email in powershell.

Below is the script:

function status_notification{
param( [string]$body,[string] $subject ,[string] $emailTo)
$email =”[email protected]
$psCred = New-Object System.Net.NetworkCredential(“$email”, ” your email password “);
$smtpServer = “smtp.gmail.com”
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer,587)
$smtp.EnableSsl = $true
$msg.IsBodyHTML = $true
$msg.From = “$email”
$msg.To.Add(“$ emailTo”)
$msg.Subject = “$subject”
$msg.Body = “$body”
$smtp.Credentials = $psCred
$smtp.Send($msg)
}

SQL Server Date and Time styles

One of the most frequently question I asked is which date format style number I should to use. Here’s a summary of different date and time style matrix that we can used in T-SQL as part of the CONVERT function.

Matrix of Date and Time style

Style numberInput/Output Format
100 mon dd yyyy hh:miAM (or PM)
1011 = mm/dd/yy
101 = mm/dd/yyyy
1022 = yy.mm.dd
102 = yyyy.mm.dd
1033 = dd/mm/yy
103 = dd/mm/yyyy
1044 = dd.mm.yy
104 = dd.mm.yyyy
1055 = dd-mm-yy
105 = dd-mm-yyyy
1066 = dd mon yy
106 = dd mon yyyy
1077 = Mon dd, yy
107 = Mon dd, yyyy
108hh:mi:ss
109 mon dd yyyy hh:mi:ss:mmmAM (or PM)
11010 = mm-dd-yy
110 = mm-dd-yyyy
11111 = yy/mm/dd
111 = yyyy/mm/dd
11212 = yymmdd
112 = yyyymmdd
113 dd mon yyyy hh:mi:ss:mmm (24h)
114hh:mi:ss:mmm (24h)
 120 yyyy-mm-dd hh:mi:ss (24h)
 121 yyyy-mm-dd hh:mi:ss.mmm (24h)
22 mm/dd/yy hh:mi:ss AM (or PM)
23yyyy-mm-dd
126yyyy-mm-ddThh:mi:ss.mmm (no spaces)
127yyyy-mm-ddThh:mi:ss.mmmZ (no spaces)
130dd mon yyyy hh:mi:ss:mmmAM
131dd/mm/yyyy hh:mi:ss:mmmAM

Sample convert string to datetime

select convert(datetime, ‘2015/12/31 13:58:03’,111)

select convert(datetime, ‘2012-11-07T18:26:20.096Z’,127)

How to compare SQL Server Table schemas

We can use sp_help to display all table schema

sp_help import_OAN_clients

Will get below result:

But how we can compare two tables schema quickly via T-SQL? We can use INFORMATION_SCHEMA.COLUMNS to compare two tables schema easily, but this way cannot compare table cross database or instance. In this blog I will show how to compare it with internal function sys.dm_exec_describe_first_result_set

Create Demo Tables

CREATE TABLE demo1(
	[User_ID] [nvarchar](50) NULL,
	[FirstName] [nvarchar](50) NULL,
	[LastName] [nvarchar](100) NULL,
	[BirthDate] datetime NULL,
	[Citizenship] [nvarchar](100) NULL,
	[Country] [nvarchar](20) NULL
) 
CREATE TABLE demo2(
	[User_ID] [nvarchar](50) NULL,
	[FirstName] [nvarchar](100) NULL,
	[LastName] [nvarchar](100) NULL,
	[BirthDate] [nvarchar](30) NULL,
	[Citizenship] [nvarchar](100) NULL,
	[Country] [nvarchar](100) NULL
) 

How sys.dm_exec_describe_first_result_set is working

If you want to know all detail specs of sys.dm_exec_describe_first_result_set, you can go to microsoft developer community

before we start compare two tables, let’s see what kind of information the function can get

select * from sys.dm_exec_describe_first_result_set ( N’SELECT * FROM dbo.demo1′, NULL, 1)

We can see the returned schema as below:

Doing Comparisons

To do two tables comparison, we just need to simply doing OUTER JOIN between the two function results. (To do comparisons between to database or instance, please use linked servers and full part name to address the table. like product.dbo.demo2)

select a.column_ordinal,b.column_ordinal,a.name,b.name ,a.is_nullable,b.is_nullable ,a.system_type_name,b.system_type_name,a.max_length,b.max_length,a.precision,b.precision,a.scale,b.scale
from sys.dm_exec_describe_first_result_set ( N’SELECT * FROM dbo.demo1′, NULL, 1) as a
full outer join sys.dm_exec_describe_first_result_set ( N’select * from dbo.demo2′, NULL, 1) as b
on a.name = b.name

We can see the comparison result between to tables now.

SSIS: Execute SSIS Package from a SQL Server Stored Procedure

I have a Integration Services Project:SSIS_UK_Projects contains the package LEI_Lookup.dtsx. But how I can run it from Stored Procedure?

1. calls catalog.create_execution to create an instance of execution for the LEI_Lookup.dtsx package.

EXEC [SSISDB].[catalog].[create_execution]
@package_name=N’LEI_Lookup.dtsx’,
@execution_id=@execution_id OUTPUT,
@folder_name=N’UK’,
@project_name=N’SSIS_UK_Projects’,
@use32bitruntime=False,
@reference_id=@env_reference_id

2 calls catalog.set_execution_parameter_value to set values for the parameter, and LOGGING_LEVEL parameters.

Select @execution_id
EXEC [SSISDB].[catalog].[set_execution_parameter_value] @execution_id, @object_type=50, @parameter_name=N’LOGGING_LEVEL’,@parameter_value=1

EXEC [SSISDB].[catalog].[set_execution_parameter_value] @execution_id, @object_type=30, @parameter_name=N’Client_code’, @parameter_value=@parameter1
EXEC [SSISDB].[catalog].[set_execution_parameter_value] @execution_id, @object_type=30, @parameter_name=N’Source_table’, @parameter_value=N’processing_hold’

3. calls catalog.start_execution to start an instance of execution.

EXEC [SSISDB].[catalog].[start_execution] @execution_id

How to install an SSL certificate on Microsoft IIS 10

Before install the SSL certificate, you need to know which kind of certificate you have.

If you certificate file is .cer file then just simply install it via IIS manager. But if have .key file and .crt file then you need to convert the certificate to .pfx file.

Installing certificate via IIS manager

  • go to the Start menu, choose Administrative Tools and select Internet Information Services (IIS) Manager. Otherwise, access it via Win+R >> inetmgr >> OK.
  • Click on the required server name and go to the Server Certificates option in the center menu.

Press the Complete Certificate Request button in the Actions right-side section.

Following the wizard to finish the import step by step. Once imported certificate is shown in the list of Server Certificates. Then you can assign the certificate to the website.

  • pand the Sites menu and choose the website you want to assign the certificate to. After that, click on the Bindings option in the Actions section.
  • In the Site Bindings window, click Add.
  • In the Add Site Binding window, choose the following parameters:

Type – https;

IP address – All Unassigned, or your IP address;

Port – 443;

SSL certificate – friendly name of the imported certificate.

After all details are selected, click OK button.

  • The new binding has been successfully created.

If the site already has https enabled, and if you want to update the SSL certificate, you will need to choose the Edit button in binding for port 443, select a friendly name for the new certificate from the dropdown list and click OK to apply the changes.



Converting to PFX file

If you have the private key in PEM format (.key file), you need to generate the certificate in PKCS#12 format (.pfx).

Use this tool to generate the certificate in PKCS#12. Use your certificate with .crt extension, CA bundle with .ca-bundle extension and the saved key with .key extension.

If there’s an OpenSSL client installed on the server, you can create PFX file out of a certificate in PEM format (.pem, .crt, .cer) or PKCS#7/P7B format (.p7b, .p7c) and the private key using the following commands.

PEM (.pem, .crt, .cer) to PFX

openssl pkcs12 -export -out certificate.pfx -inkey privatekey.key -in certificate.crt -certfile more.crt

*where “more.crt” is the name of the CA Bundle file

SSIS:Truncation may occur due to inserting data from data flow column

I have an SSIS package with a Data Flow that import data from Flat File Source to ADO NET Destination. Output column type of Flat File Source was configured correctly. Input Columns type and External Columns type in ADO NET Destination is automatically generated. But I got below warning messages:

Warning: 0x802092A7 at DFT_import_json, ADO NET Destination Transaction [62]: Truncation may occur due to inserting data from data flow column “reduced_trade_units” with a length of 100 to database column “reduced_trade_units” with a length of 50.

I see the column length of External Columns in ADO NET Destination Input is different with Input Columns length. I manually changed the length to the same as Input Columns length in the Advanced Editor for ADO NET Destination. But It reverted to wrong length again once I closed and open it again. How can I solve it?

After long investigate, I find out it’s related with Destination property setting. By default ADO NET Destination property ValidateExternalMetadata‘s value is true and it will automatically update the external columns type. Need change this property value to false if we wan to manually set external columns type.

How to Setup WordPress Custom Permalinks on Windows IIS

This blog will show how to config WordPress Custom URL Structure under Windows and IIS is how to handle URL Rewrites

Log in to the Admin section of your WordPress

Click on Settings -> Permalinks

Choose URL structure or enter a custom structure

Save below web.config file into WordPress root folder can allow permalinks (or “pretty URLs”) on Windows IIS.

<?xml version="1.0" encoding="UTF-8"?>

<configuration>
  <system.webServer>
    <rewrite>
      <rules>
        <rule name="wordpress" stopProcessing="true">
          <match url="." />
        <conditions>
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true"/>
            <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true"/>
        </conditions>
        <action type="Rewrite" url="/index.php"/>
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>

Once web.config file created, ‘Pretty URL’ will working properly, but if you have images or other kind of static files in the website, you need to add another rewrite rule to stop above rewrite:


<rule name="Redirect Image to HTTP" stopProcessing="true">
<match url=".*\.(gif|jpg|jpeg|png|css|js)$" ignoreCase="true"/>
<action type="Rewrite" url="{R:0}"/>
</rule>

SSIS: Execute SQL Task get Stored Procedure return values

In some project we need to execution stored procedure in SSIS Execute SQL Task and want to capture the stored procedure’s return value. Can we do this in SSIS Execute SQL Task? How can implement it in SSIS?

The answer is yes. The Execute SQL Task configuration is depend on your stored procedure parameter setting and database connection type. For output parameter and stored procedure return value, the SSIS parameter config is different For OLE DB connection and ADO.NET connection config are different as well. In below example, I am using OLE DB connection.

Firstly, we need to create one stored procedure with both output parameter and return value:

create procedure sp_check_import_status @client_code varchar(10),@currenty_day varchar(8),
@output int output
as
begin
declare @result int
select @result = count(1) from transfer_history
where client_code=@client_code and day_transactions >= @currenty_day
set @output = @result
if @result = 1
begin
return @result
end
return 0
end

Secondly, need to config SSIS Execute SQL Task

Double click Execute SQL Task to open task editor. In the editor window, set ResultSet to None and SQLSourceType to Direct Input, input following statement to SQLStatement

exec ? = sp_import_oan_check_import_status ?,?,? output

Switch to the Parameter Mapping window, click Add to add the variables/parameters created in SSIS project and map the parameters with the stored procedure result as follows:

Note that the ParameterName values are numeric. You need to set the name starting from 0 according to the parameters occurrence sequence in the SQLStatement.

SQL SERVER BCP UTF8 import and export, support UTF8 after SQL SERVER 2014SP2

The bulk copy program utility (bcp) is a very powerful bulk copy tool between database and flat file. Normally used for large number of data import and export. But if you sql server version is older then 2014 SP2, unlucklly you cannot process UTF8 data even if there is a code page parameter -C , but it doesn’t support UTF8. Refer to Microsoft community document

code_pageSpecific code page number; for example, 850.

** Important ** SQL Server does not support code page 65001 (UTF-8 encoding).

If you want to process UTF8, then must need to upgrade SQL Server to version 2014 SP2 or later. Once you installed the upgrade, then you can use BCP to import or export UTF8 as below:

To import UTF-8 data to SQL Server, use the BCP utility and run the following bcp t_users in “D:\test.csv” -c -C 65001

To export UTF-8 data to SQL Server, use the BCP utility and run the following bcp t_users out ” D:\test.csv ” -c -C 65001