Showing posts with label dataset. Show all posts
Showing posts with label dataset. Show all posts

Thursday, March 29, 2012

ForEachLoop Container and Variables

Hi Guys

I am trying to do the following and am quite new to SSIS.

I have to select a dataset from a database on server A, check if it exists on server B and perform an Update or Insert dependant on the existence.

I have created a SQL task to do the Select from server A with the results set passed to a variable of Vendors. I have added a ForEach Loop container with an enumerator of Foreach ADO Enumerator and the source variable is set to Vendors.

I have created 2 variables in the Foreach Loop called Code and Supplier - both as strings - as there are 2 fields from the initial Select that need to be passed to the final Update/ Insert.

I have then created another SQL task insert the Foreach which will perform the Update/Insert.

obviously when I run it at the moment it performs the Update/ Insert but just adds the rows with both Code and Supplier as NULL.

having looked at a couple of examples in books I have i know i need to add something in the Expressions of the Update/Insert SQL task but it is here i get a bit lost.

Which of the properties from the drop down do i need to use to map the variables against?

Any help would be massively appreciated asI am tearing my hair out!

Thanks

Scott

Hi Scott,

We're all still learning SSIS.

It sounds like you're most of the way there.

There are a couple ways to approach this solution. The simplest way, from what I understand from your post, is to use placeholders and parameters in your Update/Insert statements. If you already have the Code and Supplier variables defined, you could perform an insert using an Execute SQL Task with something similar to the following code:

Code Snippet

INSERT INTO Vendors

(Code, Supplier)

VALUES(?, ?)

You could then supply Parameters:

Code Snippet

VariableName Direction DataType ParameterName ParameterSize

User::Code Input Int 0 -1

User::Supplier Input VarChar 1 -1

This would substitute the question marks in the SQL Statement property with the values contained in your variables.

Hope this helps,

Andy

|||

Scott,

Any special reason for not using a dataflow with a lookup transform to detect if the rows exists(update) or not (insert). That is by far a pretty common practice in these scenarios.

|||

Hi Rafael

Still new to this (and database stuff as a whole) and am going on someone elses advice!

I have looked at your suggestion and have got as far as the following:

OLEDB Source with a SQL select statement to return the data required

Look Up transform to look up the 2 columns from the Select against the destination table

After that I am a bit lost. I guess i have to add a OLEDB destination but do I do it to a table or a SQL Command?

thanks again

Scott

|||

I think you are on the right track. I would add an OLE DB Destination against the destination table.

Keep in mind you have to tweak the lookup to 'redirect' errors. Lookup will treat the no matches as errors; hence will be send to the error output of the component (red arrow). Then you have to connect the error output of the Lup to the input of the destination.

Now the updates; every row going to the green output of the L.up is an existing/to-updated row. Here you have 2 options; use an OLE DB Commnad to update the row in the destination table; or send those rows to an estiging table (yes a seconf OLE DB Destination) and then back in control flow use an Execute SQl task to do a 1 time update. The advantage of the second method is performance. the Update runs 1 time updating all the required rows. The First one will perform an update for every row passing trhough; wich depending on the volume of data can be performance killer; the good thing is that you don't need a second table.

This thread has some examples

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1211340&SiteID=1

Tuesday, March 27, 2012

Foreach Loop is killing me!!!!!!!!!!!!!!!!!!!!!!

I would like your help with the Foreach loop container. Boy, am I having issues with using it for looping through an ADO.NET dataset or what!?!? My control flow has a data flow task that is executing a Data Reader task (creating a .NET dataset for me). Now I go back to the control flow and add a Foreach loop container to loop through each record in my dataset. But which type of the Foreach loop container should I be using? I see an option to use Foreach ADO.NET schema enumerator but not sure how to configure it. Also I used the Foreach ADO option, but setting up variables to go through every column is an absolute pain in the you know what!!?!! I have about 200 columns and I want an easier way to refer to those columns in my transformation phase.

Hope you are able to help me out here.

Jamie posted an answer on another thread!!?!!?!!?!!?!!!!!!!!?!!!!!

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=424137&SiteID=1

Wednesday, March 7, 2012

FOR XML AUTO, ELEMENTS

Hi,
Is it possible to include an attribute within the dataset returned using FOR
XML AUTO, ELEMENTS ?
The format returned when using ELEMENTS is perfect for what I need but I
just need to make the ID field an attribute rather than an element.
Grateful for any advice.
Alex
I am not an expert, but according to the docs you cannot do that with
AUTO. You will need to do it with RAW.
|||Actually, RAW (in SQL Server 2000) will return everything as attributes, and
as I understand it you want a mix of attributes and elements. To do that
you'd need to use EXPLICIT mode, as in this example:
USE Northwind
SELECT 1 AS Tag,
NULL AS Parent,
ProductID AS [Item!1!ProductID],
ProductName AS [Item!1!Name!element],
UnitPrice AS [Item!1!Price!element]
FROM [Products]
FOR XML EXPLICIT
Cheers,
Graeme
Graeme Malcolm
Principal Technologist
Content Master
- a member of CM Group Ltd.
www.contentmaster.com
"ali" <ali.jan@.gmail.com> wrote in message
news:1121413418.618949.238900@.g14g2000cwa.googlegr oups.com...
I am not an expert, but according to the docs you cannot do that with
AUTO. You will need to do it with RAW.
|||NB: I'm assuming you're using SQL Server 2000 - in SQL Server 2005 you could
use PATH mode.
Graeme Malcolm
Principal Technologist
Content Master
- a member of CM Group Ltd.
www.contentmaster.com
"Graeme Malcolm" <graemem_cm@.hotmail.com> wrote in message
news:O81hWwRiFHA.3544@.TK2MSFTNGP15.phx.gbl...
Actually, RAW (in SQL Server 2000) will return everything as attributes, and
as I understand it you want a mix of attributes and elements. To do that
you'd need to use EXPLICIT mode, as in this example:
USE Northwind
SELECT 1 AS Tag,
NULL AS Parent,
ProductID AS [Item!1!ProductID],
ProductName AS [Item!1!Name!element],
UnitPrice AS [Item!1!Price!element]
FROM [Products]
FOR XML EXPLICIT
Cheers,
Graeme
Graeme Malcolm
Principal Technologist
Content Master
- a member of CM Group Ltd.
www.contentmaster.com
"ali" <ali.jan@.gmail.com> wrote in message
news:1121413418.618949.238900@.g14g2000cwa.googlegr oups.com...
I am not an expert, but according to the docs you cannot do that with
AUTO. You will need to do it with RAW.

Sunday, February 26, 2012

For security reasons DTD is prohibited in this XML document and System.OutOfMemory

I am getting this error while running a very large dataset. Please help..

The full description of the error is : "For security reasons DTD is prohibited in this XML document. To enable DTD processing set the ProhibitDtd property on XmlReaderSettings to false and pass the settings into XmlReader.Create method."

I am having hard time figuring out a solution for this. I am using sql 05, SSRS 05, Report viewer

I have a very big Dataset which pulls up millions of rows.

When I pass parameters from Windows forms to run on Reportviewer through webservice it is giving me "System.OutOfMemory" Exception and when I pass same parameters on the server(i.e. http:\\server\reportmanager) it is giving me this error "For security reasons DTD is prohibited in this XML document. To enable DTD processing set the ProhibitDtd property on XmlReaderSettings to false and pass the settings into XmlReader.Create method".

I have no problem when I pass small dataset.

Please help me solve this annoying this problem.

|||

I found this article and i guess this should answer your question.

http://support.microsoft.com/default.aspx/kb/909678

|||

Thanks Chaitanya for the informative link.

I've already checked that and done necessary changes for the long running reports.

Right now I was able to print 55541 pages. We decided to take printout in batches if the requirement more than 55541. They best possible solution would be schedule a report and call it from Windows service using webservices.

Please let me know if we can print unlimited pages(i.e about 3 million pages).

For security reasons DTD is prohibited in this XML document

I am getting this error while running a very large dataset. Please help..

The full description of the error is : "For security reasons DTD is prohibited in this XML document. To enable DTD processing set the ProhibitDtd property on XmlReaderSettings to false and pass the settings into XmlReader.Create method."

I am having hard time figuring out a solution for this. I am using sql 05, SSRS 05, Report viewer

I have a very big Dataset which pulls up millions of rows.

When I pass parameters from Windows forms to run on Reportviewer through webservice it is giving me "System.OutOfMemory" Exception and when I pass same parameters on the server(i.e. http:\\server\reportmanager) it is giving me this error "For security reasons DTD is prohibited in this XML document. To enable DTD processing set the ProhibitDtd property on XmlReaderSettings to false and pass the settings into XmlReader.Create method".

I have no problem when I pass small dataset.

Please help me solve this annoying this problem.

|||

I found this article and i guess this should answer your question.

http://support.microsoft.com/default.aspx/kb/909678

|||

Thanks Chaitanya for the informative link.

I've already checked that and done necessary changes for the long running reports.

Right now I was able to print 55541 pages. We decided to take printout in batches if the requirement more than 55541. They best possible solution would be schedule a report and call it from Windows service using webservices.

Please let me know if we can print unlimited pages(i.e about 3 million pages).

Sunday, February 19, 2012

For each loop

Hi,

I have a dataset created in a data flow task. I want use a for each loop in the control flow area of the package to process every row individually in the dataset. How can I do this?

I am currently using a "Recordset Destination" because it says that it creates and populates in memory ADO dataset and I read somewhere that this could be used anywhere in the package. I have even assigned a variable DS which has an object datatype to the Recordset Destination task. How do I use this now in the for each loop container in the Control Flow task? I also need to know if I can use the DataReader Destination task instead as it uses ADO.NET vs just ADO for performance reasons?

Please help me with the above problem as I am working on a demo.

Thank you!!

Here's how you do it: http://blogs.conchango.com/jamiethomson/archive/2005/07/04/1748.aspx and there is a downloadable demo on there as well

The only difference is that in the example given here the recordset object variable is populated with an Execute SQL Task rather than the Recordset destination but thereafter the same applies.

You cannot use the DataReader destination by the way!

-Jamie