Archiving Large table

Hi,

I am having problems with archiving. Any help will be greatly appreciated.

I have a table with 12 million records, and it needs to be archived to another table.

Since there is a primary key(let's say, SessionID), so I tried to copy 1000 records at a time, and delete those records afterwards.

Problem is, it used to take 1 second to process 1000 records. Now, it takes anywhere from 2 minutes to 14 minutes!!!

Does anyone have a better idea of doing this? I am really stuck...RE: I am having problems with archiving. Any help will be greatly appreciated. I have a table with 12 million records, and it needs to be archived to another table. Since there is a primary key(let's say, SessionID), so I tried to copy 1000 records at a time, and delete those records afterwards. Problem is, it used to take 1 second to process 1000 records. Now, it takes anywhere from 2 minutes to 14 minutes!!! Does anyone have a better idea of doing this? I am really stuck...

Q1 [It used to take 1 second to process 1000 records. Now, it takes 2 minutes to 14 minutes. Why?]

A1 There may be many different issues, (insufficient information to suggest a reasonably good guess and / or answer).

Q2 [Does anyone have a better, i.e.(FASTER?) idea of doing this?]

A2 Archiving may be accomplished efficiently. What is "Better" really depends on existing overall constraints and designs, available resources, and the details of the circumstances. An example that should be fairly quick (but not overly 'user friendly') would be Archiving a Test table in a Demo DB to an ArchiveDB database table named ArchiveTest.

Demo..Test To ArchiveDB..ArchiveTest

Use Demo
Go

INSERT INTO
[ArchiveDB].[dbo].[ArchiveTest]
([Parent], [Child])
SELECT
[Parent], [Child]
FROM
[Demo].[dbo].[Test]
GO

Alter Database Demo
Set Restricted_User
With
RollBack Immediate
Go

Alter Database Demo
Set Single_User
With
RollBack Immediate
Go

Alter Database Demo
Set Recovery Simple
With
RollBack Immediate
Go

-- drop and recreate, or truncate, delete, etc.
Drop TABLE [Test]
Go
CREATE TABLE [Test] (
[Parent] [varchar] (50) NOT NULL ,
[Child] [varchar] (50) NOT NULL)

Alter Database Demo
Set Recovery Full
With
RollBack Immediate
Go

Alter Database Demo
Set Multi_User
With
RollBack Immediate
Go|||how many indexs do you have on this table? Are any of them clustered?

I would suggest
1. copying all data to your archive table
2. script out all indexes and then drop them
3. build one non clustered index that would allow you to join to the archive table.
4. begin a transaction, delete a few thousand records, commit the transaction.
5. adjust the number of deleted records for best performance
6. restore indexes from step 2.|||Originally posted by Paul Young
how many indexs do you have on this table? Are any of them clustered?

I would suggest
1. copying all data to your archive table
2. script out all indexes and then drop them
3. build one non clustered index that would allow you to join to the archive table.
4. begin a transaction, delete a few thousand records, commit the transaction.
5. adjust the number of deleted records for best performance
6. restore indexes from step 2.

Thank you for your replies,

Actually, there is only primary index with identity on. That's it.
The only problem is that, this table should be on-line all the time. i cannot restrict the access to this table.

Somehow, the records don't seem to be sorted at all when I open the table. I tried to add sort(desc) option on the table, and it seemed to be working. However, after a couple of archiving procedure run, the performance gets worse. If I open the table again, it is again a mess. I don't see sorted order in this table.

Once it is properly sorted, the performance is great. What can I do to keep the old record + new records sorted at all times? I cannot manually sort the table, and this process hurts the server badly.|||RE: Thank you for your replies, Actually, there is only primary index with identity on. That's it. The only problem is that, this table should be on-line all the time. i cannot restrict the access to this table. Somehow, the records don't seem to be sorted at all when I open the table. I tried to add sort(desc) option on the table, and it seemed to be working. However, after a couple of archiving procedure run, the performance gets worse. If I open the table again, it is again a mess. I don't see sorted order in this table. Once it is properly sorted, the performance is great. What can I do to keep the old record + new records sorted at all times? I cannot manually sort the table, and this process hurts the server badly.

Q1 [I tried to add sort(desc) option on the table, and it seemed to be working. However, after a couple of archiving procedure run, the performance gets worse.]
A1 You are probably not updating your indexes at a suitable interval (to ensure optimal performance).

Q2 What can I do to keep the old record + new records sorted at all times?
A2 Cluster both TABLES on the desired column.|||my first inclination is that you have a corrupted index. the overall sorting should not change (aside from changes in data) due to inserting, updateing or deleting data.

During a one week prieod I rebuilt all my index once if not twice on very dynamic tables. Are you doing this?

If your primary index is clustered, you are reordering some part of your data everytime you insert, update or delete. This can lead to slow performance at times. If you must have this index then you just live with it, if you don't need it then change to non-clustered.|||Thank you, Paul Young.

Actually, I never rebuilt indexes on any of the tables. My bad...

What do I have to do to rebuild indexes? I tried DBCC DBREINDEX, and it didn't improve the performance of the archiving.

Can you guide me step-by-step what has to be done?

Thank you again.|||Generally I use maintinance plans to rebuild indexes and statistics along with other things however to answer your question DBCC DBREINDEX will do the trick.

Even if you haven't EVER rebuilt your index(s) they still should produce a result set correctly sorted. Again My hunch is that you have a corrupt index. To fix this you will need to drop the index an re-create it. You can do this while other are using the system but I would NOT advise it.|||One more thing, once you rebuild your index you probably want to update statistics so the correct optimization plans will be used.|||Thank you, Paul.

I tried to rebuild the index using 'DROP EXISTING'. It took about 5 minutes, and I opened the table, and it still looks messy.

But now, the index seems to be functioning faster. The problem is that, I don't use the primary key as query condition. Usually, my query condition is the 'CreationTime' which gets filled with default values getdate().

Basically, I query all the data created during a time period.

Should I create another index on CreationTime?

Thank you,|||If this table is used in an OLTP environment you want to keep the number of indexes to a minimum because evryting you inset/update/delete a row you also have to update ALL indexes. In your case you only have one index so adding one more shouldn't cause you a noticable slowdown and will GREATLY improve the prformance of your select.

before adding any indexes drop your select statement into Query Analyzer, turn on Show Execution Plan, Show Server Trace and Show Client Statistics and execute your select. Look at the "Execution Plan" tab and you will get a diagram representing what your select is doing.

Next click on Index Tunning Wizard and let SQL server suggest indexes to be built. Concider the suggestions and implament whatever you tinks looks good. Now rerun your select and look at the differences on the "Execution Plan" tab.

All of this is covered in Books Online, an excelent source of info once you know what to look up.|||Q1 Usually, my query condition is the 'CreationTime' Should I create another index on CreationTime?

A1 If you are looking for good performance, Yes. Generally, one wants a (well maintained) index available for the query parser to take advantage of for any column that is frequently queried.

arabic characters are not saved (was "A serious problem ...- plz. help"

Dear all ..
i have a serious problem & all ur comments will be appreciated..

i have bought an ASP .NET publishing tool which i receieved an sql script with it to execute on either Ms sql server or MSDE .

i executed it on MSDE as i don't have Ms sql server on my windows dedicated server .

I wanted the tool for publishing (Arabic Language)content for a highly traffic soccer website..

After executing the sql script i tested the tool but i found arabic characters are not saved when i add articles .. they were saved as question marks (??).

so i re-executed the sql script on a new db after modifying every code containg (varchar) to (nvarchar) to support unicode & thus arabic.

it worked & i succeeded in saving arabic articles

BUT >>>>>>>>>>>>>

i found that only short arabic articles r saved fine while any article that reaches around (1 microsoft word page) is not saved well with arabic characters but saved as question marks ( ? ) .. !!

=======

so i checked the db tables using ASP.NET Enterprise manager & i found that

the field of article has (ntext) & infront of it number 16 ..

it seems that the (ntext) has a limit to wt it can save ..

so i believe there's a way which i don't know to make the (ntext) accepts long articles entry .

========

Here's the code in the original sql script i received with the tool & i hope u can guide me in details to any modification to do so that the ntext limit is raise to save any long arabic article.

========

code :

CREATE TABLE [dbo].[xlaANMarticles] (
[articleid] [int] IDENTITY (1, 1) NOT NULL ,
[posted] [nvarchar] (50) NOT NULL ,
[lastupdate] [nvarchar] (50) NOT NULL ,
[headline] [nvarchar] (350) NOT NULL ,
[headlinedate] [nvarchar] (255) NOT NULL ,
[startdate] [nvarchar] (50) NOT NULL ,
[enddate] [nvarchar] (50) NOT NULL ,
[source] [nvarchar] (255) NOT NULL ,
[summary] [nvarchar] (3000) NOT NULL ,
[articleurl] [nvarchar] (1000) NOT NULL ,
[article] [ntext] NOT NULL ,
[status] [tinyint] NOT NULL ,
[autoformat] [nvarchar] (50) NOT NULL ,
[publisherid] [int] NOT NULL ,
[clicks] [int] NOT NULL ,
[editor] [int] NOT NULL ,
[relatedid] [nvarchar] (2000) NOT NULL ,
[isfeatured] [nvarchar] (10) NULL ,
[keywords] [nvarchar] (255) NULL ,
[description] [nvarchar] (255) NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

i highlighted the specific column for the article field with red color ..

i tried making it (nvarchar) but the sql manager i use saud it can't be done coz. there has to be a field for TEXTIMAGE coz. it's set to be (on) in the code.

waiting for ur help plz. ... i am desperate .. :confused:Masry-

I maximum length of NTEXT is 1,073,741,823 characters. I can't imagine a page of MS word document holding more than that limit. In addition, BOL says prefix the unicode character strings with N. You might want to try with N. Not sure what 16 means.|||The problem is that saving Arabic characters (like any unicode or 16 bit characters) requires a full 16 bit data path from begining to end. If any part of the path reverts back to 8 bit characters, then any character that isn't supported by the 16 bit to 8 bit translation will appear as a question mark.

Apparently something in the data path that handles the translation from Word documents larger than a given (roughly one page) threshold to a database column causes the data to pass through as 8 bit characters. The problem lies in isolating whatever that weak point is!

-PatP|||Masry,
BLOB values are not saved in row, but elsewhere in data file, unless you use sp_tableoption. The 16 in front of ntext column is only a pointer to where the actual ntext value is located.
But about your problem, I suggest to check the path that your data is traversing to be saved to DB. There might some variables or ... be used that cause the Unicode information be lost!
Regards,
Leila

Append Data

Hi,
Appreciated it if someone could help me with regards to
the abovesaid problem.
I don't know how to append data using SQL Query Analyzer
from table a to table b because it will create duplicate
table b.recordnumber(auto number) in table b. I tried to
use DTS but failed due to the duplicate record number(auto
number).
tq.
Hi,
Use "SET IDENTITY_INSERT" to allow explicit values in acolumn with identity.
See SET IDENTITY_INSERT in books online. There is good example as well which
shows the usage.
Sample:-
create table id1(i int identity, k char(10))
create table id2(i int identity, k char(10))
go
insert into id1(K) values('hari')
insert into id2(K) values('raj')
go
select * from id1
select * from id2
go
set identity_insert id1 on
insert into id2(k) select k from id1
go
select * from id2
Thanks
Hari
MCDBA
"GW" <anonymous@.discussions.microsoft.com> wrote in message
news:657801c4753d$a64178e0$a501280a@.phx.gbl...
> Hi,
> Appreciated it if someone could help me with regards to
> the abovesaid problem.
> I don't know how to append data using SQL Query Analyzer
> from table a to table b because it will create duplicate
> table b.recordnumber(auto number) in table b. I tried to
> use DTS but failed due to the duplicate record number(auto
> number).
> tq.

Aparent DataReader collisions/conflicts between users

This is causing me such grief! Any help would be MUCH appreciated.

Basic summary of the problem:

I have an Intranet site I have developed that uses a SQL Server backend. It uses SqlConnection and SqlDataReader objects to get data form the database.

When more than one person uses the site at the same time, conflicts occur. It seems that the users' processes are attempting to share the SqlDataReaders, which results in problems because one user tries to open the SqlDataReader but it is already open by the other user, and similarly when they try to close a DataReader.

Full details are as follows:

SQL Server 7 backend database

webserver running Windows Windows 2003 server 5.2 SP1

IIS 6.0

.Net version 1.1 1927

Aparent collisions are occuring when more than one person works on the website at the same time.

I am using SQLConnection and SQLDataReader objects for data access from the database, a typical example being:

in Global.asax.vb:

Application("SQLString") = "workstation id={WebserverName};packet size=4096;user id=sa;password={password};data source={SQLServermachine};persist security info=False;initial catalog={DatabaseName}"

in a local class module:

Dim SQLConnection_docstatustable =New SQLConnection(HttpContext.Current.Application("SQLString"))

PublicFunction GetAll()As DocStatuses

'Returns a collection of all the Document Statuses from the database

Dim TheseDocStatusesAsNew DocStatuses

Dim commandAsNew SqlCommand("select * from lkDocStatus where Deleted = 0", SQLConnection_docstatustable)

Dim drAs SqlDataReader

Try

If SQLConnection_docstatustable.State = ConnectionState.ClosedThen SQLConnection_docstatustable.Open()

dr = command.ExecuteReader(CommandBehavior.SingleResult)

While dr.Read()

TheseDocStatuses.Add(PopulateItem(dr))

EndWhile

Finally

IfNot drIsNothingAndAlsoNot dr.IsClosedThen

dr.Close()

EndIf

SQLConnection_docstatustable.Close()

command.Dispose()

EndTry

Return TheseDocStatuses

EndFunction

The above function uses the following function to read each record as it loops through the datareader:

Function PopulateItem(ByVal drAs IDataRecord)As DocStatus

Dim thisDocStatusAsNew DocStatus

thisDocStatus.DocStatusID = Convert.ToInt32(dr("DocStatID"))

IfNot dr("DocStatus")Is DBNull.ValueThen

thisDocStatus.DocStatusString = Convert.ToString(dr("DocStatus"))

EndIf

Return thisDocStatus

EndFunction

This example is a very small one, but all my database queries are done in pretty much the same way. The main difference is that most include a lot more fields.

I have consistently used try/finally blocks, and closed both the SQL Connection and the DataReader in there, so that everything is being tidied up for each specific user.

Depending on precisely what point the two users are at when the problem happens, I get various different .NET error messages, but they all seem to boil down to collisions between the different users, with particular regard to DataReaders.

Some error messages I have received:

Object reference not set to an instance of an object - pointing to adr = command.ExecuteReader(CommandBehavior.SingleResult) statement (I take it to mean the other user's process has just closed this DataReader).

There is already an open DataReader associated with this Connection which must be closed first - pointing to the same place (I take it to mean the other user's process has just opened this DataReader).

ExecuteReader requires an open and available Connection. The connection's current state is closed - pointing to the same place again (I assume the other users's process has just closed the Connection).

Also:

Internal Connection Fatal Error pointing - toSQLConnection_productstable.Close()(I take this also to mean that the other user's process has just closed this Connection).

Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding (I am not sure about this one)

Finally, I get one or two errors that sound like queries may not have not returned all the expected records from the database, perhaps because of a DataReader closing too early.

I understand that .NET 1.1 allows only one datareader per connection, but when I tried turning connection pooling off it made no difference. Nor did allowing a high Min Pool size, although when I tried this I could see the many connections in the Windows Perfomance Manager.

I don't really see why I need to check whether the connection object is closed before opening it, or why I need to check that the datareader is not nothing or closed before closing it, but I found I got worse results still if I didn't do these.

I have tried to replicate this problem on my own computer, but it doesn't happen. This is a standalone PC with SQL Server and webserver all on the same machine. The software is the same, except that I have Windows XP Pro 5.1 SP2, and IIS 5.1 The fact this works OK on mine makes me wonder if there might be a .NET setting different on the two, or perhaps the network might cause problems - their webserver is separate from their SQL one. But the errors I am getting do sound like .NET issues that I could fix by changing the program, if only I could work out what to change!

Regards

David Vaughan

Are any of your data access classes or their functions/properties declared as static or shared?|||I was having issues very similar to yours, and suspected that there was either some sort of cross-contamination of my Connection objects or that the DataReader objects used internally by DataSets were somehow not being closed properly prior to the DataReader's associated Connection being returned to the pool.

Apparently, neither was the case. It's still too early in our testing to tell, but it looks like our actual cuplrits were a handful of procedures that used DataReader objects directly. In each case, the DataReader was either not issued a Close followed by a Dispose, or was not nested inside a try/catch block to do the same on an exception.

So, if I were you, I'd comb through the code once more to see if you have any "bad apples" like this spoiling the bunch. In our case, about 1-2 dozen such procedure calls were to blame, hidden within about 19000 total database calls from managed code over the course of roughly a half hour.|||

misfit815:

Apparently, neither was the case. It's still too early in our testing to tell, but it looks like our actual cuplrits were a handful of procedures that used DataReader objects directly. In each case, the DataReader was either not issued a Close followed by a Dispose, or was not nested inside a try/catch block to do the same on an exception.

I've had the same experience as misfit815. All of my DataReaders are encapsulated using blocks (It's like a Try/Finally block which autodisposes), and I make sure that I don't call other functions that might hit the database inside one of those blocks.