Showing posts with label design. Show all posts
Showing posts with label design. Show all posts

Sunday, March 25, 2012

Archive & Restore design question

Hi,

I have a design question.

So, I am designing an web application that will store lots of data.

Because of that we need to allow administrator users to archive some of the data.

However, the user needs the ability to view the archive data (just to know that it is there)

The administrator user can restore the data by using the web application.

My design includes 2 databases, one for the actual data, and the second for the archive data, and I manipulate it by moving data from one database to the other. What do you think about that?

Another thing is the fact that every month administrator will want to empty the archive database and save it on a disk. What is the best way of doing so? and lets say i want to take one of these backup file and restore several records from it into the archive database, what method should I be using?

Finally is there any best practices articles for this kind of issues?

Thanks,

Are you using SQL Server 2005? If you are, you can take advantage of table partitioning. But if not, then you will have to create an application that uses distributed transactions on the two databases you mentioned. Do you have to do all of this in your web application?|||

10x for the response,

All of that should be achived via the web application!

I am using sql 2005.

I am trying to figure out what are the alternatives and what is the best practice for it.

My idea, if using the current design of 2 databases, is to add 3rd database that will handle the restore from file.

Another option is to create a design that will actually to Import/Export to file.

|||

I too have a similar requirement as you have. I wonder if you have found a good design.. Please let me know how are going about diong this task.

Archive & Restore design question

Hi,

I have a design question.

So, I am designing an web application that will store lots of data.

Because of that we need to allow administrator users to archive some of the data.

However, the user needs the ability to view the archive data (just to know that it is there)

The administrator user can restore the data by using the web application.

My design includes 2 databases, one for the actual data, and the second for the archive data, and I manipulate it by moving data from one database to the other. What do you think about that?

Another thing is the fact that every month administrator will want to empty the archive database and save it on a disk. What is the best way of doing so? and lets say i want to take one of these backup file and restore several records from it into the archive database, what method should I be using?

Finally is there any best practices articles for this kind of issues?

Thanks,

Are you using SQL Server 2005? If you are, you can take advantage of table partitioning. But if not, then you will have to create an application that uses distributed transactions on the two databases you mentioned. Do you have to do all of this in your web application?|||

10x for the response,

All of that should be achived via the web application!

I am using sql 2005.

I am trying to figure out what are the alternatives and what is the best practice for it.

My idea, if using the current design of 2 databases, is to add 3rd database that will handle the restore from file.

Another option is to create a design that will actually to Import/Export to file.

|||

I too have a similar requirement as you have. I wonder if you have found a good design.. Please let me know how are going about diong this task.

sql

Archive & Restore design question

Hi,

I have a design question.

So, I am designing an web application that will store lots of data.

Because of that we need to allow administrator users to archive some of the data.

However, the user needs the ability to view the archive data (just to know that it is there)

The administrator user can restore the data by using the web application.

My design includes 2 databases, one for the actual data, and the second for the archive data, and I manipulate it by moving data from one database to the other. What do you think about that?

Another thing is the fact that every month administrator will want to empty the archive database and save it on a disk. What is the best way of doing so? and lets say i want to take one of these backup file and restore several records from it into the archive database, what method should I be using?

Finally is there any best practices articles for this kind of issues?

Thanks,

Are you using SQL Server 2005? If you are, you can take advantage of table partitioning. But if not, then you will have to create an application that uses distributed transactions on the two databases you mentioned. Do you have to do all of this in your web application?|||

10x for the response,

All of that should be achived via the web application!

I am using sql 2005.

I am trying to figure out what are the alternatives and what is the best practice for it.

My idea, if using the current design of 2 databases, is to add 3rd database that will handle the restore from file.

Another option is to create a design that will actually to Import/Export to file.

|||

I too have a similar requirement as you have. I wonder if you have found a good design.. Please let me know how are going about diong this task.

Thursday, March 22, 2012

Architecture design to support global data replication

Hi,
In our global company, there is a proposal to have SQL Server hubs at
specific locations, say one in North America (NA) and the other in Asia (with
scalability option that other hubs could be added as our company grows).
With this mind, we are in the process of designing a strategy to support
bi-directional (and may be 3-way or X number of way in the future)
transactional replication of our PORTAL database between NA and Asia so that
any changes made in the DB in NA server is reflected to the DB in the Asia
server and vice versa. NA server will be used by NA clients and the Asia
server, by Asian clients.
Any ideas what would be the best strategy to implement this, keeping
latency, network speed, and most importantly choice of replication method in
mind? The PORTAL database is designed to be less than 15 GB.
Your thoughts and comments are appreciated. TIA.
I would recommend transactional replication. It if it two way replication I
would use bi-directional transactional replication. With careful filtering
you should be able to scale this to several more nodes.
Peer-to-peer replication is also an option in SQL 2005, however all updates
should originate on one node.
Merge can be used, but it adds latency to each DML.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"Rob" <Rob@.discussions.microsoft.com> wrote in message
news:C84C50BF-571D-4F43-BBB7-BF81BE06E34F@.microsoft.com...
> Hi,
> In our global company, there is a proposal to have SQL Server hubs at
> specific locations, say one in North America (NA) and the other in Asia
> (with
> scalability option that other hubs could be added as our company grows).
> With this mind, we are in the process of designing a strategy to support
> bi-directional (and may be 3-way or X number of way in the future)
> transactional replication of our PORTAL database between NA and Asia so
> that
> any changes made in the DB in NA server is reflected to the DB in the Asia
> server and vice versa. NA server will be used by NA clients and the Asia
> server, by Asian clients.
> Any ideas what would be the best strategy to implement this, keeping
> latency, network speed, and most importantly choice of replication method
> in
> mind? The PORTAL database is designed to be less than 15 GB.
> Your thoughts and comments are appreciated. TIA.

Architecture design to support global data replication

Hi,
In our global company, there is a proposal to have SQL Server hubs at
specific locations, say one in North America (NA) and the other in Asia (wit
h
scalability option that other hubs could be added as our company grows).
With this mind, we are in the process of designing a strategy to support
bi-directional (and may be 3-way or X number of way in the future)
transactional replication of our PORTAL database between NA and Asia so that
any changes made in the DB in NA server is reflected to the DB in the Asia
server and vice versa. NA server will be used by NA clients and the Asia
server, by Asian clients.
Any ideas what would be the best strategy to implement this, keeping
latency, network speed, and most importantly choice of replication method in
mind? The PORTAL database is designed to be less than 15 GB.
Your thoughts and comments are appreciated. TIA.I would recommend transactional replication. It if it two way replication I
would use bi-directional transactional replication. With careful filtering
you should be able to scale this to several more nodes.
Peer-to-peer replication is also an option in SQL 2005, however all updates
should originate on one node.
Merge can be used, but it adds latency to each DML.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"Rob" <Rob@.discussions.microsoft.com> wrote in message
news:C84C50BF-571D-4F43-BBB7-BF81BE06E34F@.microsoft.com...
> Hi,
> In our global company, there is a proposal to have SQL Server hubs at
> specific locations, say one in North America (NA) and the other in Asia
> (with
> scalability option that other hubs could be added as our company grows).
> With this mind, we are in the process of designing a strategy to support
> bi-directional (and may be 3-way or X number of way in the future)
> transactional replication of our PORTAL database between NA and Asia so
> that
> any changes made in the DB in NA server is reflected to the DB in the Asia
> server and vice versa. NA server will be used by NA clients and the Asia
> server, by Asian clients.
> Any ideas what would be the best strategy to implement this, keeping
> latency, network speed, and most importantly choice of replication method
> in
> mind? The PORTAL database is designed to be less than 15 GB.
> Your thoughts and comments are appreciated. TIA.

Architecture design to support global data replication

Hi,
In our global company, there is a proposal to have SQL Server hubs at
specific locations, say one in North America (NA) and the other in Asia (with
scalability option that other hubs could be added as our company grows).
With this mind, we are in the process of designing a strategy to support
bi-directional (and may be 3-way or X number of way in the future)
transactional replication of our PORTAL database between NA and Asia so that
any changes made in the DB in NA server is reflected to the DB in the Asia
server and vice versa. NA server will be used by NA clients and the Asia
server, by Asian clients.
Any ideas what would be the best strategy to implement this, keeping
latency, network speed, and most importantly choice of replication method in
mind? The PORTAL database is designed to be less than 15 GB.
Your thoughts and comments are appreciated. TIA.I would recommend transactional replication. It if it two way replication I
would use bi-directional transactional replication. With careful filtering
you should be able to scale this to several more nodes.
Peer-to-peer replication is also an option in SQL 2005, however all updates
should originate on one node.
Merge can be used, but it adds latency to each DML.
--
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"Rob" <Rob@.discussions.microsoft.com> wrote in message
news:C84C50BF-571D-4F43-BBB7-BF81BE06E34F@.microsoft.com...
> Hi,
> In our global company, there is a proposal to have SQL Server hubs at
> specific locations, say one in North America (NA) and the other in Asia
> (with
> scalability option that other hubs could be added as our company grows).
> With this mind, we are in the process of designing a strategy to support
> bi-directional (and may be 3-way or X number of way in the future)
> transactional replication of our PORTAL database between NA and Asia so
> that
> any changes made in the DB in NA server is reflected to the DB in the Asia
> server and vice versa. NA server will be used by NA clients and the Asia
> server, by Asian clients.
> Any ideas what would be the best strategy to implement this, keeping
> latency, network speed, and most importantly choice of replication method
> in
> mind? The PORTAL database is designed to be less than 15 GB.
> Your thoughts and comments are appreciated. TIA.

Architectural Design Question ?

We currently have a number of SSAS 2000 physical cubes and a single KPI virtual cube that our finance department reports from.A colleague is prototyping a new approach in SSAS 2005 and I’m looking for some feedback.

Please see http://www.dailyware.com/SSAS2005Design.jpg for a high level overview.The lines in red illustrate custom .Net applications that push aggregated data from cubes to some other reporting source (SQL tables, XML?) that our finance department will report from.

Thought that come to my mind include: many points of failure, significant custom development required, none-mainstream, high maintenance cost….

Thanks,

Gary

Same as in AS2000 , in AS2005 you can consolidate several different cubes into a single one using new feature called Linked Measure groups.

Another point here is: you should look into providing your end users not only with static reports, but give them ability for Ad-hoc analysis of your data. There are quite a few applications talking directly to Analysis Services.
For instance, you will see great improvements in Pivot Tables in upcoming release of Office 2007.

As for the building redundancy into your system, you can use ether NLB clustering or MSCS ( microsoft clustering services ) solutions.

Edward.
--
This posting is provided "AS IS" with no warranties, and confers no rights.

|||

I agree...my concern with the proposed design is the fact that it will make it difficult to use several tools coming out later this year. Moreover, our users have requirements to drill into data and this model makes that difficult.

Thanks for the input,

Gary

|||Let me ask another question....do you consider the attached design to be very non-mainstream and possibly difficult to support?|||

To save maitanance costs assosiated with having many cubes, many customers choose to create a single cube ( where possible) with multiple measure groups.

This is one of the main advantages switiching to AS2005, you can bring several fact tables into a single cube and point your reports and client appications to it.

You would start splitting cube into separate once if your calculation logic becomes complex and begins to slow down query performance.

Edward.
--
This posting is provided "AS IS" with no warranties, and confers no rights.

|||

Hi Gary,

Edward already provided detailed feedback - I would just add that, what jumped out at me was the KPI (SQL/XML) layer between the cubes and reporting applications. What is its purpose - you can easily create static reports directly from cubes, in addition to the ad hoc analytic capabilities that Edward mentioned?

What I can conceive of, in the context of KPI's, is the need for supporting metadata beyond what is provided for KPI's in AS 2005. Things like the role of a KPI within the framework of a customized Performance Management Scorecard - an example of this would be the Business Scorecard Manager:

http://office.microsoft.com/en-us/assistance/HA012225141033.aspx

Thursday, March 8, 2012

Application/Security Design: Stored Procedures versus SQL queries

Hello everyone,

I don't know what category would be appropriate for this question but security seems to be close enough.

I have this case scenario: I am running an automated application that extracts data from a web site and stores the data into a table on SQL server 2005. This information is not confidential in the extreme of social insurance #'s, bank account #s, but should not be seen by a typical employee (it has no use for them). After the data has been stored, it retrieves the data from the same table, processes it, and updates the same table. This application runs every hour infinitely.

Should all the insert, update, and select queries be stored under a stored procedure? I am not concern with performance. My concern would fall under design and security.

Is it worth to hide the details of inserting/updating/selecting behind a stored procedure? Or should I just allow the program to send select/update/insert SQL queries?

No employee (other then the developer and the DB admin) or customer ever access this table (They do not have permission from SQL). The username and passwords were created with security in mind.

Any thoughts or ideas?

Thanks for your time, Adrian

It sounds as though 'a typical employee' would not have access to the table. As long as the PUBLIC role cannot access the table, and 'a typical employee' does not have permissions, you are covered.

Using Stored Procedures 'may' be excessive in this situation.

|||

Thank your for your response.

I still am left wondering whether it is worth to hide the details of inserting/updating/selecting behind a stored procedure or whther I should just allow the program to send select/update/insert SQL queries.

I consider the stored procedure as the last "line of defense" when it comes to enforcing business rules. Although in theory of a 3-tier system, the business layer takes care of this, I have seen other developers program some sort of application and they forgot one aspect of the business or some sort of formating error. A stored procedure would correct this b/c it doesn't matter which business layer accesses the SQL server, they all will be enforced by the server.

Do you have an understanding of where I am coming from? Where do we draw the line of what the business layer enforces and what the SQL server enforced?

This is where I'd like to get other opinions on it too.

Thank you for your time, Adrian

|||

Adrian,

First, let me state that in 'almost' all situations, I recommend using stored procedures for a several reasons, including: reuseability, abstraction and security. Reuseabilitiy infurs that the same code is called multiple times from the application. Abstraction allows the procedure code (and how/where the data is actually stored) to be 'tuned' without having to redeploy the application. And of course, security protects the tables from inadvertent alteration and maintains a minimum level of data protection -and may even inject some form of audit trailing.

That said, I recommend against putting excessive 'business rules' into stored procedures. I prefer business rules to be in the 'middle tier' -mainly for scalability reasons, as well as my attempts to keep the data server dedicated to protecting the data. Here, however, you will find a great range of opinions, and as hardware becomes more robust and less expensive, and data clusters have become easier to configure and operate -'scalability' becomes less of an issue.

Now in your situation, you posit that the application is automatic -little chance for SQL injection, and users have no reason for direct data access. You have to balance the extra effort for creating and testing the stored procedures against the benefit. Granted, there 'may' be a slight increase in performance due to reusing compiled procedures -but you are doing the same activities over and over again so even the 'ad-hoc' queries would most likely be in the procedure cache (make sure the queries are properly parameterized). But the main question to ask is: "What is the benefit and what is the cost of mandating stored procedure use above direct table access?"

|||

May I know how to give all store procedures exec permission to one user instead one by one. Thanks in advance for your advice.

eg.

GRANT EXECUTE ON [dbo].[SP_xxx] TO [yyy]

GO

one by one procedure how can do it all for one go.

With regards

Bala

|||Create a database role. Add all users to the role. Grant permissions to the role.|||

Hi,

Thanks for the reply, we everyday delete the store procedure and recreate atleast few hundred. Once we delete the permission lost or new store procedures no permission set. If we know what is the command to run for all store procedures, instead of one by one i will run every time after creating the store procedure.

With regards

Bala

|||

I always recommend adding stored procedures to source control. Each stored procedure as a separate file. Databases can be 'refreshed' from source control in order to keep out development 'detritus'.

Each stored procedure file also contains the necessary permissions. Here is an example: (Works in SQL 2000/2005)

IF EXISTS
( SELECT ROUTINE_NAME
FROM INFORMATION_SCHEMA.ROUTINES
WHERE ROUTINE_NAME = 'MyProcedureName'
)
DROP PROCEDURE dbo.MyProcedureName
GO

CREATE PROCEDURE dbo.MyProcedureName
/****************************************************************
* PROCEDURE: MyProcedureName
* DATE:
* AUTHOR:
*-
* DESCRIPTION:
*
*-
* CODE REVIEW: Date/Who/Status
*-
* VSS revision history: (location in source control)
****************************************************************/
( @.Parameter1 datatype,
, @.Parameter2 datatype
)
AS

SET NOCOUNT ON

-- Procedure Code Here

SELECT
@.Err = @.@.ERROR
, @.RowsAffected = @.@.ROWCOUNT

IF ( @.Err != 0 )
RETURN @.Err

IF ( @.RowsAffected = 0 )
RETURN -1

RETURN 0
GO

GRANT EXECUTE ON dbo.MyProcedureName TO MyCustomRole
GO


|||

Hi Arnie,

Thankyou very much i will do that.

With regards

Bala

Application/Security Design: Stored Procedures versus SQL queries

Hello everyone,

I don't know what category would be appropriate for this question but security seems to be close enough.

I have this case scenario: I am running an automated application that extracts data from a web site and stores the data into a table on SQL server 2005. This information is not confidential in the extreme of social insurance #'s, bank account #s, but should not be seen by a typical employee (it has no use for them). After the data has been stored, it retrieves the data from the same table, processes it, and updates the same table. This application runs every hour infinitely.

Should all the insert, update, and select queries be stored under a stored procedure? I am not concern with performance. My concern would fall under design and security.

Is it worth to hide the details of inserting/updating/selecting behind a stored procedure? Or should I just allow the program to send select/update/insert SQL queries?

No employee (other then the developer and the DB admin) or customer ever access this table (They do not have permission from SQL). The username and passwords were created with security in mind.

Any thoughts or ideas?

Thanks for your time, Adrian

It sounds as though 'a typical employee' would not have access to the table. As long as the PUBLIC role cannot access the table, and 'a typical employee' does not have permissions, you are covered.

Using Stored Procedures 'may' be excessive in this situation.

|||

Thank your for your response.

I still am left wondering whether it is worth to hide the details of inserting/updating/selecting behind a stored procedure or whther I should just allow the program to send select/update/insert SQL queries.

I consider the stored procedure as the last "line of defense" when it comes to enforcing business rules. Although in theory of a 3-tier system, the business layer takes care of this, I have seen other developers program some sort of application and they forgot one aspect of the business or some sort of formating error. A stored procedure would correct this b/c it doesn't matter which business layer accesses the SQL server, they all will be enforced by the server.

Do you have an understanding of where I am coming from? Where do we draw the line of what the business layer enforces and what the SQL server enforced?

This is where I'd like to get other opinions on it too.

Thank you for your time, Adrian

|||

Adrian,

First, let me state that in 'almost' all situations, I recommend using stored procedures for a several reasons, including: reuseability, abstraction and security. Reuseabilitiy infurs that the same code is called multiple times from the application. Abstraction allows the procedure code (and how/where the data is actually stored) to be 'tuned' without having to redeploy the application. And of course, security protects the tables from inadvertent alteration and maintains a minimum level of data protection -and may even inject some form of audit trailing.

That said, I recommend against putting excessive 'business rules' into stored procedures. I prefer business rules to be in the 'middle tier' -mainly for scalability reasons, as well as my attempts to keep the data server dedicated to protecting the data. Here, however, you will find a great range of opinions, and as hardware becomes more robust and less expensive, and data clusters have become easier to configure and operate -'scalability' becomes less of an issue.

Now in your situation, you posit that the application is automatic -little chance for SQL injection, and users have no reason for direct data access. You have to balance the extra effort for creating and testing the stored procedures against the benefit. Granted, there 'may' be a slight increase in performance due to reusing compiled procedures -but you are doing the same activities over and over again so even the 'ad-hoc' queries would most likely be in the procedure cache (make sure the queries are properly parameterized). But the main question to ask is: "What is the benefit and what is the cost of mandating stored procedure use above direct table access?"

|||

May I know how to give all store procedures exec permission to one user instead one by one. Thanks in advance for your advice.

eg.

GRANT EXECUTE ON [dbo].[SP_xxx] TO [yyy]

GO

one by one procedure how can do it all for one go.

With regards

Bala

|||Create a database role. Add all users to the role. Grant permissions to the role.|||

Hi,

Thanks for the reply, we everyday delete the store procedure and recreate atleast few hundred. Once we delete the permission lost or new store procedures no permission set. If we know what is the command to run for all store procedures, instead of one by one i will run every time after creating the store procedure.

With regards

Bala

|||

I always recommend adding stored procedures to source control. Each stored procedure as a separate file. Databases can be 'refreshed' from source control in order to keep out development 'detritus'.

Each stored procedure file also contains the necessary permissions. Here is an example: (Works in SQL 2000/2005)

IF EXISTS
( SELECT ROUTINE_NAME
FROM INFORMATION_SCHEMA.ROUTINES
WHERE ROUTINE_NAME = 'MyProcedureName'
)
DROP PROCEDURE dbo.MyProcedureName
GO

CREATE PROCEDURE dbo.MyProcedureName
/****************************************************************
* PROCEDURE: MyProcedureName
* DATE:
* AUTHOR:
*-
* DESCRIPTION:
*
*-
* CODE REVIEW: Date/Who/Status
*-
* VSS revision history: (location in source control)
****************************************************************/
( @.Parameter1 datatype,
, @.Parameter2 datatype
)
AS

SET NOCOUNT ON

-- Procedure Code Here

SELECT
@.Err = @.@.ERROR
, @.RowsAffected = @.@.ROWCOUNT

IF ( @.Err != 0 )
RETURN @.Err

IF ( @.RowsAffected = 0 )
RETURN -1

RETURN 0
GO

GRANT EXECUTE ON dbo.MyProcedureName TO MyCustomRole
GO


|||

Hi Arnie,

Thankyou very much i will do that.

With regards

Bala