Serge's Technology View

Talk about Technologies, Software Architecture and Management

SQL tasks automation: apply scripts from folder

You will be amazed how common this question is:

I have a folder with arbitrary set of SQL scripts and want to apply them all automatically. How do I do that?

In other words:

  • Having a folder – ex: C:\AutoSQLScripts – with some unknown set of SQL scripts
  • And a SQL Instance where scripts need to be applied
  • Run a scheduled task to perform the operation automatically.

Solution:

First, lets figure out single operation - run SQL Script from command line in unattended mode using osql utility:

::
osql -S MYSERVER -d MYDATABASE -E -i MySqlScript.sql  -o report.txt
::

Where:

-S indicates server name
-d is an alternative to USE db_name inside the script
-E indicates that trusted connection would be used
-i identifies the file that contains a batch of SQL statements
-o identifies the file that receives output from osql

Simple so far…

Fun starts when we need to go through files in the folder. Using “magic” of Power Shell, it is also easy:

::
FOR /f "TOKENS=*" %%a IN ('dir /b "%1*.sql"') do ECHO %%a

::or

SET _PATH=%1
IF (NOT %_PATH%=="") SET _PATH = '-p "%_PATH%"'
FORFILES %_PATH% -s -m *.* -c "CMD /C ECHO @FILE"
::

Let’s ignore the end for now and concentrate on the syntax of the loop itself.

FOR /f indicates that we need to perform a loop against a set of files specified in IN. To produce the set we are using “raw” dir of the folder, where %1 is, if specified, an input param of the batch file call.

FORFILES is a little more complex and given here for comparison, yet produces the same result.

Note: commands above work only with “local” (drive: based) locations. If you need apply scripts from Network folder, map it first as a local drive then apply scripts.

Now let’s combine SQL call with the batch processing and we would get the following ApplyScript.cmd which could be used to apply any scripts from current or specific folder to specified SQL Server/Database instance:

::
@ECHO OFF
SET _SERVER=MYSERVER
SET _DB=MYDATABASE

FOR /f "TOKENS=*" %%a IN ('dir /b %1*.sql') DO CALL osql -S %_SERVER% -d %_DB% -E -i %%a  -o report.txt
::

Enjoy.

Bookmark and Share

Crystal Reports 2008 and ASP.Net : speed up the first session

Did you ever noticed that after restart of the IIS ASP.Net page which has Crystal Reports Viewer would take much longer to come up?

Some of it is expected since on initial start up, a number of Crystal assemblies and objects need to be created and this does take time.
I want to emphasize that discussion below is related to the first run of any reports. We have to actually restart/reset IIS to see problem again.

Note: There was a question about IIS’ Application Pools recycling which could cause similar effect. Make sure your application pool is set properly.

Setting up a playground

Let’s assume ASP.Net pages were already migrated to use .Net 3.x and look at what happen behind the scene.

Remember that after restart IIS starts from ground up. This means several things in respect to our page content:

  • .Net assemblies need to be preloaded and validated if necessary.
    Since Crystal Reports 2008 comes as .Net 2.0-based core and we may already moved to .Net 3.x, CR2008 Engine and some additional files need to be loaded and validated aside from one which already used by any previously loaded pages.
  • For each session engine would create a cached version of RPT file in temp folder.
  • Some code would have to be brought to the client machine for Viewer to operate properly.
  • Database connection established, data retrieved, processed, paged and sent to the client Viewer.

That about it, now we ready to work with reports. Please notice that after that initial load, any other reports would come up faster. We could even close the browser or use another browser (ex. go from IE to FF) and it would be still faster than the first time, so there is something important about the first step above…

Combing the sand

Let’s try to break the IIS initialization process to get more detailed view.

Since we already had some ASP.Net pages loaded before coming to CR related page, we could disregard .Net core initialization procedure. It is there on IIS side: used and ready.

Ok, next thing is CR engine related stuff. There is some 5Mb of files in crystalreportviewers12 folder to support CR Viewer in ASP.Net and we need to send some of them over the network. In addition to that there is Crystal Reports Engine assemblies which need to be loaded by IIS at the time of the first use. And this is our spot to dig.

If we try and trace what exactly going on we would notice that aside from IIS loading a few dozen assemblies, there is also process associated with trying connect to CRL.VERISIGN.NET.

What is it?

“Problem” is that assemblies are Authenticode signed and therefore need to be verified or it technical terms they need to be checked against Certificate Revocation List (CRL) by Publisher for Code Access Security (CAS).

Default behavior is that they need to be verified by the certificate authority. If certificate is not present on the same machine (I have my doubts that SAP doing anything about that, but I could be wrong), validation need to be done via central repository mentioned above, or if machine does not have network/internet access the .NET thread might timeout waiting to connect.

Yes, by performing strong name signing of assemblies or placing the CA certificate on the same machine issue would be avoided, but it seems not being a case.

Building the castle

Since assemblies are provided by SAP, we cannot remove digital signature and it is a hassle to keep certificates current by obtaining them from CA every time they expire. Let’s concentrate on the Publisher mentioned above and turn it off.

It is all-or-nothing solution since it would require turning off CRC for the entire IIS.

When working with regular .Net apps, it can be done on the app level (assuming we already have fix for .Net 2.0) by adding the following section in <Application>.exe.config

<configuration>
  <runtime>
    <generatePublisherEvidence enabled="false"/>
  </runtime>
</configuration>

This new element described in this MSDN article. Interesting note there (why not to turn it then by default? Oh, security concerns… UAC anyone?):

We recommend that services use the <generatePublisherEvidence> element to improve startup performance. Using this element can also help avoid delays that can cause a time-out and the cancellation of the service startup.

Since there is no such config file for our ASP.Net app (web.config would not work here, because it defines settings that are only AppDomain wide and we need process wide for aspnet_isapi.dll being the hosting environment for the runtime), we would have to turn code access security (CAS) publisher policy off for the entire IIS.
There are two options:

  • Create a file called w3wp.exe.config (for IIS6, or aspnet_wp.exe.config for IIS5). This will affect all .NET based web applications on the system.
  • Or to specify this in machine.config, but then this affects every .NET application on the machine and is not available for override in individual Apps.

Adding the summer cabin

There is one more step which could be taken to improve performance by preloading some of the core assemblies while site visitor is doing something else.
I wouldn’t go into much details here, since it is implementation/application environment specific, but just give a hint:

  • some other place in application, create a background process which would create CR document object, load some not essential report file, retrieve some data and then disapear without the trace. This would allow Crystal Reports Engine being initialized in the background offsetting time needed for the actual CR related page load. Don’t force garbage collection though, this may cancel desired effect.

Results

In some situation I observed 50% to 70% drop in start-up time…
Have fun!

Bookmark and Share

How to embed Google Docs document into your page

There are plenty of presentations, documents, and spreadsheets we may want to share with the world.

We can use homegrown document handling solutions, use commercial 3rd party solutions or… utilize power provided by Google Docs.

When storing articles on Google Docs you are provided with the option to share them with some people only or with everyone. I like to have option to link my articles from anywhere but I also like to have not just plain page, but a page embedded into my web-site pages.

What it means is that instead of external page, I want to put it in my own box.

There was time when it wasn’t as easy because of the additional code Google was injecting into articles, but time changed and now it is fairly easy and strait forward.

Goals

  • Embed Google Docs document into existing page
  • PHP. This is not really a requirement per se, it is just code below is PHP.
  • Retain copyright information

Solution

The following code snippet allows to get a desired result

// id parameter is used to pass Google Docs arcticle id
// Example: mydocview.php?id=nnnnnnnnnnnnnnnn

$id    = $_REQUEST['id'];
if ($id) {
  $started = "0";
  echo('<div class="googledocs" >Stored and managed via <a title="Learn more about Google Docs" href="http://docs.google.com" target="_blank">Google Docs &copy; -- Web word processing, presentations and spreadsheets.</a></div>');

  // Read document content, parse to normalize and output
  $file = fopen ("http://docs.google.com/View?id=" . $id, "r");
  while (!feof ($file)) {
    $line = fgets ($file, 4096);

    // Parse header info out
    if ($started == "0") {
      if (substr($line, 0, 5) == '<body') { $started = "1"; }
      else { continue; }
    }

    // Parse body content until document content block is detected
    if ($started == "1") {
      if (strstr($line, '<div id="doc-contents">')) { $started = "2"; }
      else { continue; }
    }

    // Output document content
    if ($started == "2") {
      // Stop processing, end of content reached
      if (strstr($line, '<div id="google-view-footer">')) { break; }

      // Normalize image links to keep link back to google docs
      $line = str_replace('src="File?id', 'src="http://docs.google.com/File?id', $line);

      // Normalize links to other articles
      $line = str_replace('href="View?docid=', 'href="/mydocview.php?id=', $line);
      $line = str_replace('href="View?id=', 'href="/mydocview.php?id=', $line);
      $line = str_replace('href="View?docID=', 'href="/mydocview.php?id=', $line);
      $line = str_replace('href="View?', 'href="/mydocview.php?', $line);

      // Output result
      echo($line);
    }
  }
  fclose($file);
}

Sample page:

<html>
<head></head>
<body>
<? // Include code above here
?>
</body>
</html>
Bookmark and Share

Expiration of CrystalReport Viewer pages with long running queries

Wow! It worked…

I continue my journey with Crystal Reports Viewer (CRV) inside ASP.Net pages.
Topic of this post is to discuss problems and solutions related to long running queries in CRV.

Before continue I would like to thank Daniel Paulsen from SAP team for his help resolving the issue discussed below.

What are we talking about here?

As it usually happens at some point there is to much data/not optimal query/broken indexes/etc and it is takes significant time to return a result for CR report. As a result ASP.Net page with CRV start to throw errors, acts erratically, and overall strange.

Since we know that it is ASP.net and CRV uses page state and session information then we have usual suspects - session timeout and execution timeout.

Easy? Not so fast… Lets time our query – if it is more then 10 but less then 20 minutes then reason is not an ASP.Net timeout, but CR Engine timeouts.

Crystal Reports Engine Timeouts

Aside from timeouts imposed by .net infrastructure, there are two others which are introduced by Crystal Report itself.

Symptoms:

  • Error similar to ‘Request timed out because there has been no reply from the server in N ms’
  • First Page of the report is displayed but attempt to go to the next page gives JS error related to invalid object ID
  • If report require parameter entry, they are requested again

Cause:

Crystal Reports by default has a timeout of 10 minutes (600,000 ms) before deciding that no data is returned by the query.
Depend on how code behind is implemented for the report page, timeout error may cause different problems and be hidden.

Solution:

Per Daniel’s suggestion I have changed the following settings to a bigger number and it did solve the problem for me

HKEY_LOCAL_MACHINE\SOFTWARE\Business Objects\Suite 12.0\Report Application Server\Client SDK\CorbaAdapter\WaitReplyTimeout = 600000

and

HKEY_LOCAL_MACHINE\SOFTWARE\Business Objects\Suite 12.0\Report Application Server\InprocServer\EnterpriseRequestTimeout = 600000

Final Note:

There are other settings which are allowed to be set for the CR engine as described in Developer’s Guide for CR 2008 under “Configuring your Web.Config file“. It worth to know about some of them.

Bookmark and Share

Valid XHTML 1.0 Transitional  Valid CSS!