Tag: software development

Tech in the 603, The Granite State Hacker

Metadata in Software Development

Jim Wilcox (The Granite State Hacker)

In modern programming, there’s a lot of cool stuff that can be done with metadata. Failing to consider it in one form or another is setting aside a major tool in any system architecture. Sadly, I see this from time to time, a project with a lot of information associated with it… typically the metadata ends up being expressed only as requirements… and is not applied any further in automation. My post on Edgewater’s blog highlights code generation as one of my favorite ways metadata can be leveraged using things like standardized requirements documentation as metadata.

Wikipedia argues that metadata is divided into two categories, “data about data types” such as XSDs and WSDLs, and “data about content” is only helpful in determining application.  According to their current article, metadata is akin to information about a container such as a box..  how big is it?  how much can it hold?   Metacontent is information about what the box contains… what is it?  how much does it weigh?  what are its storage instructions?  what are the contents of the instruction manual?  The former information is often best applied at design time.  The latter, is often more economically applied at runtime.  I would argue that it goes deeper than that, as well.  One man’s metacontent is another’s metadata. The term metadata could be used to cover:

  • raw data
  • structure/language definition (syntax) (describing how to express a schema) 
  • structure/language expression definition (schema)
  • content
  • content description/classification

Metadata in information systems has two major aspects: expression and application. Expression is everything from the media used to the language (even characters) that are used to communicate information.  Expression implies how the information can be managed and transmitted, and even how the information can be applied. Application is all about how the information is used, impacting utilization of resources requried to leverage it at the time that it is applied. 

In application development, metadata can take many forms.  It can look like lots of different things.  Here’s a list enumerating a few forms in the development world, along with pros and cons of each:

  • BLOB (Fully Custom/Encoded Expression)
    • Notes:  At the core of all other computer data expression is a specialized BLOB.  I mean, on disk, there’s no visible difference between an Excel Spreadsheet and a JPEG image… it’s just a block of bytes.  The meaning of each byte is interpreted by the application that uses it.  (Its beauty is in the eye of the beholder.) 
    • Pro’s:  Most flexible expression.  Any information that can be expressed within a computer can be encoded to live in a Binary file. 
    • Con’s:  Any “beholder” must typically be created from scratch, making it typically tightly coupled with its application.  Harder to manage in terms of media & transmission. 
  • Plain Text
    • Pro’s:  Very flexible, customizable, can be loosely coupled to the application.  Lots of options with respect to media and transmission.
    • Con’s:  Plain text can take any form, often has to be parsed using custom tools, care has to be taken to make sure plain text is extensible, human readable, and structured enough for processing.
  • XML
    • Pro’s:  Much more structured by definition, easy to extend by definition.  Many many useful tools exist to define, manage, communicate, index, and consume XML.  Very loosely tied to its application(s).
    • Con’s:  Not quite as human-readable as say, plain text might be.
  • JSON
    • Pro’s:  Extensible, structured, blurs the line between code & metadata, since JSON (JavaScript Object Notation) is evaluated by runtimes as code.  Meant for web client consumption.
    • Con’s:  not nearly as broadly supported as XML, tools are not available in as many contexts.  Not so well supported for consumption outside web clients.
  • XAML
    • Pro’s:  XML based, and like JSON, is an object initialization notation.   It is more widely known for describing UI elements in WPF and Silverlight.  Directly compiles to code.
    • Con’s:  Requires WPF or Silverlight runtimes to use, not well supported beyond client-side rich apps, media may be typically described as “embedded in binary code”.
  • Embedded in Programming Languages
    • Pro’s:  typically compiled into code, metadata can be expressed as code litterals or code attribution.  This means the data is almost fundamentally instantly available to the running application that consumes it.  Done well, especially with code attribution, this tends to provide meaningful human-readable information to a programmer (typically more so than comments) and a runtime processor at the same time.
    • Con’s:  embedded metadata must be processed at runtime, every time.  In cases where there are large amounts of data involved, done wrong, this can be time consuming.  Typcially forces metadata content management into source code control, which arguably may not be the best way to manage it. (source control of metadata is a “Pro”, but putting it in something like TFS can make it hard to reach.) Litterals and attribution can tie metadata to the code that consumes it.
  • Excel Spreadsheets
    • Pro’s:  typically easily read & shared, easy to manage & communicate as a file. 
    • Con’s:  not so easy to collaborate on, typically concurrent content managers must merge changes.  Automation can be klunky.
  • Database Tables
    • Pro’s:  Easy medium for a developer to work with
    • Con’s: Not so easy a medium for anyone else, hard to manage in source control, requires custom tools for end-users to work with if they need to.
  • SharePoint Lists
    • Pro’s:  Easy medium for anyone to use / collaborate, SharePoint provides content management tools.  Data is easily decoupled, accessible via web (and web services), and exportable to files such as Excel spreadsheets.
    • Con’s:  Hard to keep metadata sync’d with source code control.  (If this is a requirement, the workaround involves exporting documents and checking them in.)

Metadata application, in programming terms, can be accomplished in several different places.  It can be applied at:

  • Design Time
    • Common application:  code generation based on metadata or metacontent information.
  • Run Time
    • Common application:  runtime behavior/presentation modification and/or extension based on metacontent information.  This could include interpreted language processors. One could argue that .NET Common Intermediate Language (CIL) is metadata for that reason, since a runtime interprets it and executes native code from it.
  • Test Time
    • Common application:  baseline test data, test configuration.  (In turn, these can be applied at “design time” such as generating unit test code, or runtime, such as gathering baseline test data to compare to results, or determining executability and/or parameters for tests.)
  • Deploy Time
    • Common application:  Deployment targets/configuration based on build content metacontent.
Code generation is one of the most compelling uses for metadata in information systems, but it’s certainly not the only use. One of my other favorite topics, SharePoint, highlights this by making metacontent a cornerstone of content and document management, classification, reporting, and search indexing. Throwing a document in a plain old file share provides you with a little metadata; the folder names it its file path typically has meaning, as well as the dates, times, and even the permissions applied to it. Throwing the same document in SharePoint provides a ton more information about the document content by default, and typically indexes it, making search a far more powerful tool.
Tech in the 603, The Granite State Hacker

Keeping in the Code

Jim Wilcox (The Granite State Hacker)

At the end of the day, the business solution is always the most important part of the equation, but it’s not the only part.  While I’m working on a solution, I’m also looking at tools, scaffolding, and framework.  This is especially true if others are going to be working on the project, and that accounts for nearly every non-trivial project.

How easy is it to set up?  How easy is it to work with?  Do the expressions make sense?  Can I hand it off to my least experienced teammate, get them to pick this up, and expect reasonable results?  (For that matter, can I hand it off to my most experienced teammate and expect them to respect the design decisions I made? )

Keeping my head in the code is critical.  Loosing touch with tools means shooting in the dark on the above questions.  It doesn’t matter what their experience is, if you ask someone to push a tack into a corkboard, hand them the wrong tools for the job, they won’t be able to push the thumbtack into the corkboard… or you’ll nuke your budget paying for tools that are overpowered for the job.  (But that thumbtack will be SO IN THERE!)

In any case, in most projects, after the architecture & technical designs have been sorted out, frameworks, built, automations put in place, I’ll take on the coding, too.

Of course, I’ve said this before…  if you can really simplify the work, what’s to stop you from taking the extra step and automating it?   I’m always eyeing code, especially “formulaic”, repetititive stuff, looking for opportunities to simplify, abstract, and/or automate.

Tech in the 603, The Granite State Hacker

Custom Guid’s

Jim Wilcox (The Granite State Hacker)

Caught a question from Stacy Draper, @StacyDraper this morning about custom guids, to make them more recognizable.

It reminded me of a post I saw recently about Facebook using hex characters to make IPv6 addresses more recognizable.

Here’s what I was thinking… create a guid that has an embedded word.

For example, the following code creates a Guid that always starts with FACEB00C:

using System;
usingSystem.Collections.Generic;
using System.Linq;
usingSystem.Text;
 
namespaceCustomGuidTest
{
    class Program
    {
        static void Main(string[] args)
        {
            GuidcustomGuid = GenerateCustomGuid();
            Console.WriteLine(customGuid.ToString(“B”));
            Console.ReadKey();
        }
 
        static Guid GenerateCustomGuid()
        {
            Guidresult;
 
            //0xFACEB00C
            //backwards, but required this to achieve desired result
            //Likely due to some standard with respect to bit order.
            byte[] custom = new byte[] { 0x0C, 0xB0, 0xCE, 0xFA  };

            byte[] random = Guid.NewGuid().ToByteArray();
            byte[] final = new byte[16];
            for(int idx = 0; idx <16; idx++)
            {
                if(idx >= custom.Length)
                {
                    final[idx] = random[idx];
                }
                else
                {
                    final[idx] = custom[idx];
                }
            }
            result = newGuid(final);
            returnresult;
        }
    }
}

Example output:
{faceb00c-7828-4e99-8cf5-280e33202670}

Tech in the 603, The Granite State Hacker

GSSPUG Hub (Free App) for Windows Phone Now Available

Jim Wilcox (The Granite State Hacker)

My “artisan portfolio” of Windows Phone apps just DOUBLED in size!  Yes, I’ve now successfully published my second Windows Phone app.  🙂

The Granite State SharePoint Users Group Hub is a somewhat minimal app, but if you’re a member of the group, it’s got some useful features.   My favorites are being able to get info about the next meeting, (both in the app, and as a live tile) and being able to RSVP through EventBright.

The direct link to find it in the Marketplace on your Windows Phone is this.

Regarding the name…  GSSPUG?  Ya, I know… it’s not quite as intuitive as NHSPUG…    

If you’re from New Hampshire, you know you search for “Granite State” any time you’re looking for something local…  and if you don’t know that, it probably is just as well you don’t find it.  😉

One other nice thing is that the content is largely driven from the group’s web site, which, of course, is a SharePoint site.   The app does require a network connection, but it can be updated without having to go through the week-long process of publishing an update. 

Like Jimmy Sudoku, the app uses your phone’s system wide theme colors.

Essentially this is what ends up in the Hub app.

And it appears like so:

Tech in the 603, The Granite State Hacker

Windows Authentication fails due to machine name alias issue

Jim Wilcox (The Granite State Hacker)

Another tidbit I ran into recently that I hit once long ago, and don’t want to forget it again:

I have a VM who’s name I changed.  Trying to develop a WCF webservice for an SOA initiative, I ran into a completely inexplicable authentication failure when I tried to authenticate with my own local user on the machine.

Turns out the quick & easy fix (identified in this post from MS  http://support.microsoft.com/kb/926642 ) ended up pointing me to the solution of:

 adding

REG_MULTI_SZ key “BackConnectionHostNames” to my registry at

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0 . 

One alias per line…

I added a couple lines for various aliases that I use for the machine, enabling me to authenticate as {alias}\{userName}, locally, on the machine.

Tech in the 603, The Granite State Hacker

Url for WSDL coming back with wrong Host name in WCF service

Jim Wilcox (The Granite State Hacker)

Here’s something I don’t want to forget…  working on a WCF Service…

Using TLS, the address of the wsdl didn’t match what was in the SSL certificate.  (the server certificate had the FQDN “friendly name” (host.domain.com), and I was just getting back the “host” name. 

Naturally, this was causing problems. 

There was lots of guidance about setting the host name using some old vbs script that turned out to be a red-herring for IIS 7, anyway.

The quick & simple solution was to add in the behavior for the service:

<system.serviceModel>

    <behaviors>
      <serviceBehaviors>
        <behavior>
          <useRequestHeadersForMetadataAddress />
          <!– To avoid disclosing metadata information,

          set the value below to false and remove the metadata

               endpoint above before deployment –>
          <serviceMetadata httpGetEnabled=“false“ httpsGetEnabled=“true“  />
          <!– To receive exception details in faults for debugging purposes,

          set the value below to true.  Set to false before deployment

          to avoid disclosing exception information –>
          <serviceDebug includeExceptionDetailInFaults=“True“ />
        behavior>
      serviceBehaviors>
    behaviors>
…
  system.serviceModel>
configuration>

Tech in the 603, The Granite State Hacker

Workstation Virtualization

Jim Wilcox (The Granite State Hacker)

I’ve been having fun (ya, really… fun!) with MS VPC 2007 SP1 lately.

I’ve put some VM’s on a portable USB disk. USB2 isn’t the best connection, but it’s workable. It does a few things… adds a spindle to the system config, offloading that overhead from the main disk.

An external disk also means the VM is “portable”. I can launch the VM on my laptop or any other host system (like my home system) without any significant difficulty. Even better, by having the VPC config files on the host, rather than on the external disk, you can tune VM settings (like memory and network connectivity) for optimal conditions on the host system.

One other nice time-saver is differencing disks. You can create a “base” virtual hard disk, and create VHD’s that are deltas of the base… by doing this, you can create a primary configuration, and then create several machines that inherit that basic config. It came in very handy for a recent product evaluation… I just created a base VM roughly according to what the client expects to host the system on, and then created VMs based on that for each product I wanted to evaluate.

Another nice feature is virtual assist hardware. At first, I didn’t know my ThinkPad had it, but it turns out to be a BIOS setting. Flip that on, do a cold boot, and VM performance is visibly better. I knew of some of the other features from past experience with MS VPC 2005, but the hardware acceleration is new to me. (Ironically, my newer home desktop, a 64-bit monster with huge RAM doesn’t support the hardware assist… performance isn’t a problem, there, tho. )

One more trick: enable the Undo disk option… It put another layer of protection on your VM, allowing you snap a line on your VM at a point in time that you can back-out to. The cool part about this is that the Undo disk is created as a temporary file on the host system, (typically on the primary system drive). This distributes load across the spindles even more, which further improves run-time performance. The downside: when it comes time to commit the undo, it can take a while.

I still love the idea, going forward, of putting client dev environments on a config like this… Not only does it create a nice level of separation between client system configurations, but when you get your hands on better hardware, migration is not an issue.

Tech in the 603, The Granite State Hacker

“ETL”ing Source Code

Jim Wilcox (The Granite State Hacker)

The past couple weeks, I’ve been between projects, which has gotten me involved in a number of “odd jobs”. An interesting pattern that I’m seeing in them is querying and joining, and updating data from very traditionally “unlikely” sources… especially code.

SQL databases are very involved, but I find myself querying system views of the schema itself, rather than its contents. In fact, I’m doing so much of this, that I’m finding myself building skeleton databases… no data, just schema, stored procs, and supporting structures.

I’m also pulling and updating metadata from the likes of SharePoint sites, SSRS RDL files, SSIS packages… and most recently, CLR objects that were serialized and persisted to a file. Rather than outputting in the form of reports, in some cases, I’m outputting in the form of more source code.

I’ve already blogged a bit about pulling SharePoint lists into ADO.NET DataSet’s. I’ll post about some of the other fun stuff I’ve been hacking at soon.

I think the interesting part is how relatively easy it’s becoming to write code to “ETL” source code.

Tech in the 603, The Granite State Hacker

Reading SharePoint Lists into an ADO.Net DataTable

Jim Wilcox (The Granite State Hacker)

[Feb 18, 2009: I’ve posted an update to show the newer technique suggested below by Kirk Evans, also compensating for some column naming issues.]

The other day, I needed to write some code that processed data from a SharePoint list. The list was hosted on a remote MOSS 2007 server.

Given more time, I’d have gone digging for an ADO.NET adapter, but I found some code that helped. Unfortunately, the code I found didn’t quite seem to work for my needs. Out of the box, the code missed several columns for no apparent reason.

Here’s my tweak to the solution:

(The ListWebService points to a web service like http://SiteHost/SiteParent/Site/_vti_bin/lists.asmx?WSDL )

private data.DataTable GetDataTableFromWSS(string listName)

{

ListWebService.Lists lists = new ListWebService.Lists();

lists.UseDefaultCredentials = true;

lists.Proxy = null;

//you have to pass the List Name here

XmlNode ListCollectionNode = lists.GetListCollection();

XmlElement List = (XmlElement)ListCollectionNode.SelectSingleNode(String.Format(“wss:List[@Title='{0}’]”, listName), NameSpaceMgr);

if (List == null)

{

throw new ArgumentException(String.Format(“The list ‘{0}’ could not be found in the site ‘{1}'”, listName, lists.Url));

}

string TechListName = List.GetAttribute(“Name”);

data.DataTable result = new data.DataTable(“list”);

XmlNode ListInfoNode = lists.GetList(TechListName);

System.Text.StringBuilder fieldRefs = new System.Text.StringBuilder();

System.Collections.Hashtable DisplayNames = new System.Collections.Hashtable();

foreach (XmlElement Field in ListInfoNode.SelectNodes(“wss:Fields/wss:Field”, NameSpaceMgr))

{

string FieldName = Field.GetAttribute(“Name”);

string FieldDisplayName = Field.GetAttribute(“DisplayName”);

if (result.Columns.Contains(FieldDisplayName))

{

FieldDisplayName = FieldDisplayName + ” (“ + FieldName + “)”;

}

result.Columns.Add(FieldDisplayName, TypeFromField(Field));

fieldRefs.AppendFormat(“”, FieldName);

DisplayNames.Add(FieldDisplayName, FieldName);

}

result.Columns.Add(“XmlElement”, typeof(XmlElement));

XmlElement fields = ListInfoNode.OwnerDocument.CreateElement(“ViewFields”);

fields.InnerXml = fieldRefs.ToString();

XmlNode ItemsNode = lists.GetListItems(TechListName, null, null, fields, “10000”, null, null);

// Lookup fields always start with the numeric ID, then ;# and then the string representation.

// We are normally only interested in the name, so we strip the ID.

System.Text.RegularExpressions.Regex CheckLookup = new System.Text.RegularExpressions.Regex(“^\\d+;#”);

foreach (XmlElement Item in ItemsNode.SelectNodes(“rs:data/z:row”, NameSpaceMgr))

{

data.DataRow newRow = result.NewRow();

foreach (data.DataColumn col in result.Columns)

{

if (Item.HasAttribute(“ows_” + (string)DisplayNames[col.ColumnName]))

{

string val = Item.GetAttribute(“ows_” + (string)DisplayNames[col.ColumnName]);

if (CheckLookup.IsMatch((string)val))

{

string valString = val as String;

val = valString.Substring(valString.IndexOf(“#”) + 1);

}

// Assigning a string to a field that expects numbers or

// datetime values will implicitly convert them

newRow[col] = val;

}

}

newRow[“XmlElement”] = Item;

result.Rows.Add(newRow);

}

return result;

}

// The following Function is used to Get Namespaces

private static XmlNamespaceManager _nsmgr;

private static XmlNamespaceManager NameSpaceMgr

{

get

{

if (_nsmgr == null)

{

_nsmgr = new XmlNamespaceManager(new NameTable());

_nsmgr.AddNamespace(“wss”, “http://schemas.microsoft.com/sharepoint/soap/”);

_nsmgr.AddNamespace(“s”, “uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882”);

_nsmgr.AddNamespace(“dt”, “uuid:C2F41010-65B3-11d1-A29F-00AA00C14882”);

_nsmgr.AddNamespace(“rs”, “urn:schemas-microsoft-com:rowset”);

_nsmgr.AddNamespace(“z”, “#RowsetSchema”);

}

return _nsmgr;

}

}

private Type TypeFromField(XmlElement field)

{

switch (field.GetAttribute(“Type”))

{

case “DateTime”:

return typeof(DateTime);

case “Integer”:

return typeof(int);

case “Number”:

return typeof(float);

default:

return typeof(string);

}

}